Create a Synced outage and recover through Git

Goal

On a real k3s and Argo CD, you perform a healthy deployment → a readiness breakage → a recovery with Git revert. Afterward, you turn into code what you must check when approving a new deployment.

Why it matters

GitOps faithfully applies even a wrong declaration. If you do not separate Synced from service availability, you end up approving a breakage as a success. This time you observe real processes, EndpointSlices, and Service HTTP. Recreate and 1 replica are experimental conditions for seeing the outage clearly on purpose, and they are not a recommended production availability configuration. This is the recovery of a stateless web configuration, not a DB recovery or a zero-downtime canary verification. Do not run commands against any cluster or Git repository outside this VM.

What is prepared

• k3s v1.36.4+k3s1, Argo CD v3.5.2, an nginx image, and the cnpe-shop namespace inside a personal VM
• The Git working copy /srv/gitops and the push target /srv/bare/app.git
• The URL Argo reads: git://gitd.gitsrv.svc.cluster.local:9418/app.git
• The read-only observation tool python3 /opt/fixtures/cnpe-gitops-lab/evidence.py

The first setup can take several minutes. The estimated study time is 55 minutes, and if you need more, extend the time before it expires. After the session ends, the VM and files are reclaimed, so download any records you need beforehand.

Steps

1. In /srv/gitops/app/release.json, write the cnpe-shop/web Deployment and Service as a v1 List. Use 1 replica, Recreate, a progress deadline of 20 seconds, the container web, the image docker.io/library/nginx:1.27-alpine, label and selector app=web, and Service 80→80. When the container starts, write labhub-cnpe-v1 without a trailing newline into nginx's index.html and run nginx. The readiness probe is /·80, a period of 2 seconds with a failure threshold of 1, the request is 25m/32Mi, and the limit is 200m/128Mi. Commit and push, and save the full SHA in /root/cnpe/baseline.sha.
2. In /root/cnpe/project.json, write the AppProject cnpe-delivery in the argocd namespace, and in /root/cnpe/application.json, write the Application cnpe-shop, and apply them. The project allows only the one prepared Git URL, the cnpe-shop destination, and Deployment/Service, and leaves the cluster-scoped allowlist empty. The Application uses this project, the app path in Git, the initial full SHA, the in-cluster destination, and automated sync, prune, and selfHeal.
3. With the observation tool, check Synced, Healthy, replicas/endpoints of 1, HTTP 200, and the exact body for the initial SHA, and save the result in /root/cnpe/baseline.json. You keep this file afterward as the record of the initial success.
4. Change only the readiness path in the Git manifest to /not-ready, push a new commit, and save the full SHA in /root/cnpe/bad.sha. Update the Application targetRevision to this SHA as well and refresh it so that it reads the new Git. Do not patch the Deployment directly.
5. Observe that it became Synced but Degraded, ProgressDeadlineExceeded, ready/available/endpoints of 0, and HTTP failure, and save it in /root/cnpe/failed.json. Do not guess the contents; collect the actual API and request results.
6. Use git revert on the breaking commit to create a new recovery commit whose content is the same as the initial manifest. Push it, save the full SHA in /root/cnpe/recovered.sha, and update the Application to this new SHA as well. Do not erase past history or forcibly rewind to the initial SHA.
7. On the new recovery SHA, check again Synced, Healthy, replicas/endpoints of 1, and the exact HTTP response, and save it in /root/cnpe/recovered.json. Grading checks both the saved files and the current service.
8. Write /root/cnpe/release-gate.py. Its arguments are the status JSON path, the expected full SHA, and the expected HTTP body. If all of the approval contract below holds, it exits with exit code 0, otherwise with a non-zero value. Do not modify the status file. Check both the current healthy state and the counterexamples of per-field errors and omissions.

Step 8 approval contract

• The expected SHA is 40 lowercase hexadecimal digits and matches both target_revision and observed_revision
• sync=Synced, health=Healthy
• replicas, updated, available, ready, and ready_endpoints are each the integer 1 (booleans and strings are rejected)
• generation is a positive integer, observed_generation is also an integer, and the two match
• http_code=200 and http_body exactly matches the expected body
• If any of the required fields above is missing or different, reject. progress_deadline_exceeded is not a required field of this approval contract

Reference

For observation, use .../evidence.py observe, and to wait for a condition, use .../evidence.py wait healthy 전체SHA or .../evidence.py wait failed 전체SHA (the placeholder stands for the full SHA). Save to the JSON file for the relevant step with >. If the condition is not met within 45 seconds at most, it shows an error, so check the API, probes, and events and then run it again. The wait command does not deploy in your place or fix the state.

Steps 1, 3, 4, 5, and 6 check the Git history and the observation records you kept. Do not erase the past outage record just because you recovered. Step 2 rechecks the current wiring and permissions, and steps 7 and 8 recheck the current service. So the full grading can pass even after recovery. The record JSON is a learning record and not tamper-proof evidence. Step 8 grading injects counterexamples into a temporary copy and does not modify the student's file.

• Automated sync and selfHeal
• Git tracking strategies
• Deployment status and progress deadline

Leave a healthy release in Git

In /srv/gitops/app/release.json, write the cnpe-shop/web Deployment and Service as a v1 List. Use 1 replica, Recreate, a progress deadline of 20 seconds, the container web, the image docker.io/library/nginx:1.27-alpine, label and selector app=web, and Service 80→80. When the container starts, write labhub-cnpe-v1 without a trailing newline into nginx's index.html and run nginx. The readiness probe is /·80, a period of 2 seconds with a failure threshold of 1, the request is 25m/32Mi, and the limit is 200m/128Mi. Commit and push, and save the full SHA in /root/cnpe/baseline.sha.

If you only create the file, the repo-server cannot see it. Push to the prepared bare repository and record the full SHA.

Connect the Application with a narrowed scope

In /root/cnpe/project.json, write the AppProject cnpe-delivery in the argocd namespace, and in /root/cnpe/application.json, write the Application cnpe-shop, and apply them. The project allows only the one prepared Git URL, the cnpe-shop destination, and Deployment/Service, and leaves the cluster-scoped allowlist empty. The Application uses this project, the app path in Git, the initial full SHA, the in-cluster destination, and automated sync, prune, and selfHeal.

An AppProject is not just something whose name differs from the default. Look at the actual sourceRepos, destinations, and the resource kind allowlist.

Collect evidence of the initial success

With the observation tool, check Synced, Healthy, replicas/endpoints of 1, HTTP 200, and the exact body for the initial SHA, and save the result in /root/cnpe/baseline.json. You keep this file afterward as the record of the initial success.

observe reads once, and wait healthy followed by the full SHA waits for the condition for up to 45 seconds. Distinguish stderr from stdout.

Create a valid but wrong deployment

Change only the readiness path in the Git manifest to /not-ready, push a new commit, and save the full SHA in /root/cnpe/bad.sha. Update the Application targetRevision to this SHA as well and refresh it so that it reads the new Git. Do not patch the Deployment directly.

An Application pinned to a full SHA does not automatically track new commits on main. Change the SHA to deploy as well.

Observe an outage that is Synced

Observe that it became Synced but Degraded, ProgressDeadlineExceeded, ready/available/endpoints of 0, and HTTP failure, and save it in /root/cnpe/failed.json. Do not guess the contents; collect the actual API and request results.

A running process and readiness are different things. Before the progress deadline passes, it may still be Progressing.

Recover while preserving Git history

Use git revert on the breaking commit to create a new recovery commit whose content is the same as the initial manifest. Push it, save the full SHA in /root/cnpe/recovered.sha, and update the Application to this new SHA as well. Do not erase past history or forcibly rewind to the initial SHA.

revert creates a new commit with the opposite change. Recover the Git content and the targetRevision together.

Compare the new commit with the actual response

On the new recovery SHA, check again Synced, Healthy, replicas/endpoints of 1, and the exact HTTP response, and save it in /root/cnpe/recovered.json. Grading checks both the saved files and the current service.

Do not assume the current service is healthy based only on the saved healthy JSON. The current targetRevision, endpoints, and HTTP must match too.

A gate that does not approve empty evidence

Write /root/cnpe/release-gate.py. Its arguments are the status JSON path, the expected full SHA, and the expected HTTP body. If all of the approval contract below holds, it exits with exit code 0, otherwise with a non-zero value. Do not modify the status file. Check both the current healthy state and the counterexamples of per-field errors and omissions.

True equals 1 in Python, but it is not the correct type for a replica count. Check omissions, types, generations, and body separately.