TT Lab
Get started
Learn Learning paths Courses

CNPA — Cloud Native Platform Engineering Associate

A change meant for dev showed up in prod too

Continue in TT Lab

Goal

On a real k3s and Argo CD, you deploy two environments, dev and prod, from Git with a single ApplicationSet. You experience an incident in which one line in the common base changes both environments at once, then bind prod to a promotion branch to turn promotion between environments into a Git operation, and confirm how drift recovery, promotion, and reverting are each recorded in the repository and the cluster.

Why it matters

In GitOps, Git is the desired state of each environment, and the in-cluster reconciler pulls that state in and matches it. So the answer to "what is in prod" must be given by the commit that the Git reference prod tracks points to, and promotion becomes the act of moving that reference. If all environments watch the same branch, the promotion step disappears and dev experiments become prod immediately. Conversely, if you split references per environment, how far a change has gone is proven by the repository history, and even if someone edits the cluster by hand, the reconciler reverts it to Git. This structure is the basic skeleton when a platform team provides a delivery path to many teams.

Steps

  1. Create the bare repository /srv/bare/shop-envs.git and clone it to /root/cnpa-env/repo. Put the ConfigMap shop-config (no namespace, data FEATURE_CHECKOUT_V2: "off", LOG_LEVEL: "info") in base/configmap.yaml with a base/kustomization.yaml that contains it, and in envs/dev/kustomization.yaml and envs/prod/kustomization.yaml put a patch that loads ../../base and changes LOG_LEVEL to debug and warn respectively. Commit and push to main.
  2. Write and apply the ApplicationSet shop (argocd namespace) in /root/cnpa-env/appset.yaml. The list generator's elements are {env: dev, revision: main} and {env: prod, revision: main}, and the template uses the name shop-{{env}}, project default, repoURL git://gitd.gitsrv.svc.cluster.local:9418/shop-envs.git, targetRevision {{revision}}, path envs/{{env}}, destination namespace shop-{{env}}, automated sync with prune and selfHeal, and CreateNamespace=true. Wait until both Applications become Synced.
  3. To turn on the new checkout screen in dev, change FEATURE_CHECKOUT_V2 in base/configmap.yaml to "on", commit, and push. After both apps have synced to that commit, write commit (that commit's SHA), dev_value, and prod_value (the actual value of FEATURE_CHECKOUT_V2 in each environment's ConfigMap) into /root/cnpa-env/incident.json.
  4. Create and push a branch release/prod at the commit just before the incident (the parent of the step 3 commit), and change the prod element's revision in the ApplicationSet to release/prod (leave dev as main). After confirming that shop-prod synced to that commit and FEATURE_CHECKOUT_V2 is off again, write release_prod_initial (the SHA of the commit where you created the branch) and prod_value_after_pin into /root/cnpa-env/pin.json.
  5. In the ConfigMap shop-config in the shop-prod namespace, change LOG_LEVEL to debug with kubectl patch, and measure the seconds it takes Argo CD to revert it to the value in Git. Write uid (the uid of that ConfigMap), edited (debug), restored (the value it returned to), and seconds (an integer) into /root/cnpa-env/drift.json.
  6. Having confirmed the new checkout screen in dev, promote to prod. Push release/prod as a fast-forward to the step 3 commit (no force push), and wait until shop-prod syncs to that commit and FEATURE_CHECKOUT_V2 becomes on. Write from (the release/prod SHA before the move) and to (the SHA after the move) into /root/cnpa-env/promote.json.
  7. Add a file missing.yaml that is not in the resources of envs/dev/kustomization.yaml, then commit and push. When a comparison error appears in shop-dev, read that message, and at the same moment read shop-prod's status.sync.revision; then revert with git revert, push, and wait for shop-dev to become Synced again. Write bad_commit, revert_commit, dev_error (part of the error message), and prod_revision_during into /root/cnpa-env/revert.json.
  8. Write dev_tracks and prod_tracks (each app's targetRevision), promoted_commit (the to value from step 6), drift_seconds (step 5), bad_commit_reached_prod (whether the step 7 bad_commit is in the release/prod history now, a boolean), and rollback (which of revert or reset you used in step 7) into /root/cnpa-env/report.json.

Notes

Two environments in one repository

Create the bare repository /srv/bare/shop-envs.git and clone it to /root/cnpa-env/repo. Put the ConfigMap shop-config (no namespace, data FEATURE_CHECKOUT_V2: "off", LOG_LEVEL: "info") in base/configmap.yaml with a base/kustomization.yaml that contains it, and in envs/dev/kustomization.yaml and envs/prod/kustomization.yaml put a patch that loads ../../base and changes LOG_LEVEL to debug and warn respectively. Commit and push to main.

Write only the differences in the overlay for per-environment differences. You can put a small patch targeting the ConfigMap name in the kustomization's patches. Before you push, check the render result with kubectl kustomize envs/prod. The bare repository is read by the git daemon Pod, so chmod -R a+rX is needed.

One ApplicationSet creates two environments

Write and apply the ApplicationSet shop (argocd namespace) in /root/cnpa-env/appset.yaml. The list generator's elements are {env: dev, revision: main} and {env: prod, revision: main}, and the template uses the name shop-{{env}}, project default, repoURL git://gitd.gitsrv.svc.cluster.local:9418/shop-envs.git, targetRevision {{revision}}, path envs/{{env}}, destination namespace shop-{{env}}, automated sync with prune and selfHeal, and CreateNamespace=true. Wait until both Applications become Synced.

The ApplicationSet controller fills in the template's {{...}} for each element to create an Application and ties them together with an owner reference. Put values that must differ per environment as keys in the element. Argo CD reads Git at its default interval, so if you do not want to wait, attach the argocd.argoproj.io/refresh=hard annotation to the Application.

A change made in dev also showed up in prod

To turn on the new checkout screen in dev, change FEATURE_CHECKOUT_V2 in base/configmap.yaml to "on", commit, and push. After both apps have synced to that commit, write commit (that commit's SHA), dev_value, and prod_value (the actual value of FEATURE_CHECKOUT_V2 in each environment's ConfigMap) into /root/cnpa-env/incident.json.

If both environments track the same branch and load the same base, one line in the base is a change to both environments. It means there is no promotion step between environments. Read the value with kubectl -n shop-prod get cm shop-config -o jsonpath=....

Bind prod to a promotion branch

Create and push a branch release/prod at the commit just before the incident (the parent of the step 3 commit), and change the prod element's revision in the ApplicationSet to release/prod (leave dev as main). After confirming that shop-prod synced to that commit and FEATURE_CHECKOUT_V2 is off again, write release_prod_initial (the SHA of the commit where you created the branch) and prod_value_after_pin into /root/cnpa-env/pin.json.

If prod tracks a reference that moves separately rather than main, changes on main do not reach prod until someone moves that reference. You can create a branch directly on the remote with git push origin <SHA>:refs/heads/release/prod (where the placeholder stands for the commit SHA). If you edit the Application directly, the ApplicationSet reverts it to the template, so you must edit the generator's element.

When prod was edited by hand, it reverted a few seconds later

In the ConfigMap shop-config in the shop-prod namespace, change LOG_LEVEL to debug with kubectl patch, and measure the seconds it takes Argo CD to revert it to the value in Git. Write uid (the uid of that ConfigMap), edited (debug), restored (the value it returned to), and seconds (an integer) into /root/cnpa-env/drift.json.

An app with selfHeal turned on compares again immediately when an object it manages changes, and if it differs from Git, it matches it to Git. It fixes the object rather than deleting and recreating it, so the uid stays the same. Wait while reading the value at 0.5 to 1 second intervals.

Promotion is one commit that moves a branch

Having confirmed the new checkout screen in dev, promote to prod. Push release/prod as a fast-forward to the step 3 commit (no force push), and wait until shop-prod syncs to that commit and FEATURE_CHECKOUT_V2 becomes on. Write from (the release/prod SHA before the move) and to (the SHA after the move) into /root/cnpa-env/promote.json.

If promotion is a Git operation, who put what into prod and when remains as is in the repository history, and reverting is done the same way. A fast-forward works only when the commit before the move is an ancestor of the commit after the move.

The commit that broke dev did not reach prod

Add a file missing.yaml that is not in the resources of envs/dev/kustomization.yaml, then commit and push. When a comparison error appears in shop-dev, read that message, and at the same moment read shop-prod's status.sync.revision; then revert with git revert, push, and wait for shop-dev to become Synced again. Write bad_commit, revert_commit, dev_error (part of the error message), and prod_revision_during into /root/cnpa-env/revert.json.

Argo CD does not apply a commit it cannot render, and leaves a ComparisonError in the app conditions (status.conditions). Prod tracks a different reference, so it never sees this commit. Reverting is not a reset that erases history but a revert that stacks the opposite change as a new commit.

Record the environments and the promotion as values

Write dev_tracks and prod_tracks (each app's targetRevision), promoted_commit (the to value from step 6), drift_seconds (step 5), bad_commit_reached_prod (whether the step 7 bad_commit is in the release/prod history now, a boolean), and rollback (which of revert or reset you used in step 7) into /root/cnpa-env/report.json.

Calculate from the JSON of the earlier steps and git merge-base --is-ancestor. The grader checks the same files, repository, and Applications again.