The Configuration Changed. Why Didn't the Process?

In one line

Recovery is complete only when you know not just where a configuration is stored but when the process consumes it.

Why this was needed

You found the configuration that makes the business path return 500. You fixed it to 200 in Git, and Argo CD is Synced to the new commit too. If you read the ConfigMap through the Kubernetes API, the new value is clearly there. Yet curl keeps getting 500. If at this point you delete the Pod, saying "GitOps isn't listening," the problem may disappear for a moment. But if you miss which configuration was delivered to the process by which process, the same incident will repeat at the next change. Let us first separate three layers: the source, the delivered file, and the running process.

How it works

A ConfigMap is configuration data stored in the Kubernetes API. Depending on how it is consumed, the update behavior differs. A value injected as an environment variable does not automatically replace the environment variable of an existing process. A file projected through an ordinary volume can be updated, but propagation may take time. Even if the file changes, whether the application re-reads it is a separate matter. In particular, a container that mounts a single file with subPath, as in this lab, does not receive subsequent updates to the ConfigMap. If you memorize the sentence "a ConfigMap is updated automatically" while omitting how it is consumed, you will be wrong in exactly this experiment. Official ConfigMap update conditions.

This Deployment mounts the single file nginx.conf at /etc/nginx/nginx.conf with subPath. The first configuration commit fixes only the nginx.conf in the ConfigMap. The Deployment's spec.template is preserved. So there is no reason for a new Pod to be created. The previous Pod UID and container remain, the file read inside the container is also the previous value, and the HTTP response is still 500. This is not the result of guessing "it's just that the cache is slow"; it is a delivery boundary confirmed by comparing files and identities. Blindly increasing the sleep time does not change this way of consuming.

The second commit computes the SHA-256 of the new configuration bytes and puts it in the checksum/config annotation of the Pod template. The name of this annotation is not backed by some built-in Kubernetes magic. It is a convention that makes the spec.template value differ so that the Deployment creates a new ReplicaSet and Pod. If you attach the checksum only to the Deployment's top-level metadata, the template does not change, so you cannot get the same effect. The new Pod builds its mount from the current ConfigMap and starts nginx. If you separate the commit that changes only the ConfigMap from the commit that also changes the template, you can follow this difference with your own eyes. Official Deployment updates.

A checksum is not a value you write down by eye. You must compute it from the actual configuration bytes, and whether or not there is a final newline also changes the result. This helper computes it by encoding, as is, the nginx.conf value of the saved ConfigMap observation. In real operations, the pipeline that generates configuration files and templates must share the same rule. The checksum of a different file or an arbitrary timestamp can also trigger a rollout, but that is not evidence that explains which configuration we deployed.

What it looks like in the field

You must distinguish an emergency measure from a permanent recovery. Even if you can manually delete a Pod to make it read the new configuration, that measure does not leave the desired deployment state and the intent of the change in Git. A direct edit of a resource with selfHeal turned on may be reconciled back to the state in Git. In this lab, you do not change the platform's operating configuration; you leave the recovery process in two commits in a student-only repository. Automated sync is not a feature that fixes every wrong configuration, and it requires human review and testing. Official automated sync.

For confirming that a rollout is complete, names alone are not enough either. The new Pod's UID must change, but the Application and the Deployment must keep the same UID. Deleting the previous Pod and recreating the resource wholesale under the same name is not the same recovery process. Only by checking together the desired template's checksum, observedGeneration, updatedReplicas, Ready, the file inside the new Pod, and the actual response can you make the claim that "a new process that read the new configuration responded."

The per-step JSON in the lab is learning material for the convenience of writing the report. It was made so that preserving the raw data and hashes makes it fail if you accidentally overwrite a previous observation, but the limitation remains that root on the same VM can change every file. Do not call it remote attestation or a tamper-proof audit system just because hashes exist. For production auditing, you must separately design logs sent outside the trust boundary, access control, and a retention policy.

What you will do in the next lab

You preserve the observations before and after the configuration change and after the template change. Investigate whether, in the middle observation, the ConfigMap has the new value but the mounted file and the response have the old value. Finally, you follow even the parent relationship of the Git commits to tell the two changes apart, and confirm the new Pod's 200 response. Re-grading a past step must read past records and must not re-inject the failure. Only when the current-state check also passes do you declare recovery within the investigated scope, and you record the external path and continuous observation as remaining work.