TT Lab
Get started
Learn Learning paths Courses

GitOps and Argo CD

We Ignored the Constant OutOfSync, and Real Drift Vanished Too

Continue in TT Lab

Goal

You write ignore rules in the two places, argocd-cm and the Application, and confirm with the CLI's diff which lines a rule actually removes from the comparison. You see with your own eyes what disappears together when you ignore too broadly.

Why it matters

If you turn on automatic synchronization, you soon meet "always OutOfSync". It is because of things that cannot be written in the repository but inevitably arise in the cluster: replicas changed by an HPA, a sidecar inserted by a webhook, annotations attached by a controller. An ignore rule is the tool for handling this, but it has a force that tilts only one way — if you ignore broadly, the screen goes quiet, and if you ignore narrowly, it stays noisy. So everyone goes to the broad side. The price of a broad rule is not visible right away. It comes to light only on the day, months later, when someone changes an image by hand and nobody knows. The habit of not writing rules by guesswork but confirming with your own eyes "the lines this rule erases" and then committing prevents this price.

Steps

  1. In /root/ga-ignore/live.yaml, in the ga-ignore namespace, create a Deployment web — spec.replicas is 4, there are two containers, app (image nginx:1.25) and sidecar (image envoy:1.31), and in metadata.annotations put deployment.kubernetes.io/revision: "7". /root/ga-ignore/argocd-cm.yaml is an empty ConfigMap with data: {}. Save the output of argocd admin settings resource-overrides ignore-differences /root/ga-ignore/live.yaml --argocd-cm-path /root/ga-ignore/argocd-cm.yaml to /root/ga-ignore/none.txt.
  2. In /root/ga-ignore/argocd-cm-pointer.yaml, put the key resource.customizations.ignoreDifferences.apps_Deployment and make it ignore, with jsonPointers, only /spec/replicas. Save the output for /root/ga-ignore/live.yaml to /root/ga-ignore/pointer.txt.
  3. In /root/ga-ignore/argocd-cm-jq.yaml, make it ignore only the image of the container named sidecar with a single jqPathExpressions. Save the output to /root/ga-ignore/jq.txt. The app container's image must not be removed.
  4. In a single /root/ga-ignore/argocd-cm-both.yaml, put jsonPointers (/spec/replicas) and jqPathExpressions (the sidecar image) together and save the output to /root/ga-ignore/both.txt. Both must be removed and the app container's image must remain.
  5. In /root/ga-ignore/argocd-cm-wide.yaml, with jsonPointers, make it ignore the whole /spec and save the output to /root/ga-ignore/wide.txt. This output includes even the container image lines — meaning that even if someone secretly changes the image, it is not caught in the comparison.
  6. In /root/ga-ignore/argocd-cm-mfm.yaml, in managedFieldsManagers, put only kubectl and save the output to /root/ga-ignore/mfm-only.txt. Then, in /root/ga-ignore/argocd-cm-mfm2.yaml, put the same managedFieldsManagers together with jsonPointers (/metadata/annotations) and save the output to /root/ga-ignore/mfm-plus.txt. See how the two outputs differ.
  7. In a single /root/ga-ignore/argocd-cm-updates.yaml, put two keys — resource.customizations.ignoreDifferences.apps_Deployment ignores, with jsonPointers, /spec/replicas, and resource.customizations.ignoreResourceUpdates.apps_Deployment ignores, with jsonPointers, /metadata/annotations. For the same file, save the ignore-differences output to /root/ga-ignore/updates-diff.txt and the ignore-resource-updates output to /root/ga-ignore/updates-upd.txt.
  8. In /root/ga-ignore/application.yaml, create the Application ga-ignore-web (namespace argocd, project default) and apply it to the kwok cluster. In spec.ignoreDifferences, put one entry with group apps and kind Deployment, and put inside it both the /spec/replicas pointer and the jq expression that picks the sidecar image. And in /root/ga-ignore/ignore-matrix.tsv, write four or more lines of <ConfigMap파일>\t<찾을 글자>\t<yes|no> (ConfigMap file, string to look for, yes or no), check all of them with /root/ga-ignore/check-ignore.sh, and save the output to /root/ga-ignore/ignore-result.txt. The script must write to standard output only and must end with a non-zero code if even one line is wrong.

Notes

First look at the output when there is no rule

In /root/ga-ignore/live.yaml, in the ga-ignore namespace, create a Deployment web — spec.replicas is 4, there are two containers, app (image nginx:1.25) and sidecar (image envoy:1.31), and in metadata.annotations put deployment.kubernetes.io/revision: "7". /root/ga-ignore/argocd-cm.yaml is an empty ConfigMap with data: {}. Save the output of argocd admin settings resource-overrides ignore-differences /root/ga-ignore/live.yaml --argocd-cm-path /root/ga-ignore/argocd-cm.yaml to /root/ga-ignore/none.txt.

If there is not a single rule, this command does not invent a verdict and says 'it is not configured'. The later steps keep using this one file as material, so get the values exactly right.

Point at one field and take it out

In /root/ga-ignore/argocd-cm-pointer.yaml, put the key resource.customizations.ignoreDifferences.apps_Deployment and make it ignore, with jsonPointers, only /spec/replicas. Save the output for /root/ga-ignore/live.yaml to /root/ga-ignore/pointer.txt.

A JSON pointer is a path that descends with slashes. The lines starting with < in the output are the 'lines removed from the comparison' — only the replicas line should be removed and the container images must remain as they are.

Only jq can pick from an array by a condition

In /root/ga-ignore/argocd-cm-jq.yaml, make it ignore only the image of the container named sidecar with a single jqPathExpressions. Save the output to /root/ga-ignore/jq.txt. The app container's image must not be removed.

A JSON pointer can point to an array only by position number, so if the order changes, you end up ignoring the wrong element. With a jq expression, you can pick by a condition like select(.name == "sidecar"). Start the expression at .spec.template.spec.containers[].

Use both methods together in one rule

In a single /root/ga-ignore/argocd-cm-both.yaml, put jsonPointers (/spec/replicas) and jqPathExpressions (the sidecar image) together and save the output to /root/ga-ignore/both.txt. Both must be removed and the app container's image must remain.

There is one rule block for a kind, and the two lists sit side by side inside it. If you try to split the rules into several blocks per kind, the keys overlap and only the later one remains.

If you ignore too broadly, drift never shows up again

In /root/ga-ignore/argocd-cm-wide.yaml, with jsonPointers, make it ignore the whole /spec and save the output to /root/ga-ignore/wide.txt. This output includes even the container image lines — meaning that even if someone secretly changes the image, it is not caught in the comparison.

The fastest way to get rid of OutOfSync is to ignore broadly, which is why it is also the mistake made most often in the field. Compare whether the string nginx is visible in the step 4 output and in this output.

A rule that ignores by manager name cannot be previewed

In /root/ga-ignore/argocd-cm-mfm.yaml, in managedFieldsManagers, put only kubectl and save the output to /root/ga-ignore/mfm-only.txt. Then, in /root/ga-ignore/argocd-cm-mfm2.yaml, put the same managedFieldsManagers together with jsonPointers (/metadata/annotations) and save the output to /root/ga-ignore/mfm-plus.txt. See how the two outputs differ.

This rule decides what to ignore not by 'which field' but by 'who wrote the field'. So you cannot calculate what will be removed from the resource YAML alone, and the preview command cannot render anything from this list alone either.

The synchronization verdict and waking the reconcile loop are different handles

In a single /root/ga-ignore/argocd-cm-updates.yaml, put two keys — resource.customizations.ignoreDifferences.apps_Deployment ignores, with jsonPointers, /spec/replicas, and resource.customizations.ignoreResourceUpdates.apps_Deployment ignores, with jsonPointers, /metadata/annotations. For the same file, save the ignore-differences output to /root/ga-ignore/updates-diff.txt and the ignore-resource-updates output to /root/ga-ignore/updates-upd.txt.

The two keys have different purposes — the first is 'do not count this difference as OutOfSync', and the second is 'do not wake the reconcile loop for this change'. The latter is a handle that reduces the controller's load, so it does not change the synchronization verdict. Record as they are how the two outputs differ.

A rule that applies to just one app, and a regression check

In /root/ga-ignore/application.yaml, create the Application ga-ignore-web (namespace argocd, project default) and apply it to the kwok cluster. In spec.ignoreDifferences, put one entry with group apps and kind Deployment, and put inside it both the /spec/replicas pointer and the jq expression that picks the sidecar image. And in /root/ga-ignore/ignore-matrix.tsv, write four or more lines of <ConfigMap파일>\t<찾을 글자>\t<yes|no> (ConfigMap file, string to look for, yes or no), check all of them with /root/ga-ignore/check-ignore.sh, and save the output to /root/ga-ignore/ignore-result.txt. The script must write to standard output only and must end with a non-zero code if even one line is wrong.

A global rule (argocd-cm) applies to all apps, and an Application's rule applies only to that app. The two are merged and applied, so hanging a broad rule on the global side and narrowing it in the app is impossible — the narrowing direction must be put on the app side from the start. In the table, mix in lines that check 'does this rule erase this string', such as argocd-cm-wide.yaml and nginx.