TT Lab
Get started
Learn Learning paths Courses

GitOps and Argo CD

My Value Keeps Reverting: Finding the Field's Owner

Continue in TT Lab

Goal

On the kwok cluster, you create two managers with server-side apply and read managedFields directly. You trigger with real objects the conflict and force, the letting go of fields, and even the conflict on the scale subresource, and at the end you connect it with Argo CD's ignore rule.

Why it matters

Even if you turn on GitOps, the subject that writes the cluster's objects is not just one. Argo CD writes, an autoscaler writes replicas, a webhook inserts a container, and a person writes with kubectl when in a hurry. Server-side apply is a device by which the API server holds, as a record, "who is this field's owner". Only if you can read this record can you answer three questions — why is my apply rejected, why, if I push it in by force, is the other side rejected next time, and why, when I let go of a value, does it not return to the old value. And this record connects to the Argo CD side. This is because the managedFieldsManagers of the ignore rule picks what to compare by this ownership record, not by field path.

Steps

  1. On the kwok cluster, create the namespace ga-ssa, and in /root/ga-ssa/deploy.yaml, write a Deployment web — spec.replicas 2, the selector and label are app: web, and one container (web, image nginx:1.25). Apply it with kubectl apply --server-side --field-manager=gitops.
  2. Read the ownership record with kubectl -n ga-ssa get deploy web -o json --show-managed-fields and save it, one line per manager in the form <관리자> | <연산> (manager, then operation), to /root/ga-ssa/managers.txt. gitops must be in it with the Apply operation.
  3. In /root/ga-ssa/hotfix.yaml, write the same Deployment but with spec.replicas at 5 and the image at nginx:1.27. Try a server-side apply with --field-manager=hotfix, and save the failure output, including standard error, to /root/ga-ssa/conflict.txt. You do not use --force-conflicts yet.
  4. Do the same apply again together with --force-conflicts and save the success output to /root/ga-ssa/force.txt. Now the cluster's image is nginx:1.27, and the owner of that field is hotfix.
  5. Create /root/ga-ssa/hotfix-slim.yaml — the same as hotfix.yaml but without the spec.replicas line. After a server-side apply with --field-manager=hotfix, save the list of managers that own spec.replicas to /root/ga-ssa/release.txt (an empty file if there are none). Check what the cluster's replicas becomes.
  6. In /root/ga-ssa/web-csa.yaml, write a Deployment web-csa (same namespace, the label and container name are web-csa, image nginx:1.25) and put it up, without --server-side, with a plain kubectl apply. For the two Deployments (web and web-csa), save the list of metadata.annotations keys to /root/ga-ssa/csa.txt.
  7. In /root/ga-ssa/scale-deploy.yaml, write a Deployment web-scale (2 replicas, label and container name web-scale, image nginx:1.25) and apply it server-side with --field-manager=gitops. Then raise it with kubectl -n ga-ssa scale deploy web-scale --replicas=4, and try applying the same file again with the same manager name. Save the failure output to /root/ga-ssa/scale-conflict.txt.
  8. Save to /root/ga-ssa/live.yaml the cluster's web as YAML including the ownership record (-o yaml --show-managed-fields). In /root/ga-ssa/argocd-cm.yaml, put the key resource.customizations.ignoreDifferences.apps_Deployment and in managedFieldsManagers, write hotfix, and in jsonPointers, /spec/replicas. Save the output of argocd admin settings resource-overrides ignore-differences /root/ga-ssa/live.yaml --argocd-cm-path /root/ga-ssa/argocd-cm.yaml to /root/ga-ssa/ignore.txt.

Notes

State your name and apply

On the kwok cluster, create the namespace ga-ssa, and in /root/ga-ssa/deploy.yaml, write a Deployment web — spec.replicas 2, the selector and label are app: web, and one container (web, image nginx:1.25). Apply it with kubectl apply --server-side --field-manager=gitops.

--field-manager is the name that tells the API server "who made this change". Server-side apply records which fields are owned, per this name — a concept that client-side apply does not have.

Read the ownership record

Read the ownership record with kubectl -n ga-ssa get deploy web -o json --show-managed-fields and save it, one line per manager in the form <관리자> | <연산> (manager, then operation), to /root/ga-ssa/managers.txt. gitops must be in it with the Apply operation.

If you leave out this option, .metadata.managedFields comes out as null — because kubectl hides it by default. Go through the array with jq and pull out manager and operation.

When a second manager tries to write the same field

In /root/ga-ssa/hotfix.yaml, write the same Deployment but with spec.replicas at 5 and the image at nginx:1.27. Try a server-side apply with --field-manager=hotfix, and save the failure output, including standard error, to /root/ga-ssa/conflict.txt. You do not use --force-conflicts yet.

The server rejects, saying "someone else holds this field", and tells you in a list which fields. Without this rejection, two automations would quietly overwrite each other's values. Be careful that your answer sheet does not stop with the failure.

Take ownership by force

Do the same apply again together with --force-conflicts and save the success output to /root/ga-ssa/force.txt. Now the cluster's image is nginx:1.27, and the owner of that field is hotfix.

Force does not remove the conflict; it moves the ownership. The old owner drops out of that field, so the next time the old owner applies its own value again, this time it meets the conflict — if two automations use force by turns, it becomes an endless fight.

If you let go of a field, the value does not go back to the owner

Create /root/ga-ssa/hotfix-slim.yaml — the same as hotfix.yaml but without the spec.replicas line. After a server-side apply with --field-manager=hotfix, save the list of managers that own spec.replicas to /root/ga-ssa/release.txt (an empty file if there are none). Check what the cluster's replicas becomes.

If you remove a field from the manifest, that manager lets go of that field in the ownership list. But the value does not return to the old owner's value — a field nobody owns becomes the API default. For the owner list, with jq you can pick the entries where .fieldsV1."f:spec"."f:replicas" is not null.

What client-side apply leaves behind

In /root/ga-ssa/web-csa.yaml, write a Deployment web-csa (same namespace, the label and container name are web-csa, image nginx:1.25) and put it up, without --server-side, with a plain kubectl apply. For the two Deployments (web and web-csa), save the list of metadata.annotations keys to /root/ga-ssa/csa.txt.

Client-side apply puts "the manifest I sent last time" whole into one annotation and compares against it to decide what to delete. So there is no concept of an owner, and when two tools handle the same object, they delete each other's fields. The one put up server-side does not have that annotation.

kubectl scale comes in through a different door

In /root/ga-ssa/scale-deploy.yaml, write a Deployment web-scale (2 replicas, label and container name web-scale, image nginx:1.25) and apply it server-side with --field-manager=gitops. Then raise it with kubectl -n ga-ssa scale deploy web-scale --replicas=4, and try applying the same file again with the same manager name. Save the failure output to /root/ga-ssa/scale-conflict.txt.

kubectl scale writes not to the whole object but to a subresource called scale. So the manager kubectl and the subresource scale are recorded separately in the ownership record, and the conflict message also tells you that fact. The conflict an autoscaler creates has exactly this shape.

Remove from the comparison by owner name

Save to /root/ga-ssa/live.yaml the cluster's web as YAML including the ownership record (-o yaml --show-managed-fields). In /root/ga-ssa/argocd-cm.yaml, put the key resource.customizations.ignoreDifferences.apps_Deployment and in managedFieldsManagers, write hotfix, and in jsonPointers, /spec/replicas. Save the output of argocd admin settings resource-overrides ignore-differences /root/ga-ssa/live.yaml --argocd-cm-path /root/ga-ssa/argocd-cm.yaml to /root/ga-ssa/ignore.txt.

These two are the ways to express in Argo CD "do not compare fields the autoscaler wrote" — remove by path or remove by manager name. However, the preview command renders only path rules, so what the manager-name rule will actually filter, a person must read the ownership record to judge.