The values file said 2, but 3 pods came up

Goal

In a real Argo CD, use a Helm chart and a Kustomize overlay as sources, and confirm from the results that appear in the cluster where values are decided, how sync phases, waves, and hooks change the application order, and what selfHeal can be made not to revert.

Why it matters

Argo CD does not use Helm as a package manager, only as a template engine. So there is nothing in helm list, and rollback is handled not by Helm but by Git and Argo CD. Instead, values can come from four places (the chart defaults, a values file, the Application's valuesObject, and parameters), so "I edited the values file in Git, so why didn't it change?" becomes a common incident. Kustomize overrides are the same: an image written in the Application is a desired state that is not in the Git repository. Sync is not applied all at once; it is divided into PreSync, Sync, and PostSync phases and waves, it waits until earlier waves become Healthy, and if a hook fails, the later phases do not start. Finally, fields owned by another reconciler, such as an HPA, must be excluded from comparison and sync with ignoreDifferences so that they do not fight selfHeal.

Steps

Create a bare repository /srv/bare/src.git, clone it to /root/capa-src/repo, and commit and push a Helm chart at charts/web. Include Chart.yaml (name web, version 0.1.0), values.yaml (replicaCount: 1, greeting: chart-default), values-prod.yaml (replicaCount: 2, greeting: from-values-file), and two templates (a ConfigMap {{ .Release.Name }}-greeting with data.greeting, and a Deployment {{ .Release.Name }}-web with replicas and the image nginx:1.27-alpine). Give the Application src-helm in /root/capa-src/app-src-helm.yaml (repository git://gitd.gitsrv.svc.cluster.local:9418/src.git main and charts/web, target namespace src-helm, automated sync and CreateNamespace) helm.valueFiles: [values-prod.yaml], apply it, and confirm Synced and Healthy.
Add valuesObject (replicaCount: 3, greeting: from-values-object) and parameters (greeting = from-parameter) to the source.helm of src-helm and apply again. When the sync finishes, write replicas (the spec.replicas of the Deployment src-helm-web in the src-helm namespace, a number) and greeting (the value of the ConfigMap src-helm-greeting) into /root/capa-src/precedence.json.
Argo CD only renders the chart and applies it; it does not create a Helm release. Check the number of Secrets of type helm.sh/release.v1 in the src-helm namespace and the tracking marker Argo CD left on the ConfigMap src-helm-greeting, and write helm_release_secrets (a number), tracking_annotation (the value of that ConfigMap's argocd.argoproj.io/tracking-id), and instance_label (the value of the label app.kubernetes.io/instance if it exists, otherwise null) into /root/capa-src/tracking.json.
Commit and push a Kustomize structure at /root/capa-src/repo/kust. In base, put a Deployment api (replicas 1, label app: api, image nginx:1.27-alpine) and a kustomization, and in overlays/prod, put a kustomization that points at base, sets namePrefix: prod-, and changes api to 2 replicas with replicas. Write the Application src-kust (path kust/overlays/prod, target namespace src-kust, automated sync and CreateNamespace) into /root/capa-src/app-src-kust.yaml, specify nginx=nginx:1.28-alpine with source.kustomize.images, and apply it. The Deployment prod-api must be Healthy with 2 Pods and image 1.28.
Commit and push four files in /root/capa-src/repo/waves. They are a ConfigMap settings (sync-wave -1), a Deployment app with a readinessProbe (sync-wave 0, image nginx:1.27-alpine), a Job smoke (sync-wave 1, busybox:1.36 running echo smoke ok), and a PreSync hook Job (generateName migrate-, hook-delete-policy BeforeHookCreation, busybox:1.36 running echo migrate ok). Apply the Application src-waves (path waves, target namespace src-waves, automated sync and CreateNamespace) as /root/capa-src/app-src-waves.yaml and confirm Synced, Healthy, and the operation Succeeded. Record your observations at that moment in /root/capa-src/waves.json as hook_job (the name of the successful PreSync hook Job), hook_created (that Job's creationTimestamp), and settings_created (the creationTimestamp of the ConfigMap settings).
In one commit, change the mode in waves/config.yaml to green and the command in waves/migrate.yaml to echo migrate failed; exit 1, push, and hard refresh src-waves. After you see the operation fail, write commit (that commit SHA), phase (status.operationState.phase), and live_mode (the mode of the ConfigMap settings in src-waves) into /root/capa-src/failed-hook.json. Then push a new commit that reverts only the migrate command to its original (keeping mode as green), and, if needed, finish the failed operation, so that src-waves becomes Synced, Healthy, and Succeeded on the new commit and mode changes to green.
Add ignoreDifferences (group apps, kind Deployment, jsonPointers /spec/replicas) and the syncOption RespectIgnoreDifferences=true to src-kust and apply again (keep automated sync and selfHeal). Then scale prod-api up to 4 with kubectl scale, wait at least 40 seconds, confirm that replicas is still 4 and src-kust is Synced, and write scaled_at (Unix seconds right after the scale), checked_at, replicas (the value at the time of checking, a number), and sync_status into /root/capa-src/ignore.json.
In /root/capa-src/report.json, write helm_winner (where greeting was decided: one of chart, valueFiles, valuesObject, parameters), replicas_winner (where replicaCount was decided, same choices), helm_installed (whether a Helm release was created, a boolean), image_source (where the prod-api image 1.28 is defined: git or application), hook_blocked_sync (whether the mode did not change when step 6 failed, a boolean), and replicas_owner (the name of the managedFields manager that currently owns the spec.replicas field of prod-api).

Notes

Inside the VM there are k3s, Argo CD v3.5.2, and a git daemon (gitd.gitsrv). /srv/bare/<이름>.git appears as git://gitd.gitsrv.svc.cluster.local:9418/<이름>.git (the placeholder is the repository name).
To preview the rendering result: kubectl -n argocd get app <이름> -o jsonpath='{.status.resources}' (the placeholder is the Application name), and for the operation result: .status.operationState.syncResult.resources.
To make it re-read immediately: kubectl -n argocd annotate app <이름> argocd.argoproj.io/refresh=hard --overwrite (the placeholder is the Application name).
Common mistake: writing the valueFiles path relative to the repository root. It is relative to the chart directory.
Common mistake: in step 6, fixing the hook and moving on after seeing only Synced. If an operation retrying the failed revision remains, the new commit is not applied.
Helm · Kustomize · Sync Phases and Waves · Diffing · Resource Tracking

Put the chart in Git and point at it with an Application

Create a bare repository /srv/bare/src.git, clone it to /root/capa-src/repo, and commit and push a Helm chart at charts/web. Include Chart.yaml (name web, version 0.1.0), values.yaml (replicaCount: 1, greeting: chart-default), values-prod.yaml (replicaCount: 2, greeting: from-values-file), and two templates (a ConfigMap {{ .Release.Name }}-greeting with data.greeting, and a Deployment {{ .Release.Name }}-web with replicas and the image nginx:1.27-alpine). Give the Application src-helm in /root/capa-src/app-src-helm.yaml (repository git://gitd.gitsrv.svc.cluster.local:9418/src.git main and charts/web, target namespace src-helm, automated sync and CreateNamespace) helm.valueFiles: [values-prod.yaml], apply it, and confirm Synced and Healthy.

Argo CD treats a path as a Helm source if it has a Chart.yaml. If you do not give a release name separately, the Application name is used. The values file path is relative to the chart directory.

The values file said 2, but there were 3 Pods

Add valuesObject (replicaCount: 3, greeting: from-values-object) and parameters (greeting = from-parameter) to the source.helm of src-helm and apply again. When the sync finishes, write replicas (the spec.replicas of the Deployment src-helm-web in the src-helm namespace, a number) and greeting (the value of the ConfigMap src-helm-greeting) into /root/capa-src/precedence.json.

The same key exists in the chart defaults, valueFiles, valuesObject, and parameters. Do not guess which one wins; read the values actually created in the cluster.

helm list shows nothing

Argo CD only renders the chart and applies it; it does not create a Helm release. Check the number of Secrets of type helm.sh/release.v1 in the src-helm namespace and the tracking marker Argo CD left on the ConfigMap src-helm-greeting, and write helm_release_secrets (a number), tracking_annotation (the value of that ConfigMap's argocd.argoproj.io/tracking-id), and instance_label (the value of the label app.kubernetes.io/instance if it exists, otherwise null) into /root/capa-src/tracking.json.

helm install leaves a release record Secret in the namespace. The way Argo CD recognizes its own resources (the tracking method) is set by application.resourceTrackingMethod in argocd-cm, so check this version's default yourself.

1.27 in Git, but 1.28 in the cluster

Commit and push a Kustomize structure at /root/capa-src/repo/kust. In base, put a Deployment api (replicas 1, label app: api, image nginx:1.27-alpine) and a kustomization, and in overlays/prod, put a kustomization that points at base, sets namePrefix: prod-, and changes api to 2 replicas with replicas. Write the Application src-kust (path kust/overlays/prod, target namespace src-kust, automated sync and CreateNamespace) into /root/capa-src/app-src-kust.yaml, specify nginx=nginx:1.28-alpine with source.kustomize.images, and apply it. The Deployment prod-api must be Healthy with 2 Pods and image 1.28.

The kustomize field of an Application is overlaid by Argo CD with kustomize edit just before rendering. Remember that this value is in the Application object, not in Git.

The smoke Job was created only after the app was ready

Commit and push four files in /root/capa-src/repo/waves. They are a ConfigMap settings (sync-wave -1), a Deployment app with a readinessProbe (sync-wave 0, image nginx:1.27-alpine), a Job smoke (sync-wave 1, busybox:1.36 running echo smoke ok), and a PreSync hook Job (generateName migrate-, hook-delete-policy BeforeHookCreation, busybox:1.36 running echo migrate ok). Apply the Application src-waves (path waves, target namespace src-waves, automated sync and CreateNamespace) as /root/capa-src/app-src-waves.yaml and confirm Synced, Healthy, and the operation Succeeded. Record your observations at that moment in /root/capa-src/waves.json as hook_job (the name of the successful PreSync hook Job), hook_created (that Job's creationTimestamp), and settings_created (the creationTimestamp of the ConfigMap settings).

Argo CD divides work into PreSync → Sync → PostSync phases, applies in wave-number order within a phase, and before moving to the next wave waits for the resources of the earlier wave to become Healthy. The grader compares creation times with the time the Deployment became Available. A BeforeHookCreation hook Job is deleted at the next sync, so you must record its name and time now for the evidence to remain later.

When the hook failed, the changed setting was not applied

In one commit, change the mode in waves/config.yaml to green and the command in waves/migrate.yaml to echo migrate failed; exit 1, push, and hard refresh src-waves. After you see the operation fail, write commit (that commit SHA), phase (status.operationState.phase), and live_mode (the mode of the ConfigMap settings in src-waves) into /root/capa-src/failed-hook.json. Then push a new commit that reverts only the migrate command to its original (keeping mode as green), and, if needed, finish the failed operation, so that src-waves becomes Synced, Healthy, and Succeeded on the new commit and mode changes to green.

If a PreSync hook fails, that sync operation does not proceed to the Sync phase. Automated sync can retry with the failed revision and block the next commit, so look at status.operationState.operation.sync.revision. The core-mode command (argocd app terminate-op --core) requires a kubeconfig whose current namespace is argocd.

selfHeal leaves the hand-scaled replicas alone

Add ignoreDifferences (group apps, kind Deployment, jsonPointers /spec/replicas) and the syncOption RespectIgnoreDifferences=true to src-kust and apply again (keep automated sync and selfHeal). Then scale prod-api up to 4 with kubectl scale, wait at least 40 seconds, confirm that replicas is still 4 and src-kust is Synced, and write scaled_at (Unix seconds right after the scale), checked_at, replicas (the value at the time of checking, a number), and sync_status into /root/capa-src/ignore.json.

With ignoreDifferences alone, the field is left out only of the comparison, and when a sync happens for another reason it can be overwritten with the Git value. RespectIgnoreDifferences makes sync not touch that field either. It is a common setting for apps whose replicas are managed by an HPA.

Who decided the value: Git, the Application, or the cluster?

In /root/capa-src/report.json, write helm_winner (where greeting was decided: one of chart, valueFiles, valuesObject, parameters), replicas_winner (where replicaCount was decided, same choices), helm_installed (whether a Helm release was created, a boolean), image_source (where the prod-api image 1.28 is defined: git or application), hook_blocked_sync (whether the mode did not change when step 6 failed, a boolean), and replicas_owner (the name of the managedFields manager that currently owns the spec.replicas field of prod-api).

Base it on the JSON you left in the earlier steps and the cluster's managedFields. In kubectl get --show-managed-fields -o json, find the entry that has f:replicas under f:spec. The replicas in status is written by the controller.