Two Grammars for Handling Environment Differences

In one sentence

If you express the difference between dev and prod with copies of YAML, they will inevitably diverge, but if you express it with a Kustomize overlay or a Helm values file, only the difference remains as a file.

Why this was needed

Suppose you deploy the same app to three environments. Only three things differ: replicas, the log level, and the image tag. If you make deployment-dev.yaml and deployment-prod.yaml separately, the first month is comfortable. But after six months the two files have become different creatures. The security setting added only to prod is missing in dev, and the probe setting fixed in dev is not reflected in prod. The moment you reach a state where nobody knows which side is the right one, the sentence "but it worked in dev" begins.

The direction of the solution is the same. Write what is common just once and write only the differences separately. Kustomize solves this as "lay patches on a base", and Helm solves it as "inject values into a template".

How it works

Kustomize's unit is kustomization.yaml. A base has complete manifests and the instruction file that ties them together, and an overlay brings in the base with resources and writes only the transformations.

Field	What it does
resources	What to bring in (files or another kustomization directory)
namePrefix / nameSuffix	Strings to attach before and after names
namespace	Sets the namespace of all resources at once
labels (formerly commonLabels)	Attaches a common label to all resources
patches	Overrides only specific fields with a strategic merge patch
images	Swaps the image name and tag
configMapGenerator	Generates a ConfigMap from files or literals

There are two important properties. First, the result of kustomize build does not contain the Kustomization itself. This is because it is an instruction file, not an output. If a Kustomization appears mixed into the build result, somewhere an instruction file was wrongly brought in with resources.

Second, a content hash is attached to the end of the name of a ConfigMap made by configMapGenerator. It becomes a name like app-config-9b2f4kt6md, and the reference name in a Deployment that refers to that ConfigMap changes automatically along with it. Thanks to this design, when you edit a configuration file, the Pod template changes and a rollout happens by itself. Without the hash, only the ConfigMap changes and the Pods keep running holding the old configuration, the class of incident that is hardest to trace to a cause.

Helm takes a different approach. It builds the manifests by injecting values.yaml into templates, and gives environment differences through value files such as values-prod.yaml or individual parameters. In ArgoCD these two correspond to spec.source.helm.valueFiles and spec.source.helm.parameters. One thing you must know here — ArgoCD does not do helm install. The repo-server renders the manifests with helm template and then applies them. So even if you run helm list on the cluster, nothing shows up, and rollback is also done with a git commit rather than a Helm revision.

What you see in the field

First, the mistake of fixing the base to fit dev. If you want dev's replicas to be 1 and change the base to 1, prod becomes 1 too. The base must be the common denominator of all environments, and values specific to an environment must always be in an overlay.

Second, the trap of targetRevision: HEAD. Following the latest of a branch is convenient, but the same Application definition deploys something different at different times. If you want a reproducible deployment, pin it to a tag or a commit hash. It is exactly the same reason as forbidding the :latest image tag.

Third, the name prefix and the patch target. If namePrefix is attached in the base, the name in the build result is labhub-web, but a patch finds its target by the original name written in the base. If you don't know this rule, you wander for a long time in "the patch isn't being applied".

What you must decide when using both together

Argo CD can render a Helm chart and then lay kustomize on top. It is convenient, but if you do not decide how far each tool is responsible for, nobody can predict the final result.

The criterion is simple. What can be expressed as values goes to Helm's values, and what cannot be expressed as values (adding a sidecar, patching a specific field, putting in one more resource) goes to a kustomize patch. If you override with kustomize a value that Helm already exposes, you must look in two places to know the final value.

Look at the rendered result with your own eyes and commit it. If you produce the YAML that will actually be applied in the pipeline and put it up for review, the reviewer looks at the result rather than the value files.

helm template app ./chart -f values-prod.yaml \
  | kustomize cfg cat > rendered/prod.yaml

This approach (keeping rendered manifests in git) makes the repository larger, but in exchange, what will change shows up as it is in the diff. Half of GitOps's benefits come from here.

When you leave rendering to Argo CD, pin the versions. If the versions of Helm and kustomize change, the render result can change. Pin the version on the Argo CD side, and render with the same version in CI too to compare.

Do not decide the namespace in two places. If the chart's namespace value, kustomize's namespace:, and the Argo CD Application's destination.namespace differ from one another, resources get scattered. Decide it in only one place.

CRDs are the exception. Helm's crds/ is not updated on upgrade, and kustomize patches pass schema validation only if the CRD already exists. It is more stable to split CRDs into a separate Application and put sync-wave in front.

Never commit secrets through either tool. Both tools render plain text as it is.

What you will do in the next lab

Under /root/gitops/kustomize/, you make a base and dev and prod overlays, lay on name prefixes, common labels, namespaces, strategic merge patches, configMapGenerator, and images one by one, and leave in files how the kustomize build result changes each time. Then you write an Application that uses a Helm source and an Application that points to a Kustomize overlay, and express as declarations how ArgoCD accepts the two syntaxes.