Environment Differences With Bases and Overlays

Goal

You produce the two environments dev and prod from a single base, and become able to connect those overlays as the source of an ArgoCD Application.

Why it matters

If you manage per-environment YAML by copying, it will inevitably diverge. After it diverges, nobody knows which side is the right one, and "but it worked in dev" begins. An overlay is a structure that writes what is common once and leaves only the differences as files, so it structurally prevents that divergence. What to look at especially in this lab is the hash suffix of configMapGenerator — when the configuration content changes, the ConfigMap name changes, the Pod template that refers to it changes, and a rollout happens by itself. Without the hash, only the ConfigMap is updated and the Pods keep running holding the old configuration, the class of incident hardest to trace to a cause. Finally, it also touches on how ArgoCD handles Helm — ArgoCD does not do helm install; it applies the result the repo-server renders with helm template. So even if you run helm list on the cluster, nothing is visible, and rollback is done with a git commit, not a Helm revision.

Steps

The base instruction file is /root/gitops/kustomize/base/kustomization.yaml. Copy /opt/lab/fixtures/gitops/seed/deployment.yaml and service.yaml to /root/gitops/kustomize/base/, and create kustomization.yaml in the same directory. Write apiVersion: kustomize.config.k8s.io/v1beta1, kind: Kustomization, and the two file names in resources. Keep the base's spec.replicas at 2 (if the base is 1, it is treated as a failure).
Save the result with kustomize build /root/gitops/kustomize/base > /root/gitops/kustomize/out/base.yaml. The result must have kind: Deployment and kind: Service and must not contain kind: Kustomization.
In the base kustomization.yaml, put namePrefix: labhub- and add the common label app.kubernetes.io/part-of: labhub-platform (both the form labels: followed by - pairs: and commonLabels: are accepted). When you build again and update out/base.yaml, the Deployment name must become labhub-web and the Service's metadata.labels must also carry that label.
Create /root/gitops/kustomize/overlays/dev/kustomization.yaml. The first item of resources is ../../base, namespace is gitops-dev, and nameSuffix is -dev. Save it with kustomize build /root/gitops/kustomize/overlays/dev > /root/gitops/kustomize/out/dev.yaml.
Put a strategic merge patch file (for example, patch-deployment.yaml) in the dev overlay and, in kustomization.yaml, write its path under patches. The patch must lower the Deployment's spec.replicas to 1 and, to the container web, add the environment variable LOG_LEVEL=debug. The patch file's metadata.name is web, the original name written in the base. The base's replicas must stay 2. Build again and update out/dev.yaml.
In the dev overlay's kustomization.yaml, use configMapGenerator to create a ConfigMap named app-config (for example, in literals, LOG_FORMAT=json). And in the patch from step 5, make the container web refer to it, in envFrom, with configMapRef.name: app-config. When you build again, a content hash must be attached to the end of the ConfigMap name and the reference name in the Deployment must have changed to that hashed name. In /root/gitops/kustomize/out/hash-note.txt, write in Korean that thanks to the hash suffix, a configuration change leads to a Pod rollout.
In the argocd namespace, create kind: Application with metadata.name: platform-helm. In spec.source.helm.valueFiles, put values-prod.yaml, and in spec.source.helm.parameters, put the name image.tag with a value (for example, 1.27.3), and set spec.source.targetRevision not to HEAD but to a pinned value (for example, v1.4.0).
Create the overlay /root/gitops/kustomize/overlays/prod/ — resources is ../../base, namespace is gitops-prod, with a patch make spec.replicas 3 or more, and with images change the image tag to be different from dev (for example, name: nginx, newTag: 1.27.3). Save the build result to /root/gitops/kustomize/out/prod.yaml. Then, in the argocd namespace, create kind: Application with metadata.name: web-dev — spec.source.path must contain overlays/dev, and put at least one image override in spec.source.kustomize.images.

Notes

To create the Applications in steps 7 and 8, the ArgoCD CRDs must be registered first. As in the previous lab, find and apply them from the offline bundle in /opt/crds/ (grep -l applications.argoproj.io /opt/crds/*.yaml).
kustomize build prints the result to standard output. Before redirecting, create the output directory (/root/gitops/kustomize/out/) first.
Even if a name prefix is attached, a patch finds its target by the original name written in the base. If you get an error that it cannot find the target, instead of patches, you may write it with the old patchesStrategicMerge.
Common mistake 1: fixing the base to make dev's replicas 1. Then prod becomes 1 too. The base is the common denominator, and values specific to an environment go in an overlay.
Common mistake 2: fixing an overlay and not building again. Grading reads the out/*.yaml files, so every time you fix a configuration, you must save the corresponding build result anew.

Set up a kustomize base

The base instruction file is /root/gitops/kustomize/base/kustomization.yaml. Copy /opt/lab/fixtures/gitops/seed/deployment.yaml and service.yaml to /root/gitops/kustomize/base/, and create kustomization.yaml in the same directory. Write apiVersion: kustomize.config.k8s.io/v1beta1, kind: Kustomization, and the two file names in resources. Keep the base's spec.replicas at 2 (if the base is 1, it is treated as a failure).

kustomization.yaml is an instruction file, not a manifest. The names written in resources must actually exist in the same directory.

Save the base build result

Save the result with kustomize build /root/gitops/kustomize/base > /root/gitops/kustomize/out/base.yaml. The result must have kind: Deployment and kind: Service and must not contain kind: Kustomization.

Capture the standard output of kustomize build 디렉터리 (the placeholder stands for the directory) into a file. The instruction file itself must not come out mixed into the result — because it is not an output.

Attach a name prefix and a common label

In the base kustomization.yaml, put namePrefix: labhub- and add the common label app.kubernetes.io/part-of: labhub-platform (both the form labels: followed by - pairs: and commonLabels: are accepted). When you build again and update out/base.yaml, the Deployment name must become labhub-web and the Service's metadata.labels must also carry that label.

Name transformation and label attachment are both declared in the base's kustomization. For the label, both the new labels with pairs and the old commonLabels work. After fixing it, you must build again for the result file to change.

Create the dev overlay

Create /root/gitops/kustomize/overlays/dev/kustomization.yaml. The first item of resources is ../../base, namespace is gitops-dev, and nameSuffix is -dev. Save it with kustomize build /root/gitops/kustomize/overlays/dev > /root/gitops/kustomize/out/dev.yaml.

An overlay puts the base into resources by a relative path. The overlay decides the namespace and the name suffix, and save the build result separately as a dev-only file.

Override the base with a patch

Put a strategic merge patch file (for example, patch-deployment.yaml) in the dev overlay and, in kustomization.yaml, write its path under patches. The patch must lower the Deployment's spec.replicas to 1 and, to the container web, add the environment variable LOG_LEVEL=debug. The patch file's metadata.name is web, the original name written in the base. The base's replicas must stay 2. Build again and update out/dev.yaml.

You do not touch the base. In the patch file, write the original name written in the base, and the container name must match for the merge to happen.

Connect the configuration generator and the hash suffix

In the dev overlay's kustomization.yaml, use configMapGenerator to create a ConfigMap named app-config (for example, in literals, LOG_FORMAT=json). And in the patch from step 5, make the container web refer to it, in envFrom, with configMapRef.name: app-config. When you build again, a content hash must be attached to the end of the ConfigMap name and the reference name in the Deployment must have changed to that hashed name. In /root/gitops/kustomize/out/hash-note.txt, write in Korean that thanks to the hash suffix, a configuration change leads to a Pod rollout.

A content hash is attached to the end of the name the generator made. If a workload refers to that ConfigMap, the reference name is updated along with it — why this is useful is the key of this step.

Write an Application that uses a Helm source

In the argocd namespace, create kind: Application with metadata.name: platform-helm. In spec.source.helm.valueFiles, put values-prod.yaml, and in spec.source.helm.parameters, put the name image.tag with a value (for example, 1.27.3), and set spec.source.targetRevision not to HEAD but to a pinned value (for example, v1.4.0).

Per-environment value files and individual parameters that change with each deployment go into different places. If you leave the revision at the latest of a branch, the same declaration deploys something different at different times.

An overlay-based Application and the prod build

Create the overlay /root/gitops/kustomize/overlays/prod/ — resources is ../../base, namespace is gitops-prod, with a patch make spec.replicas 3 or more, and with images change the image tag to be different from dev (for example, name: nginx, newTag: 1.27.3). Save the build result to /root/gitops/kustomize/out/prod.yaml. Then, in the argocd namespace, create kind: Application with metadata.name: web-dev — spec.source.path must contain overlays/dev, and put at least one image override in spec.source.kustomize.images.

The prod overlay uses the same base as dev but must differ in namespace, replica count, and image tag. The Application side also has a separate place to swap the image tag.