Push Overlays to a Real Cluster and Promote Them

Goal

You put a kustomize base and overlays onto a real cluster and use kubectl to check the results produced by namePrefix, common labels, configMapGenerator, and the replicas and image patches. Then you reproduce by hand the flow in which changing a single image-tag line is all it takes to finish a prod promotion.

Why it matters

Half of what a GitOps agent does is rendering. When Argo CD's repo-server sees a kustomization.yaml, it runs kustomize build, and when it sees a Chart.yaml, it runs helm template, to produce the final manifest. In other words, the output you see here with kubectl kustomize is exactly the desired state the agent computes. If you never build this yourself, you cannot explain a situation such as "replicas is not in Git, yet the cluster has 3." The hash suffix of configMapGenerator in particular matters a great deal in practice — when the configuration changes, the ConfigMap name changes, and when the name changes, the Deployment spec changes, so a rolling update happens automatically. If you pin the name, then when only the configuration changes the Pods are not restarted, and you get an incident where they quietly keep running with the old values.

Steps

In /root/cgoa-envs/ns.yaml, declare the namespaces cgoa-dev and cgoa-prod and apply it to the cluster.
In /root/cgoa-envs/base/, write a Deployment web (replicas 1, selector and Pod labels app: web, container name web, image nginx:1.27-alpine, referencing the ConfigMap web-config through envFrom) and a Service web (port 80, targetPort 80). In base/kustomization.yaml, list the two resources, add a setting that attaches the label app.kubernetes.io/part-of: cgoa-shop to all objects, and use configMapGenerator to put the literal GREETING=hello into the name web-config. Do not apply yet; check only the render with kubectl kustomize /root/cgoa-envs/base.
In /root/cgoa-envs/overlays/dev/kustomization.yaml, write resources: [../../base], namespace: cgoa-dev, and namePrefix: dev-, and apply it to the cluster with kubectl apply -k.
In the apply result, check the name of the ConfigMap that the envFrom of the Deployment dev-web points to, and check that a ConfigMap with that same name actually exists in the cgoa-dev namespace. (The name must have a hash suffix after it.)
Create /root/cgoa-envs/overlays/prod/kustomization.yaml — resources: [../../base], namespace: cgoa-prod, namePrefix: prod-, set web to 3 with replicas, and pin the newTag of nginx to 1.27-alpine with images. Then apply it with kubectl apply -k.
Carry out the promotion. Change only the image tag of the prod overlay to 1.28-alpine and apply again. After applying, prod-web in cgoa-prod must be nginx:1.28-alpine, and dev-web in cgoa-dev must still be nginx:1.27-alpine.
In /root/cgoa-envs/diff-dev-prod.txt, save the diff output between the dev overlay render result and the prod overlay render result.

Notes

kubectl apply -k <디렉터리> (the placeholder is the directory) applies the result of kubectl kustomize as is.
The common-label field of kustomization is labels: or commonLabels: depending on the kustomize version. Use whichever of the two works — grading looks at the labels of the resulting objects.
diff a b > out.txt returns exit code 1 when there are differences. Append || true so the shell does not stop.
Common mistake: if you put namespace: in the base, it conflicts with the overlay's namespace and causes confusion. Decide the namespace only in the overlay.

Two environment namespaces

In /root/cgoa-envs/ns.yaml, declare the namespaces cgoa-dev and cgoa-prod and apply it to the cluster.

Get into the habit of creating namespaces declaratively too. You can put two documents in one file and apply them at once, or split them into separate files.

Base and configMapGenerator

In /root/cgoa-envs/base/, write a Deployment web (replicas 1, selector and Pod labels app: web, container name web, image nginx:1.27-alpine, referencing the ConfigMap web-config through envFrom) and a Service web (port 80, targetPort 80). In base/kustomization.yaml, list the two resources, add a setting that attaches the label app.kubernetes.io/part-of: cgoa-shop to all objects, and use configMapGenerator to put the literal GREETING=hello into the name web-config. Do not apply yet; check only the render with kubectl kustomize /root/cgoa-envs/base.

configMapGenerator appends a hash of the content to the name. Do not put it on the cluster yet; check only the render result with kubectl kustomize.

Apply the dev overlay to the cluster

In /root/cgoa-envs/overlays/dev/kustomization.yaml, write resources: [../../base], namespace: cgoa-dev, and namePrefix: dev-, and apply it to the cluster with kubectl apply -k.

kubectl apply -k <디렉터리> (the placeholder is the directory) does the render and the apply in one go. You can see the objects only if you look them up by the name with the namePrefix attached.

Is the hash suffix linked to the workload?

In the apply result, check the name of the ConfigMap that the envFrom of the Deployment dev-web points to, and check that a ConfigMap with that same name actually exists in the cgoa-dev namespace. (The name must have a hash suffix after it.)

kustomize also rewrites every place that references the generated ConfigMap name. Look at which name the Deployment's envFrom points to, and check that a ConfigMap with that name really exists.

Apply the prod overlay

Create /root/cgoa-envs/overlays/prod/kustomization.yaml — resources: [../../base], namespace: cgoa-prod, namePrefix: prod-, set web to 3 with replicas, and pin the newTag of nginx to 1.27-alpine with images. Then apply it with kubectl apply -k.

It is the same base, but replicas and the image tag must differ. Solve it with the overlay's kustomization.yaml alone.

Promote with a one-line image tag change

Carry out the promotion. Change only the image tag of the prod overlay to 1.28-alpine and apply again. After applying, prod-web in cgoa-prod must be nginx:1.28-alpine, and dev-web in cgoa-dev must still be nginx:1.27-alpine.

A promotion is not a new deployment. Just change a single tag value in the prod overlay and apply again. Do not touch dev.

Save the difference between the two environments' renders to a file

In /root/cgoa-envs/diff-dev-prod.txt, save the diff output between the dev overlay render result and the prod overlay render result.

Produce two kubectl kustomize results and compare them with diff. When there are differences, diff's exit code is not 0, so be careful that a pipeline does not treat it as a failure.