From Pods to Self-Healing

Goal

Starting from launching a single Pod by hand, you create a Deployment, a Job, a CronJob, and a DaemonSet each, follow the ownership chain (ownerReferences), and then delete a Pod on purpose and prove with lists from before and after deletion that self-healing really runs.

Why it matters

In practice you almost never create Pods directly. Even so, the reason you need to understand Pods is that every workload controller is ultimately a shell holding a Pod template. Most problems with a Deployment are problems with the Pod template.

The criterion for choosing a controller boils down to one: does this process finish by itself? If you use a Job for a service that does not end, it never completes, and if you use a Deployment for a batch that ends, it restarts every time it exits and looks like CrashLoopBackOff. This is in fact the incident newcomers create most often.

The last step is the heart of this lab. Everyone knows "if you delete a Pod, it comes back," but few people have seen with their own eyes that the Pod that comes back is not the same Pod. The single fact that the name changes is the basis of every principle such as "don't depend on Pod names" and "don't keep state inside a Pod."

Steps

1. Create the namespace kcna-work, and in it create a Pod with image nginx:1.27-alpine named hello and take it to Running.
2. In the same namespace, create a Deployment web. The image is nginx:1.27-alpine, replicas is 3, and the Pod template label is app=web.
3. Raise the replicas of web to 5 and wait until all 5 are Ready.
4. Save the name of the ReplicaSet that web created to /root/kcna-work/rs-name.txt on one line, and pick one Pod of web and save the value of that Pod's metadata.ownerReferences[0].kind to /root/kcna-work/owner.txt on one line.
5. In the same namespace, create a Job report with image busybox:1.36 and completions of 3. Then create a CronJob nightly whose schedule is every day at 03:00 (0 3 * * *).
6. In the same namespace, create a DaemonSet node-agent. The image is busybox:1.36, and give it a long-running command so the container does not exit right away.
7. Prove self-healing. (a) Save the 5 Pod names of web to /root/kcna-work/before.txt, (b) delete one of them and save the deleted name to /root/kcna-work/deleted.txt on one line, and (c) after the new Pod becomes Ready, save the 5 Pod names to /root/kcna-work/after.txt.

Notes

• To extract only Pod names, kubectl get pods -n kcna-work -l app=web -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' is handy.
• Check the ReplicaSet name with kubectl get rs -n kcna-work, and you can see the ownerReferences with kubectl get rs <이름> -n kcna-work -o yaml.
• A DaemonSet has no kubectl create subcommand, so you have to write the YAML yourself and apply it. spec.selector.matchLabels and spec.template.metadata.labels must be the same.
• Common mistake 1: leaving the Job's restartPolicy empty in step 5. The default Always is rejected for a Job.
• Common mistake 2: grabbing after.txt right after deleting the Pod in step 7. The new Pod is not yet in the list, so you do not get 5 lines.

Start your first Pod

Create the namespace kcna-work, and in it create a Pod with image nginx:1.27-alpine named hello and take it to Running.

Create the namespace first and then the Pod inside it. You can create it quickly with kubectl run or write YAML. It may take a few seconds for the Pod to become Running, so check the status before grading.

Hand Pod management to a Deployment

In the same namespace, create a Deployment web. The image is nginx:1.27-alpine, replicas is 3, and the Pod template label is app=web.

kubectl create deployment attaches the Pod template's label automatically. Check once with -o yaml which label was attached. You can see the number of ready replicas on the status side.

Change the number of replicas

Raise the replicas of web to 5 and wait until all 5 are Ready.

You can finish in one line with kubectl scale, or edit the manifest and apply it again. Think about which of the two is the way that leaves a record. Grading looks at both spec and status.

Follow the ownership chain

Save the name of the ReplicaSet that web created to /root/kcna-work/rs-name.txt on one line, and pick one Pod of web and save the value of that Pod's metadata.ownerReferences[0].kind to /root/kcna-work/owner.txt on one line.

If you look at the namespace's ReplicaSet list, a hash is attached after the name. If you open that object's metadata.ownerReferences with -o jsonpath or -o yaml, it tells you who created it. The Pod side has the same field.

Work that finishes: Job and CronJob

In the same namespace, create a Job report with image busybox:1.36 and completions of 3. Then create a CronJob nightly whose schedule is every day at 03:00 (0 3 * * *).

A Job's Pod template cannot use Always for restartPolicy. If you think about why, the value settles itself. A CronJob's schedule is the standard five-field cron notation.

One per node: DaemonSet

In the same namespace, create a DaemonSet node-agent. The image is busybox:1.36, and give it a long-running command so the container does not exit right away.

A DaemonSet manifest has no replicas field, because the one deciding the count is not a person. You cannot create it with kubectl create, so you have to write the YAML yourself, and the selector and template.metadata.labels must match.

Delete a Pod to prove self-healing

Prove self-healing. (a) Save the 5 Pod names of web to /root/kcna-work/before.txt, (b) delete one of them and save the deleted name to /root/kcna-work/deleted.txt on one line, and (c) after the new Pod becomes Ready, save the 5 Pod names to /root/kcna-work/after.txt.

Grading works only if you leave the list before deletion, the deleted name, and the list after deletion, each as a file. You get 5 lines only if you take the list after deletion after the controller has created the new Pod. Check for yourself whether the name of the Pod that came back is the same as or different from the one you deleted.