Building a Manifest Repository and Catching Drift

Goal

You create a git repository holding manifests and apply it to the cluster, and you become able to detect manually made changes (drift) and revert to the repository's standard.

Why it matters

The rule of GitOps is one sentence — the repository is right and the cluster follows. The moment you keep this sentence, the question "what is running in production right now" becomes a question you can answer with git log, and a rollback becomes the ordinary task of git revert. Conversely, if even one habit of fixing the cluster directly remains, the repository becomes a document that cannot explain reality, and from that moment reproduction becomes impossible. This is why you use kubectl diff repeatedly in this lab — diff is a drift meter that answers with an exit code "how far apart are the declaration and reality", and it is doing by human hands exactly the same judgment as what ArgoCD shows on the screen as OutOfSync. In this environment the ArgoCD controller does not run, so in the last step you write by yourself, as a script, what that controller does for you.

Steps

Create the directory /root/gitops/repo, run git init on it, and set user.name and user.email in that repository. Then create /root/gitops/repo/README.md and make the first commit (there must be at least one commit).
Copy /opt/lab/fixtures/gitops/seed/deployment.yaml and service.yaml to /root/gitops/repo/apps/web/. In metadata.labels, the Deployment must have app.kubernetes.io/managed-by: gitops; in spec.selector.matchLabels, the app.kubernetes.io/name must be web; and spec.replicas must be stated explicitly rather than left to the default (here it starts at 2). The container image must be a value with a pinned tag, like nginx:1.27 (:latest is treated as a failure). README.md must also still be there.
Track all three files, apps/web/deployment.yaml, apps/web/service.yaml, and README.md, with git add, and commit with a meaningful message of 10 characters or more. When you finish, the output of git status --porcelain must be empty.
Apply with kubectl apply -n gitops-lab -f /root/gitops/repo/apps/web/ and save the entire output to /root/gitops/out/apply.txt. After applying, the gitops-lab namespace must have a Deployment web and a Service web, and the Deployment must have the label app.kubernetes.io/managed-by=gitops.
In the repository's apps/web/deployment.yaml, change spec.replicas to 3, commit with the word replicas in the commit message, and apply to the cluster again. When you finish, there must be 2 or more commits, the replicas in both the repository and the cluster must be 3, and the working tree must be clean.
This time, do not touch the repository and create drift with kubectl scale deploy web -n gitops-lab --replicas=5. In that state, save the output of kubectl diff -n gitops-lab -f /root/gitops/repo/apps/web/ to /root/gitops/out/drift-diff.txt and that command's exit code to /root/gitops/out/drift-exit.txt. And write two or three lines in Korean in /root/gitops/out/drift-note.txt saying that the manually made change is reverted and disappears at the next apply.
Apply again from the repository's standard to remove the drift. Then run kubectl diff -n gitops-lab -f /root/gitops/repo/apps/web/ once more and save the exit code at that time to /root/gitops/out/clean-exit.txt. You must not reflect the manually made 5 in the repository — the repository's replicas is 3 and the working tree must be clean.
Create /root/gitops/sync.sh and give it execute permission. This script must (a) check the differences before applying with kubectl diff, (b) apply with kubectl apply, and (c) record which commit was applied with git rev-parse HEAD. As the result of running it, put three keys in /root/gitops/out/sync-report.json: repo_commit (the full hash of the current HEAD), drift (the boolean false), and applied (the number of objects applied, 2 or more).

Notes

It is GitOps only if the same commit gives the same result whenever it is applied. That is why you pin the image tag and deliberately state even fields that have defaults, such as replicas. A value you do not state is decided by the cluster default at the time of applying, and at that moment the repository cannot define the state.
kubectl diff gives exit code 1 if there is a difference and 0 if not. Unlike kubectl apply --dry-run=server, it compares the actual server merge result with the live state.
You must read the exit code with echo $? right after the command. If you chain with &&, the latter part does not run at all on failure, so it is safer to chain, as in 명령 > 파일; echo $? > 코드파일 (command, file, exit-code file), with ;.
You get the current HEAD hash with git -C /root/gitops/repo rev-parse HEAD. It must be the full hash, not the short hash.
Common mistake 1: reflecting in the repository, too, the manually made 5 in step 6. Then you have not solved the drift but promoted the incident to code. If that value is right, it must be formally reflected in a separate commit, and in this lab reverting is the right answer.
Common mistake 2: putting sync.sh or out/ inside the repository (/root/gitops/repo). If you don't commit, the working tree gets dirty, and if you commit, the HEAD hash written in the report immediately goes out of line. Keep the outputs under /root/gitops/, outside the repository.

Initialize a repository to hold the declaration

Create the directory /root/gitops/repo, run git init on it, and set user.name and user.email in that repository. Then create /root/gitops/repo/README.md and make the first commit (there must be at least one commit).

In GitOps, the commit history is the audit record. For the repository to record who changed it, user.name and user.email must be set in that repository, and there must be at least one commit.

Create a manifest directory per app

Copy /opt/lab/fixtures/gitops/seed/deployment.yaml and service.yaml to /root/gitops/repo/apps/web/. In metadata.labels, the Deployment must have app.kubernetes.io/managed-by: gitops; in spec.selector.matchLabels, the app.kubernetes.io/name must be web; and spec.replicas must be stated explicitly rather than left to the default (here it starts at 2). The container image must be a value with a pinned tag, like nginx:1.27 (:latest is treated as a failure). README.md must also still be there.

Copy the fixture and use it, but a repository is also read by people. And do not rely on defaults; state the count and the image tag explicitly — if the same commit yields different results, that repository cannot define the state. Also check that the selector matches the Pod labels.

Nail the declaration down with a commit

Track all three files, apps/web/deployment.yaml, apps/web/service.yaml, and README.md, with git add, and commit with a meaningful message of 10 characters or more. When you finish, the output of git status --porcelain must be empty.

If untracked files remain, the repository cannot define the cluster. The commit message is the sentence from which, months later, you will judge whether to revert by looking at just this one line.

Apply the repository's declaration to the cluster

Apply with kubectl apply -n gitops-lab -f /root/gitops/repo/apps/web/ and save the entire output to /root/gitops/out/apply.txt. After applying, the gitops-lab namespace must have a Deployment web and a Service web, and the Deployment must have the label app.kubernetes.io/managed-by=gitops.

You can apply a whole directory at once. The manifests do not state a namespace, so you must specify it in the command, and the result of applying must be kept in a file.

Reflect a change through a commit

In the repository's apps/web/deployment.yaml, change spec.replicas to 3, commit with the word replicas in the commit message, and apply to the cluster again. When you finish, there must be 2 or more commits, the replicas in both the repository and the cluster must be 3, and the working tree must be clean.

The order is the key. Fix the repository first, commit, and then apply. If you fix the cluster first, it is not a change but drift.

Change it by hand and create drift

This time, do not touch the repository and create drift with kubectl scale deploy web -n gitops-lab --replicas=5. In that state, save the output of kubectl diff -n gitops-lab -f /root/gitops/repo/apps/web/ to /root/gitops/out/drift-diff.txt and that command's exit code to /root/gitops/out/drift-exit.txt. And write two or three lines in Korean in /root/gitops/out/drift-note.txt saying that the manually made change is reverted and disappears at the next apply.

This time you deliberately change only the cluster without touching the repository. What the exit code of the command that shows the difference is is the key of this step, and you must read that code right after the command.

Revert to the repository's standard

Apply again from the repository's standard to remove the drift. Then run kubectl diff -n gitops-lab -f /root/gitops/repo/apps/web/ once more and save the exit code at that time to /root/gitops/out/clean-exit.txt. You must not reflect the manually made 5 in the repository — the repository's replicas is 3 and the working tree must be clean.

Sneaking a manually made value into the repository is promoting an incident to code. Assume the repository is right, fit the cluster to it, and then check the exit code for when there is no difference.

Create the synchronization script and the report

Create /root/gitops/sync.sh and give it execute permission. This script must (a) check the differences before applying with kubectl diff, (b) apply with kubectl apply, and (c) record which commit was applied with git rev-parse HEAD. As the result of running it, put three keys in /root/gitops/out/sync-report.json: repo_commit (the full hash of the current HEAD), drift (the boolean false), and applied (the number of objects applied, 2 or more).

You imitate in the shell what the controller does. Before applying, see what will change, apply, and record which commit was applied. If you put the script inside the repository, the hash written in the report goes out of line.