Git accepted it, but Argo could not apply it

Goal

You see for yourself what the reconciler goes through when a broken configuration enters the state store, and you move CI's validation to the repository entrance so that the same mistake cannot reach main. You confirm, through RBAC verdicts, the division of labor in which CI writes only to Git and only the reconciler writes to the cluster.

Why it matters

GitOps does not eliminate CI/CD; it moves the boundary. CI creates and validates the desired state and records it in Git, and deployment is done by the reconciler inside the cluster pulling from Git. So there is no longer any reason to give CI write credentials for the production cluster. Instead, whatever enters Git immediately becomes a deployment command. Git does not know the meaning of YAML, so it accepts a commit like replicas: two, and the reconciler keeps failing as it tries to apply it. Treating configuration as code (Configuration as Code) means that, like code, it is allowed into the source only after passing validation, and this lab builds that validation with a validation-only identity and a server-side dry-run.

Steps

Create the bare repository /srv/bare/ci.git, clone it to /root/cgoa-ci/repo, commit a Deployment web (no namespace, replicas 2, label app: web, container web, image nginx:1.27-alpine) to deploy/web.yaml, and push to main. In /root/cgoa-ci/app.yaml, write and apply an Application ci-web (argocd namespace, project default, repository git://gitd.gitsrv.svc.cluster.local:9418/ci.git with main and the deploy path, target namespace ci-web, automated sync with prune and selfHeal, CreateNamespace=true), and check that it is Synced and Healthy.
Change replicas in deploy/web.yaml to the string two, commit and push, and hard refresh ci-web. About 20 seconds later, read the Application status and write into /root/cgoa-ci/broken.json commit (the SHA of the broken commit), sync_status (status.sync.status), operation_message (status.operationState.message), and live_replicas (the actual spec.replicas of the Deployment in the ci-web namespace, as a number). When the observation is finished, undo the broken commit with git revert and push. If, even after the revert, status.operationState is Running (retrying) on the broken commit, run argocd app terminate-op ci-web --core with /root/cgoa-ci/kubeconfig (a copy of the k3s kubeconfig, current namespace argocd) to end that operation. ci-web must be Synced and Healthy on the new main, and no operation retrying the broken commit may remain.
Create the namespaces ci and ci-dryrun, and create a ServiceAccount validator in ci. In ci-dryrun, create a Role dryrun-deployments (only get, create, and patch on deployments in the apps group) and a RoleBinding validator-dryrun that binds it to ci:validator. Using that account's token (8 hours), create the kubeconfig /root/cgoa-ci/ci-kubeconfig (server address the same as in the k3s kubeconfig, current namespace ci-dryrun). This account must be able to create Deployments only in ci-dryrun and must not be able to write anything to ci-web.
Create /root/cgoa-ci/validate.sh <디렉터리> (the placeholder is the directory) as an executable script. It must try applying the manifests in that directory to the ci-dryrun namespace with a server-side dry-run using /root/cgoa-ci/ci-kubeconfig, and exit with a non-zero value if even one is rejected. Do not rely on the KUBECONFIG environment variable; specify that kubeconfig inside the script. It must not create actual objects.
Create /srv/bare/ci.git/hooks/pre-receive as an executable hook. For every push that updates refs/heads/main, extract the deploy directory of the new commit into a temporary directory, validate it with /root/cgoa-ci/validate.sh, and reject the push if it fails. Reject deleting main and let other refs through.
In /root/cgoa-ci/repo, make a commit that changes the replicas key in deploy/web.yaml to the typo replica, try to push it, and save the entire rejected output to /root/cgoa-ci/blocked.txt. Then reset local main to remote main (git reset --hard origin/main). Remote main must stay as it was, and ci-web must remain Synced and Healthy.
Create /root/cgoa-ci/bump.sh <태그> (the placeholder is the tag) as an executable script. After aligning /root/cgoa-ci/repo with remote main, change the image in deploy/web.yaml to nginx:<태그> (the placeholder is the tag), commit with the message ci: web nginx:<태그> (the placeholder is the tag), and push. Do not use kubectl. Run bump.sh 1.28-alpine, and after a hard refresh, check that ci-web is Synced and Healthy on that commit and that the Deployment image has changed to nginx:1.28-alpine.
In /root/cgoa-ci/report.json, write ci_writes (git), cd_writes (cluster), ci_can_write_prod (whether ci:validator can patch deployments in ci-web, a boolean), gate (pre-receive), deployed_revision (the current status.sync.revision of ci-web), and broken_reached_git (whether the broken commit from step 2 remains in main's history, a boolean).

Notes

The VM has k3s, Argo CD v3.5.2, and a git daemon (gitd.gitsrv). /srv/bare/<이름>.git (the placeholder is the name) appears as git://gitd.gitsrv.svc.cluster.local:9418/<이름>.git (the placeholder is the name).
To make it re-read immediately, use kubectl -n argocd annotate app ci-web argocd.argoproj.io/refresh=hard --overwrite.
Permission verdict: kubectl auth can-i create deployments.apps -n ci-dryrun --as system:serviceaccount:ci:validator
Common mistake: validating with kubectl apply --dry-run=client. The client does not check the schema as strictly as the server, so even the commit from step 2 passes.
Common mistake: leaving main broken before you put the hook in place. Be sure to revert it in step 2 before moving on.
Common mistake: moving on after seeing only Synced after the revert. Even if the comparison result is Synced, while a failed operation is retrying the broken commit, the new commit from step 7 is not synced.
Core-mode commands look up the Argo CD configuration in the kubeconfig's current namespace. Copy the k3s kubeconfig and change it with kubectl config --kubeconfig <사본> set-context --current --namespace=argocd (the placeholder is the copy).
OpenGitOps principles · Argo CD automated sync · kubectl apply --dry-run · githooks

A baseline deployed from Git

Create the bare repository /srv/bare/ci.git, clone it to /root/cgoa-ci/repo, commit a Deployment web (no namespace, replicas 2, label app: web, container web, image nginx:1.27-alpine) to deploy/web.yaml, and push to main. In /root/cgoa-ci/app.yaml, write and apply an Application ci-web (argocd namespace, project default, repository git://gitd.gitsrv.svc.cluster.local:9418/ci.git with main and the deploy path, target namespace ci-web, automated sync with prune and selfHeal, CreateNamespace=true), and check that it is Synced and Healthy.

The gitd service exports the bare repositories under /srv/bare over the git protocol. After the first sync, both Pods must be Ready for the app to be Healthy.

The repository accepted it, but Argo failed to apply it

Change replicas in deploy/web.yaml to the string two, commit and push, and hard refresh ci-web. About 20 seconds later, read the Application status and write into /root/cgoa-ci/broken.json commit (the SHA of the broken commit), sync_status (status.sync.status), operation_message (status.operationState.message), and live_replicas (the actual spec.replicas of the Deployment in the ci-web namespace, as a number). When the observation is finished, undo the broken commit with git revert and push. If, even after the revert, status.operationState is Running (retrying) on the broken commit, run argocd app terminate-op ci-web --core with /root/cgoa-ci/kubeconfig (a copy of the k3s kubeconfig, current namespace argocd) to end that operation. ci-web must be Synced and Healthy on the new main, and no operation retrying the broken commit may remain.

Git does not know whether a value is a string or a number. The API server rejects it only at the moment the reconciler tries to apply it. Look at what stays on the cluster in the meantime. When an automated sync operation fails, it retries a set number of times with the same revision, and during that time the sync of a new commit does not start. Check status.operationState.operation.sync.revision.

A CI identity that cannot be used on production

Create the namespaces ci and ci-dryrun, and create a ServiceAccount validator in ci. In ci-dryrun, create a Role dryrun-deployments (only get, create, and patch on deployments in the apps group) and a RoleBinding validator-dryrun that binds it to ci:validator. Using that account's token (8 hours), create the kubeconfig /root/cgoa-ci/ci-kubeconfig (server address the same as in the k3s kubeconfig, current namespace ci-dryrun). This account must be able to create Deployments only in ci-dryrun and must not be able to write anything to ci-web.

Get the token with kubectl create token and assemble a new file with kubectl config --kubeconfig <파일> set-cluster/set-credentials/set-context (the placeholder is the file name). For the CA, just copy the certificate-authority-data from the k3s kubeconfig as is. Check the verdict with kubectl auth can-i --as system:serviceaccount:<ns>:<이름> (the placeholder after the namespace is the name).

Reject in CI, first, the manifest the server would reject

Create /root/cgoa-ci/validate.sh <디렉터리> (the placeholder is the directory) as an executable script. It must try applying the manifests in that directory to the ci-dryrun namespace with a server-side dry-run using /root/cgoa-ci/ci-kubeconfig, and exit with a non-zero value if even one is rejected. Do not rely on the KUBECONFIG environment variable; specify that kubeconfig inside the script. It must not create actual objects.

A client-side dry-run lets through both a string replicas and a misspelled field. Choose the mode that goes through the API server's schema validation but does not persist anything. A failure in the middle of a pipe may not show up with set -e alone.

Put validation at the repository entrance

Create /srv/bare/ci.git/hooks/pre-receive as an executable hook. For every push that updates refs/heads/main, extract the deploy directory of the new commit into a temporary directory, validate it with /root/cgoa-ci/validate.sh, and reject the push if it fails. Reject deleting main and let other refs through.

The hook runs in a bare repository, so there is no working tree. You can extract one directory of a specific commit with git archive <커밋> deploy | tar -x -C <임시> (the first placeholder is the commit and the second is the temporary directory). If the new SHA is 40 zeros, it is a deletion.

The same mistake no longer reaches main

In /root/cgoa-ci/repo, make a commit that changes the replicas key in deploy/web.yaml to the typo replica, try to push it, and save the entire rejected output to /root/cgoa-ci/blocked.txt. Then reset local main to remote main (git reset --hard origin/main). Remote main must stay as it was, and ci-web must remain Synced and Healthy.

The rejected commit is not on the remote, so you only need to undo it locally. This time the reconciler does not even get a chance to experience a failure.

CI writes to Git and Argo deploys

Create /root/cgoa-ci/bump.sh <태그> (the placeholder is the tag) as an executable script. After aligning /root/cgoa-ci/repo with remote main, change the image in deploy/web.yaml to nginx:<태그> (the placeholder is the tag), commit with the message ci: web nginx:<태그> (the placeholder is the tag), and push. Do not use kubectl. Run bump.sh 1.28-alpine, and after a hard refresh, check that ci-web is Synced and Healthy on that commit and that the Deployment image has changed to nginx:1.28-alpine.

This commit, too, must pass the hook from step 5 to get into main. The script changes only the desired state in Git and leaves the applying to the reconciler.

Report who writes where

In /root/cgoa-ci/report.json, write ci_writes (git), cd_writes (cluster), ci_can_write_prod (whether ci:validator can patch deployments in ci-web, a boolean), gate (pre-receive), deployed_revision (the current status.sync.revision of ci-web), and broken_reached_git (whether the broken commit from step 2 remains in main's history, a boolean).

Do not guess the two booleans; write the results you confirmed with kubectl auth can-i and git merge-base --is-ancestor.