We pushed, and nothing happened

Goal

You measure, within the same commit flow, the path by which Argo CD periodically pulls Git and the path by which a push webhook refreshes it immediately. You see what stops when you turn polling off, what a webhook checks before it is accepted, and how the two are layered in production.

Why it matters

A GitOps reconciler pulls the desired state by itself. There are two triggers for pulling. One is polling, which re-reads the repository at a set interval, and the other is an event, in which the repository announces "a push just happened." With polling alone, a commit waits as long as the interval before it is reflected, and as the number of repositories grows, that interval becomes load on the Git server. With events alone it is fast, but if one notification is lost, the commit silently stalls. Also, a webhook that anyone can send becomes an attack surface for waking the reconciler, so the sender must be verified with a signature. This lab also confirms that an event only tells the reconciler to "read now" and is not a deployment command. What to deploy is still decided by Git.

Steps

Create the bare repository /srv/bare/evt.git, clone it to /root/cgoa-evt/repo, commit a ConfigMap signal (no namespace, data rev: r1) to app/signal.yaml, and push to main. In /root/cgoa-evt/app.yaml, write and apply an Application evt (argocd namespace, project default, repoURL http://githttp.gitsrv.svc.cluster.local/cgi-bin/git/evt.git, targetRevision main, path app, target namespace evt, automated sync with prune and selfHeal, CreateNamespace=true), and check that it is Synced.
Put timeout.reconciliation: 30s and timeout.reconciliation.jitter: 0s into the ConfigMap argocd-cm, and restart both the argocd-application-controller StatefulSet and the argocd-repo-server Deployment. In the log of the new controller Pod, find the line that contains appResyncPeriod and save it as is to /root/cgoa-evt/interval.txt, and when the restart is finished, hard refresh evt once.
Change rev in app/signal.yaml to r2, commit and push, and, without using refresh or a webhook, wait until status.sync.revision of evt becomes that commit. In /root/cgoa-evt/poll.json, write commit, pushed_at (Unix seconds right after the push), and synced_at (Unix seconds when you saw the revision change) as numbers.
Change timeout.reconciliation in argocd-cm to 0s and restart the controller and repo-server. Then change rev to r3, push, wait at least 40 seconds without making any request, and in /root/cgoa-evt/nopoll.json write commit (the r3 commit), pushed_at, checked_at (Unix seconds when you checked after waiting), and revision_at_check (the status.sync.revision of evt at that time).
Send a GitHub push event to /api/webhook of the argocd-server service. The headers are X-GitHub-Event: push and Content-Type: application/json, and the body is JSON containing ref refs/heads/main, the r3 commit in after, and http://githttp.gitsrv.svc.cluster.local/cgi-bin/git/evt in repository.html_url. Wait until evt changes to the r3 commit, and write http_status (a number), sent_at, and synced_at in /root/cgoa-evt/webhook.json.
Save a value created with openssl rand -hex 16 to /root/cgoa-evt/webhook-secret and put it in the webhook.github.secret key of the Secret argocd-secret. Change rev to r4 and push, then first send a push event without a signature and wait 15 seconds. Next, sign the same body file with HMAC-SHA256 using that secret value, send it with the header X-Hub-Signature-256: sha256=<hex>, and wait for the sync. In /root/cgoa-evt/secret.json, write commit (r4), unsigned_status, signed_status (a number), revision_after_unsigned (the revision of evt 15 seconds after the unsigned request), and signed_synced_at.
A webhook can be lost, so bring polling back. Change timeout.reconciliation in argocd-cm to 120s, delete the timeout.reconciliation.jitter key, and restart the controller and repo-server. Save the appResyncPeriod line of the new controller log to /root/cgoa-evt/restored.txt, and hard refresh evt once. Leave the webhook secret as it is.
In /root/cgoa-evt/report.json, write poll_seconds (synced_at minus pushed_at measured in step 3), nopoll_synced (whether r3 was deployed at the time of the check in step 4, a boolean), webhook_seconds (synced_at minus sent_at in step 5), unsigned_accepted (whether the unsigned event caused a deployment, a boolean), trigger (webhook), and safety_net (timeout.reconciliation).

Notes

The VM has k3s, Argo CD v3.5.2, and a smart HTTP git server (githttp.gitsrv). /srv/bare/<이름>.git (the placeholder is the name) appears as http://githttp.gitsrv.svc.cluster.local/cgi-bin/git/<이름>.git (the placeholder is the name).
Restart: after kubectl -n argocd rollout restart statefulset argocd-application-controller and kubectl -n argocd rollout restart deployment argocd-repo-server, wait for each with rollout status.
Common mistake: restarting only the controller. Even if it compares every 30 seconds, the repo-server returns the branch's old commit from its cache, so the new commit is not visible for several minutes.
Measure time with date +%s, and read the revision with kubectl -n argocd get app evt -o jsonpath='{.status.sync.revision}'.
Common mistake: attaching the refresh annotation in steps 3 and 4. At that moment, what you are measuring is no longer polling.
Common mistake: the string you computed the signature on and the body you actually sent differing by even one byte. Even a single newline changes the HMAC.
Argo CD webhook configuration · argocd-cm example (timeout.reconciliation) · Validating GitHub webhook deliveries · OpenGitOps principles

An app that tracks a smart HTTP repository

Create the bare repository /srv/bare/evt.git, clone it to /root/cgoa-evt/repo, commit a ConfigMap signal (no namespace, data rev: r1) to app/signal.yaml, and push to main. In /root/cgoa-evt/app.yaml, write and apply an Application evt (argocd namespace, project default, repoURL http://githttp.gitsrv.svc.cluster.local/cgi-bin/git/evt.git, targetRevision main, path app, target namespace evt, automated sync with prune and selfHeal, CreateNamespace=true), and check that it is Synced.

In this lab the repository address uses http:// rather than git://. The githttp server exports the repositories under /srv/bare with git-http-backend. First check with git ls-remote that it is visible.

Shorten the pull interval to 30 seconds

Put timeout.reconciliation: 30s and timeout.reconciliation.jitter: 0s into the ConfigMap argocd-cm, and restart both the argocd-application-controller StatefulSet and the argocd-repo-server Deployment. In the log of the new controller Pod, find the line that contains appResyncPeriod and save it as is to /root/cgoa-evt/interval.txt, and when the restart is finished, hard refresh evt once.

Two components read this setting when they start. For the controller it is the interval at which it compares apps again, and for the repo-server it is how long it caches the commit a branch points to. If you restart only the controller, then even though it compares every 30 seconds, the repo-server keeps returning the old commit. The cache lives in Redis, so entries stored before the restart with the old expiry (3 minutes by default) remain, and the new expiry applies only after a hard refresh makes it read afresh once. That line in the log shows the duration in Go's time.Duration notation.

The time you waited without pressing anything

Change rev in app/signal.yaml to r2, commit and push, and, without using refresh or a webhook, wait until status.sync.revision of evt becomes that commit. In /root/cgoa-evt/poll.json, write commit, pushed_at (Unix seconds right after the push), and synced_at (Unix seconds when you saw the revision change) as numbers.

Measure the time with date +%s. While waiting, read only the revision at 2-second intervals. If you request a refresh with annotate, this measurement is no longer polling.

With polling off, the commit stalled

Change timeout.reconciliation in argocd-cm to 0s and restart the controller and repo-server. Then change rev to r3, push, wait at least 40 seconds without making any request, and in /root/cgoa-evt/nopoll.json write commit (the r3 commit), pushed_at, checked_at (Unix seconds when you checked after waiting), and revision_at_check (the status.sync.revision of evt at that time).

0 means turning periodic refresh off. See how appResyncPeriod changed in the restarted controller's log. A cluster event in which a managed object changes and re-reading Git are separate things.

How many seconds does one push event take?

Send a GitHub push event to /api/webhook of the argocd-server service. The headers are X-GitHub-Event: push and Content-Type: application/json, and the body is JSON containing ref refs/heads/main, the r3 commit in after, and http://githttp.gitsrv.svc.cluster.local/cgi-bin/git/evt in repository.html_url. Wait until evt changes to the r3 commit, and write http_status (a number), sent_at, and synced_at in /root/cgoa-evt/webhook.json.

Send it over https to the service's ClusterIP, and since the certificate is self-signed, use curl -k. Argo CD compares the repository address in the event with each Application's repoURL and refreshes only the matching apps. If you put the changed file paths in commits[].modified, the path comparison passes too.

An unsigned event is turned away at the door

Save a value created with openssl rand -hex 16 to /root/cgoa-evt/webhook-secret and put it in the webhook.github.secret key of the Secret argocd-secret. Change rev to r4 and push, then first send a push event without a signature and wait 15 seconds. Next, sign the same body file with HMAC-SHA256 using that secret value, send it with the header X-Hub-Signature-256: sha256=<hex>, and wait for the sync. In /root/cgoa-evt/secret.json, write commit (r4), unsigned_status, signed_status (a number), revision_after_unsigned (the revision of evt 15 seconds after the unsigned request), and signed_synced_at.

The signature must be computed over the exact bytes you send. Make the body into a file, send it with curl --data-binary @파일 (the placeholder is the file), and feed the same file to openssl dgst -sha256 -hmac. argocd-server re-reads Secret changes by itself.

Trust the webhook, but bring polling back as a safety net

A webhook can be lost, so bring polling back. Change timeout.reconciliation in argocd-cm to 120s, delete the timeout.reconciliation.jitter key, and restart the controller and repo-server. Save the appResyncPeriod line of the new controller log to /root/cgoa-evt/restored.txt, and hard refresh evt once. Leave the webhook secret as it is.

To delete a key in a merge patch, send its value as null. If you delete jitter, the default is used. Check how both the interval and the jitter are displayed in the log line. While the interval was 0, the repo-server stored the commit a branch points to with the default cache expiry (24 hours). Even if you restore the interval, that entry remains, so polling sees the new commit again only after a hard refresh makes it read afresh once.

Report the roles of pulling and events

In /root/cgoa-evt/report.json, write poll_seconds (synced_at minus pushed_at measured in step 3), nopoll_synced (whether r3 was deployed at the time of the check in step 4, a boolean), webhook_seconds (synced_at minus sent_at in step 5), unsigned_accepted (whether the unsigned event caused a deployment, a boolean), trigger (webhook), and safety_net (timeout.reconciliation).

Compute it from the numbers in the JSON files you left in the earlier steps. The grader cross-checks the same files against the Argo CD sync history.