Reading Argo CD and Argo Workflows: The Comparison Loop, the Worker Inside the Pod, and the Status Determination of a CronJob

Status: Draft (2026-10-17), partially measured. Baseline versions: argoproj/argo-cd v3.5.1 (commit 109ca7ca7113), argoproj/argo-workflows v4.1.5 (commit 6c80c7e56ae6). For Argo CD, because a small cluster runs the same version, we verified it on 2026-10-17 by reading only (pods, Application status, configuration keys, CronJob status fields). Argo Workflows was not present in that cluster, so we only read the code (not measured). Writing conventions: Measured data is the default. Unmeasured items are marked as [Not Measured], and unverified items are marked as [Unverified].

Table of Contents

• 1. One-Sentence Summary: The Comparison Loop and the Pod-Based Worker
• 2. Architecture Diagram ①: Argo CD's Comparison Loop
• 3. Code Map
• 4. Following the Core Path — Argo CD
  • 4.1 Refresh: When Do We Look Again?
  • 4.2 Rendering and Application: repo-server and sync
  • 4.3 Health Determination: Which Script is Chosen?
  • 4.4 CronJob Case Study: Where Documentation and Lua Differ
• 5. Following the Core Path — Argo Workflows
• 6. Author's Contributions and Further Reading
• 7. Measurement: Argo CD in a Small Cluster
• 8. Where to Start Reading
• 9. Good Places for First Contributions
• 10. Limitations
• References

1. One-Sentence Summary: The Comparison Loop and the Pod-Based Worker

Although they share the Argo name, the two projects have different characteristics.

• Argo CD is a thermostat. It continuously compares the target temperature written in Git (desired state) with the room temperature (actual state in the cluster), and if they differ, it reports it (OutOfSync) or adjusts it (sync). An endlessly running loop of this comparison is the core of the code.
• Argo Workflows is a factory work order. It receives a Workflow that groups steps by order and dependencies, launches a pod for each step, and passes files (artifacts) exchanged between steps via storage like S3.

2. Architecture Diagram ①: Argo CD's Comparison Loop

How to read the diagram:

• From top to bottom represents a single "refresh". The desired staterendered by the repo-server and the actual stateheld in the cluster cache are compared by the controller (②), and health is determined for each resource (③) and written to the Application.
• Synchronization is a separately running second queue (④, ⑤). Separating comparison and application into different queues ensures that time-consuming sync operations do not block the status updates of numerous apps.
• The cluster cache uses watches to hold all resources in memory without querying the API server every time. While comparisons are inexpensive, the controller's memory usage is high.
• The bottom box is the topic of Section 4.3 of this article: the order in which health check scripts are selected.

3. Code Map

In the v3 argo-cd repository, the formerly separate gitops-engine has been moved into the gitops-engine/ folder (ofgo.mod ). replace … => ./gitops-engineThe core of comparison, sync, and cluster cache resides here.

4. Following the Core Path — Argo CD

4.1 Refresh: When Do We Look Again?

The controller has two queues (controller/appcontroller.go:118 appRefreshQueue, line 121 appOperationQueue). appcontroller.go:955 Run ). Worker goroutines are attached to each of these two queues.

1. The refresh worker is appcontroller.go:1775 processAppRefreshQueueItem . First, it asks appcontroller.go:2105 needRefreshAppStatus "Is it really time to compare again?". If the user requested a refresh, a managed resource changed, or the set interval (default 120 seconds with up to 60 seconds jitter, argocd_application_controller.go:45) has passed, it compares.
2. Comparison is performed by controller/state.go:633 CompareAppState . This function fetches the desired state from the repo-server via gRPC using state.go:206 GetRepoObjs , retrieves the actual state from the cache using controller/cache/cache.go:729 GetManagedLiveObjs , and then compares each resource using gitops-engine's pkg/diff/diff.go:76 Diff .
3. Status is determined at the end of the same function (callingstate.go:1051 in setApplicationHealth ). We look at this in detail in Section 4.3.
4. If automatic sync is enabled, appcontroller.go:2322 autoSync writes an operation to the Application, whereupon the second queue picks up that app.

4.2 Rendering and Application: repo-server and sync

• The entry point of the repo-server is reposerver/repository/repository.go:631 GenerateManifest . It checks the cache first (repository.go:1016 getManifestCacheEntry), and if not found, checks out the repository to the specified commit, after which repository.go:1714 GenerateManifests uses repository.go:1921 GetAppSourceType to select one of helm, kustomize, plain, or plugins to render. The reason rendering is split into a separate serviceis security and resources. Since helm, kustomize, and plugins execute third-party repository contents, they are separated from the controller which holds cluster credentials, and heavy rendering is handled by scaling out only the repo-server.
• Application is handled when controller/sync.go:102 SyncAppState calls gitops-engine's pkg/sync/sync_context.go:478 Sync . Here, sync_context.go:954 getSyncTasks queues resources by phases (PreSync, Sync, PostSync hooks) and wave numbers, proceeding to the next wave only when the current wave becomes healthy. Determining "healthy" is again the health check of Section 4.3.
• The actual state cache is initialized when gitops-engine/pkg/cache/cluster.go:1094 sync receives a list of all API resources, and maintained thereafter by cluster.go:792 watchEvents receiving subsequent changes via watch.

4.3 Health Determination: Which Script is Chosen?

The health of a single resource is determined by gitops-engine's pkg/health/health.go:70 GetResourceHealth . If it is being deleted, it returns Progressing; otherwise, it first checks for overrides , and only falls back to built-in Go checks (health.go:104 GetHealthCheckFunc, Deployment, Pod, Job, etc.) if there is no answer.

The "override" on the Argo CD side is Lua. Simplifying util/lua/lua.go:169 GetHealthScript 's order yields the following:

key := GetConfigMapKey(obj.GroupVersionKind())
if script, ok := vm.ResourceOverrides[key]; ok && script.HealthLua != "" {
    return script.HealthLua, script.UseOpenLibs, nil   // argocd-cm 에 사용자가 적은 것
}
// 와일드카드 항목 → 내장 Lua 파일 → 내장 와일드카드 순으로 찾는다
builtInScript, err := vm.getPredefinedLuaScripts(key, healthScriptFile)

Source: argoproj/argo-cd v3.5.1 util/lua/lua.go(GetHealthScript (summary), Apache-2.0.

Built-in Lua resides in the resource_customizations/<그룹>/<종류>/health.lua file and is bundled directly into the binary via resource_customizations/embed.go 's go:embed . health_test.yaml and testdata/ in the same folder are tests for that script. The easiest way to contribute health checks for new CRDsis to add a Lua script and test data to this folder. No Go knowledge is required, and tests are run by util/lua/health_test.go finding all health_test.yaml files in the folder.

Resource health determined this way is aggregated by controller/health.go:20 setApplicationHealth to form the overall app health. The worst status becomes the app's status, and resources with argocd.argoproj.io/ignore-healthcheck: "true" annotations are excluded from aggregation (line 41). In v3, per-resource health is stored not in the Application object but in Redis's resource tree , because --persist-resource-health defaults to false (argocd_application_controller.go:293). This is a choice to avoid etcd writes becoming heavy as Applications grow.

4.4 CronJob Case Study: Where Documentation and Lua Differ

The built-in Lua for CronJob is a resource_customizations/batch/CronJob/health.lua 56-line script. The decision branches into five paths:

The v3.5.1 documentation (docs/operator-manual/health.md) states in the CronJob section that "if the last Job fails, it is Degraded" and "if the last Job is running, it is Progressing," and gives "suspended CronJob" as an example of Suspended status. Comparing this with the code reveals all three are incorrect: if a failure occurs and the next Job runs, it returns to Healthy; while running, it is Healthy rather than Progressing; and a suspended CronJob is Healthy rather than Suspended.

5. Following the Core Path — Argo Workflows

1. workflow/controller/controller.go:337 Run The informer and worker are started by controller.go:922 processNextItem , and
2. workflow/controller/operator.go:212 operate extracts a Workflow to create an operator. operator.go:1216 podReconciliation represents a single reconciliation: first, operator.go:2235 executeTemplate reflects the status of already launched pods into node status, and steps.go:42 executeSteps, dag.go:278 executeDAG, operator.go:3435 executeContainer descends from the entry template to determine what to do next. Depending on the template type, it branches into status.nodes . Since it walks from the beginning every time while skipping completed nodes, even if the controller restarts, as long as the Workflow's
3. is available, it can resume. workflowpod.go:341 createWorkflowPod Container steps are created as pods by main . Alongside the user container init(workflowpod.go:1092 newInitContainers, it attaches wait and main , and wraps argoexec emissary 's command withcmd/argoexec/commands/emissary.go:46). In v4, a method where a single container handles both init and wait tasks without an init supervisor has also been introduced (workflowpod.go:1127 newSupervisorContainer, initlessPod configuration).
4. Inside the pod, wait waits for main to finish and uploads the output file to the storage via(workflow/executor/executor.go:1355 Wait), executor.go:449 SaveArtifacts ).
5. The storage driver is chosen by workflow/artifacts/artifacts.go:29 NewDriver based on the type specified in the artifact. S3 is at the very front (line 38), and the access key is populated by reading a Secret on the spot. MinIO connects using the S3 driver.
6. If no storage is specified in the artifact, workflow/artifactrepositories/artifactrepositories.go:38 Resolve looks it up in the order of the workflow namespace's default ConfigMap, followed by the controller configuration's default storage.

Why it is divided this way. There is only one controller in the cluster, but there can be thousands of steps. If the controller does not handle heavy file transfers and instead delegates them to argoexec inside the step pod, only the step pods scale out when transfer volume increases. Instead, there is a cost of attaching a sidecar container to each pod, and v4's supervisor approach can be seen as an attempt to reduce that cost (design intent could not be verified in documentation, unconfirmed).

6. Author's contributions and related links

• argoproj/argo-cd issue #30209 → #30243. The initial issue (#30209) was regarding the sentence "Progressing when running" in 4.4, which was fixed and closed by another contributor's #30212. The remaining three items (Healthy when the next Job runs after failure, Healthy if it has never succeeded, and stopped CronJobs not being Suspended) were raised in a new issue, #30243. The basis for all of these is the table in 4.4, lines 3, 15, 27, and 35 of health.lua .
• argoproj/argo-workflows#17246(Under review). Since the MinIO installation section in the artifact storage documentation currently uses an unavailable image and excessively large default values, this PR updates it to use the quick-start MinIO configuration already present in the repository (manifests/quick-start/base/minio). Item 5 in Section 5 (the path to connect to MinIO using the S3 driver) is what the documentation describes. There was also a case where documentation testing (cspell) did not recognize the image name, so the word was added to the dictionary.

7. Measurement: Argo CD in a small cluster

2026-10-17, read-only commands were used. Application names and repository URLs are omitted.

In Application's status.resources , there were 44 CronJobs, and their health columns were all empty (due to the aforementioned appTree ). Therefore, the actual status fields of the 44 CronJobs (spec.suspend, status.lastScheduleTime, lastSuccessfulTime, active) were read, and classified by translating the rules in the 4.4 table into Python . The inputs are measured values, and the judgments are calculations translating Lua.

Out of 44, 27 were stopped CronJobs. According to the documentation, things that should appear as Suspended in the UI are all mixed in as Healthy, and you cannot find a single one by filtering for "Suspended". This classification demonstrates issue #30243 point 3 with actual numbers.

8. Where to read from

1. resource_customizations/batch/CronJob/ — 56 lines of Lua and test data. The smallest entry point.
2. util/lua/lua.go:169 GetHealthScript → gitops-engine/pkg/health/health.go:70.
3. controller/appcontroller.go:1775 processAppRefreshQueueItem — An entire single refresh cycle.
4. controller/state.go:633 CompareAppState — Long, but gathers all the ingredients for comparison.
5. For Workflows, grasping workflow/controller/operator.go:212 operate and workflowpod.go:341 createWorkflowPod gives you the big picture.
6. Finally, workflow/artifacts/artifacts.go:29 — Pick a single driver (s3) and follow it through to the end.

9. Good places to make your first contribution

• resource_customizations Add Lua and tests to . There are still many CRDs without status evaluation. health.lua + health_test.yaml + testdata/*.yaml Three things are enough, and you can verify it with just go test ./util/lua/ without running the full test suite (4.3).
• CronJob documentation cleanup (#30243). Aligning the documentation with the table in 4.4, along with guiding how to override (resource.customizations.health.batch_CronJob) and ignore-healthcheck comments. If the maintainers accept the issue, it can immediately lead to a PR.
• Old Lua syntax. 8 built-in Lua scripts use table.getn (including line 27 of CronJob). Since the executor supports it, there is no problem right now, but cleaning it up by changing to # length operators has a small scope. However, you will need to submit a test proving the behavior is identical for it to be accepted.
• Workflows documentation. The supervisor approach without init in 4.1.5 is well-explained in the code, but it was difficult to find this setting in the user documentation (documentation folder was not downloaded this time, unconfirmed).
• Rules: The Argo project has a Generative AI policy (argoproj/argoproj community/genai.md), and the Argo Workflows PR template has an ### AI section to disclose AI usage. Both repositories require DCO sign-off (Signed-off-by), and Argo CD has PR title format checking (.github/pr-title-checker-config.json) and a checklist item stating "PRs without tests are unlikely to be merged".

10. Limitations

• Argo Workflows is not present in the small cluster, so there are no actual measurements. The supervisor approach, artifact plugins, and archiving (persist/) were only skimmed through the code.
• Argo CD sharding, ApplicationSet generators, notifications, server-side diffs, and the hydrator (commitserver) were not covered.
• CronJob classification is a calculation translating Lua. The actual evaluations of the resource tree via the Argo CD API were not read (not done because login is required, unmeasured).

Titi's word. If you just write "Healthy!" for a player who is resting, when the manager asks "Who is resting?", not a single person shows up! Even though 27 people were resting!

References

All access dates are 2026-10-17.

• argoproj/argo-cd v3.5.1: controller/, gitops-engine/pkg/, reposerver/repository/repository.go, util/lua/lua.go, resource_customizations/batch/CronJob/health.lua, docs/operator-manual/health.md
• argoproj/argo-workflows v4.1.5: workflow/controller/, workflow/executor/executor.go, workflow/artifacts/artifacts.go, workflow/artifactrepositories/artifactrepositories.go
• Public issues and PRs: argoproj/argo-cd#30209·#30212·#30243·#24428·#24429, argoproj/argo-workflows#17246
• The quoted code snippets are from the aforementioned repositories and are all under the Apache License 2.0. They have been shortened and included in the article.