Green but Nobody Knows Why: Writing Health Rules Yourself

Goal

You write a Lua rule in resource.customizations.health.<그룹>_<종류> (group, then kind) of argocd-cm to split a CRD's health into the four slots Healthy, Progressing, Degraded, and Suspended, and you run a regression check by bundling samples and expectations into a table.

Why it matters

What people actually look at on the Argo CD screen is not synchronization but health. For built-in kinds such as Deployment, Service, and Job, the controller judges on its own, but a CRD has no built-in rule. If a resource made by an operator always shows the same color on the screen, it does not mean things are going well; it means there is no basis for judging. Writing rules yourself matters for two reasons. First, you can set up alerts only by telling apart what is still being made from what is already broken. Second, a rule is code that is forgotten once written, so you must keep the samples and the expectations together in the repository so that it does not quietly go wrong when a status field changes later.

Steps

Make /root/ga-health/argocd-cm.yaml an empty argocd-cm ConfigMap with data: {}, and put in /root/ga-health/deploy.yaml, in the ga-health namespace, a Deployment web — spec.replicas is 3 and in status you write observedGeneration: 1, replicas: 3, updatedReplicas: 2, readyReplicas: 1, and availableReplicas: 1. Save the output of argocd admin settings resource-overrides health /root/ga-health/deploy.yaml --argocd-cm-path /root/ga-health/argocd-cm.yaml to /root/ga-health/builtin.txt.
In /root/ga-health/w-ready.yaml, make, of example.com/v1, a Widget — the name is w-ready, the namespace is ga-health, spec.size is 3, and status.phase is Ready. Ask for this file's health with the empty argocd-cm from step 1 and save the output to /root/ga-health/nocustom.txt.
In /root/ga-health/argocd-cm-healthy.yaml, put the key resource.customizations.health.example.com_Widget and write Lua that, if status.phase is Ready, returns Healthy, and otherwise returns Progressing. Fill hs.message in both cases. Save the output of judging /root/ga-health/w-ready.yaml with this ConfigMap to /root/ga-health/healthy.txt.
Create /root/ga-health/w-failed.yaml — the name is w-failed, status.phase is Failed, and status.reason is DiskFull. /root/ga-health/argocd-cm-degraded.yaml adds a Failed branch to the step 3 rule and returns Degraded, but hs.message must contain that resource's status.reason value. Save the verdict output to /root/ga-health/degraded.txt.
Create /root/ga-health/w-building.yaml (status.phase is Building, name w-building) and /root/ga-health/w-nostatus.yaml (no status at all, name w-nostatus). Judge the two in turn with the step 4 ConfigMap and append the output to /root/ga-health/progressing.txt. Both must be Progressing.
Create /root/ga-health/w-paused.yaml — the name is w-paused, spec.paused is true, and status.phase is Ready. /root/ga-health/argocd-cm-full.yaml adds one more branch to the earlier rule so that if spec.paused is true, it returns Suspended before any other condition. Save the verdict output to /root/ga-health/suspended.txt.
Create /root/ga-health/argocd-cm-typo.yaml — the content is the same as step 6, but only the key name is written as resource.customizations.health.example.com_Widgets (the kind in the plural). Save the output of judging /root/ga-health/w-ready.yaml with this ConfigMap to /root/ga-health/typo.txt.
In /root/ga-health/health-matrix.tsv, write four or more lines of <샘플파일이름>\t<기대 STATUS> (sample file name, then expected STATUS) — Healthy, Degraded, Progressing, and Suspended must each appear at least once. /root/ga-health/check-health.sh reads this table, judges each sample with /root/ga-health/argocd-cm-full.yaml, prints OK … if it matches and MISMATCH … if not to standard output only, and must end with a non-zero code if even one line is wrong. Save that output to /root/ga-health/health-result.txt.

Notes

The Lua snippet receives the resource as obj and returns a table with hs.status and hs.message filled in.
The health state names are Healthy, Progressing, Degraded, Suspended, Missing, and Unknown.
The separator in the key name is not a dot but an underscore — resource.customizations.health.<그룹>_<종류>.
Common mistake: leaving out the moment when there is no status, so newly created resources start with a red light every time.
Common mistake: writing the kind name in the plural. No error occurs and it behaves as if there were no rule.
Reference: https://argo-cd.readthedocs.io/en/stable/operator-manual/health/

Look at the built-in verdict first

Make /root/ga-health/argocd-cm.yaml an empty argocd-cm ConfigMap with data: {}, and put in /root/ga-health/deploy.yaml, in the ga-health namespace, a Deployment web — spec.replicas is 3 and in status you write observedGeneration: 1, replicas: 3, updatedReplicas: 2, readyReplicas: 1, and availableReplicas: 1. Save the output of argocd admin settings resource-overrides health /root/ga-health/deploy.yaml --argocd-cm-path /root/ga-health/argocd-cm.yaml to /root/ga-health/builtin.txt.

A Deployment has a built-in health rule, so a verdict comes out even if argocd-cm is empty. It wants three and only one is ready, so predict what state it will come out as. The first line of the output is STATUS.

A CRD has no rule at all

In /root/ga-health/w-ready.yaml, make, of example.com/v1, a Widget — the name is w-ready, the namespace is ga-health, spec.size is 3, and status.phase is Ready. Ask for this file's health with the empty argocd-cm from step 1 and save the output to /root/ga-health/nocustom.txt.

Argo CD does not invent a verdict for a kind it does not know. On the screen, such resources usually look like a green light, but in reality it means "there is no basis for judging". Read the output sentence as it is.

Write the green-light condition in Lua

In /root/ga-health/argocd-cm-healthy.yaml, put the key resource.customizations.health.example.com_Widget and write Lua that, if status.phase is Ready, returns Healthy, and otherwise returns Progressing. Fill hs.message in both cases. Save the output of judging /root/ga-health/w-ready.yaml with this ConfigMap to /root/ga-health/healthy.txt.

The Lua snippet receives the resource as a global variable called obj and returns a table with hs.status and hs.message filled in. Resources with no status at all also come in, so check obj.status ~= nil first. The separator in the key name is not a dot but an underscore — <그룹>_<종류> (group, then kind).

A red light must come with its reason

Create /root/ga-health/w-failed.yaml — the name is w-failed, status.phase is Failed, and status.reason is DiskFull. /root/ga-health/argocd-cm-degraded.yaml adds a Failed branch to the step 3 rule and returns Degraded, but hs.message must contain that resource's status.reason value. Save the verdict output to /root/ga-health/degraded.txt.

If you write the message as a fixed string, you cannot tell from the screen which resource died and why. String concatenation in Lua is .., and since the value may be absent, put in a default like (obj.status.reason or "unknown").

Send an unknown state and no state to the same slot

Create /root/ga-health/w-building.yaml (status.phase is Building, name w-building) and /root/ga-health/w-nostatus.yaml (no status at all, name w-nostatus). Judge the two in turn with the step 4 ConfigMap and append the output to /root/ga-health/progressing.txt. Both must be Progressing.

What is easy to leave out when writing a rule is "the moment when the controller has not written status yet". If you give Degraded at that time, newly created resources start with a red light every time. That is the reason to make the default branch Progressing.

Something deliberately stopped is not a failure

Create /root/ga-health/w-paused.yaml — the name is w-paused, spec.paused is true, and status.phase is Ready. /root/ga-health/argocd-cm-full.yaml adds one more branch to the earlier rule so that if spec.paused is true, it returns Suspended before any other condition. Save the verdict output to /root/ga-health/suspended.txt.

Suspended is the fourth health state Argo CD knows — it is for things a person deliberately stopped, so it is a place where an alert must not ring. If you try changing the branch order, you immediately see why it must be at the very front (since phase is Ready, the green light matches first).

If you get a key name wrong by one character, the whole rule vanishes

Create /root/ga-health/argocd-cm-typo.yaml — the content is the same as step 6, but only the key name is written as resource.customizations.health.example.com_Widgets (the kind in the plural). Save the output of judging /root/ga-health/w-ready.yaml with this ConfigMap to /root/ga-health/typo.txt.

The kind in the key name must be letter for letter the same as the manifest's kind. If it is wrong, no error occurs and it simply behaves as if there were no rule — such a mistake can only be noticed by looking at the screen. Compare it with the output of step 2.

Bundle samples and expectations into a table and run a regression check on the rule

In /root/ga-health/health-matrix.tsv, write four or more lines of <샘플파일이름>\t<기대 STATUS> (sample file name, then expected STATUS) — Healthy, Degraded, Progressing, and Suspended must each appear at least once. /root/ga-health/check-health.sh reads this table, judges each sample with /root/ga-health/argocd-cm-full.yaml, prints OK … if it matches and MISMATCH … if not to standard output only, and must end with a non-zero code if even one line is wrong. Save that output to /root/ga-health/health-result.txt.

If you write only the sample file name and let the script attach the path, the table gets shorter. To pull out just the STATUS line, sed -n 's/^STATUS: //p' is convenient. If the script writes files itself, it overwrites the student's outputs when the grader runs it again, so send to standard output only.