We loosened the rule and the tests stayed green that day

Goal

You build from scratch a test harness to attach to one policy. You write input samples, expected decisions, and expected messages in a table file, write a runner that reads that table and runs admission through a server dry-run, and confirm by hand how three things (a message change, a scope drift, and a loosened rule) show up as red in the test.

Why it matters

The most common way a policy breaks is not an error but silence. If the match scope is off, the policy lets everything through without a sound, and not a single red line shows on the dashboard. So a policy that is running well and a dead policy cannot be told apart on the surface. A second force overlaps here - a policy is fixed only in the direction of loosening each time someone is blocked. A test is the only device that blocks both at once. But a test that collects only passing samples stays green even if you delete the policy entirely, so it protects nothing. That is why this lab goes as far as pinning deny samples and expected messages as a contract, and moving the state where the test itself has become meaningless (no samples, no deny expectations) out to a third exit code that is neither success nor failure.

Steps

Work in /root/poltest (export KUBECONFIG=/root/.kube/config, kubectl config use-context kwok-lab). In /root/poltest/policy.yaml, write a ValidatingAdmissionPolicy require-min-replicas of admissionregistration.k8s.io/v1 — matchConstraints.resourceRules catches CREATE and UPDATE on deployments of apps group v1, the validation is object.spec.replicas >= 2, and the messageExpression builds a sentence containing the fragment below the minimum 2 replicas. In /root/poltest/binding.yaml, write a binding require-min-replicas-bind — policyName is require-min-replicas, validationActions is ["Deny"], and matchResources.namespaceSelector.matchLabels is policy-test: "yes". Apply both, then create the namespace poltest and attach the label policy-test=yes. Also create two samples — /root/poltest/cases/ok-two.yaml is a Deployment ok-two with replicas: 2, and /root/poltest/cases/bad-one.yaml is a Deployment bad-one with replicas: 1. Finally, send only ok-two.yaml as a server dry-run and save its output to /root/poltest/01-allow.txt.
Write the golden case table in /root/poltest/cases.txt. One line is one sample and the fields are split by | — four fields: 이름 | 매니페스트 경로 | allow 또는 deny | 기대 메시지 조각 (name, manifest path, allow or deny, and expected message fragment). Write the manifest path relative to the directory where the table file is, and lines starting with # and blank lines are comments. For a line whose expectation is allow, write - in the message field. For now, put in two lines — allow-two lists cases/ok-two.yaml as allow, and deny-one lists cases/bad-one.yaml as deny with the message fragment below the minimum 2 replicas, copied as it is from the actual denial message.
Create /root/poltest/run-cases.sh. It takes the table file as an argument (if none, cases.txt next to the script), and for each line of the table sends that manifest with kubectl create -n poltest -f <파일> --dry-run=server (with a file name in place of the placeholder). If the command succeeds, the actual decision is allow, and if it fails, deny. If it equals the expectation, it prints one line PASS <이름>, and if not, one line FAIL <이름> <까닭> (with the name, and for FAIL also the reason). Resolve the manifest path relative to the directory where the table file is. On the last line, print total=<수> pass=<수> fail=<수> (each placeholder being a count), and if there is even one failure, end with a nonzero value. When you have built it, run it with bash run-cases.sh and confirm that both lines come out PASS.
Create two more deny samples — /root/poltest/cases/bad-zero.yaml is a Deployment bad-zero with replicas: 0, and /root/poltest/cases/bad-default.yaml is a Deployment bad-default that does not write the replicas field at all. Add both to cases.txt under the names deny-zero and deny-default, and keep the expected message fragment the same below the minimum 2 replicas. The table is now four lines, and three of them expect deny. bash run-cases.sh must again end with everything PASS.
Fix run-cases.sh so that a line whose expectation is deny becomes PASS only if the denial message contains the expected fragment (if the fragment is - or empty, it does not look at the message). Then create /root/poltest/policy-msgdrift.yaml — the same name and rule as policy.yaml, with only the messageExpression changed to a different sentence, and that sentence must not contain below the minimum 2 replicas. Apply it, run bash run-cases.sh, and save the whole output to /root/poltest/05-drift.txt (at least three FAIL lines must come out). Finally, reapply policy.yaml to restore the original message, and confirm that the test again ends with everything PASS.
In /root/poltest/cases/sts-one.yaml, write a StatefulSet sts-one — replicas: 1 and serviceName: sts-one. Add a deny-sts-one line to cases.txt with a deny expectation (the same message fragment below the minimum 2 replicas), run bash run-cases.sh, and save the whole output to /root/poltest/06-falsepass.txt — that sample comes out as FAIL. The policy now looks only at deployments, so it does not judge this request at all, and because there is no denial, it looks like a pass. Now widen resources in policy.yaml to ["deployments", "statefulsets"], reapply it, and confirm that the test ends with all five lines PASS.
Create /root/poltest/policy-loose.yaml — the same name, resources, and message as policy.yaml, with only the minimum lowered from 2 to 1 (the situation where someone asked "please let us also run a single-instance one" and a condition was loosened by one notch). Apply it, run bash run-cases.sh, and save the whole output to /root/poltest/07-regression.txt — at least two FAIL lines come out. Then reapply policy.yaml to restore it and confirm that the test again ends with everything PASS.
Finally fix run-cases.sh so that it can be used as a pipeline gate. Extend the summary line to total=<수> pass=<수> fail=<수> deny=<수> result=<PASS|FAIL|INVALID> (where deny is the number of lines whose expectation is deny), and also leave the entire printed output as it is in report.txt in the directory where the table file is. The exit code convention has three values — 0 if everything passes, 1 if even one is out of line with its expectation, and 2 if the table has no samples at all or no deny expectations at all (result is INVALID). After fixing, run bash run-cases.sh to leave /root/poltest/report.txt, and also run it once with a table that has the deny expectations removed to confirm that 2 comes out.

Notes

The working directory is /root/poltest, and you connect to the cluster with export KUBECONFIG=/root/.kube/config (context kwok-lab).
A server dry-run is kubectl create -n poltest -f <파일> --dry-run=server (with a file name in place of the placeholder). It skips only storage and passes admission as it is, so the denial message really comes back and no object remains. The denial message comes out on standard error, so add 2>&1 when capturing it to a file.
Even if you apply a policy, it is not reflected immediately. It takes about a second for the API server to compile the new expression, so right after changing it, wait briefly until the changed decision shows, and then run the test. This makes it easy to mistake it as "I fixed it but it's the same."
Common mistake one - a command called inside the loop reads the whole table file, so the loop runs only one line and ends. Attach the redirect, as in kubectl ... </dev/null.
Common mistake two - the denial message comes out on standard error, not standard output. When capturing to a file or comparing, do not forget 2>&1.
ValidatingAdmissionPolicy: https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
CEL syntax and where it is used: https://kubernetes.io/docs/reference/using-api/cel/
What a dry-run passes through: https://kubernetes.io/docs/reference/using-api/api-concepts/
Admission and sideEffects: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

I ran just one passing sample and believed the policy was alive

Work in /root/poltest (export KUBECONFIG=/root/.kube/config, kubectl config use-context kwok-lab). In /root/poltest/policy.yaml, write a ValidatingAdmissionPolicy require-min-replicas of admissionregistration.k8s.io/v1 — matchConstraints.resourceRules catches CREATE and UPDATE on deployments of apps group v1, the validation is object.spec.replicas >= 2, and the messageExpression builds a sentence containing the fragment below the minimum 2 replicas. In /root/poltest/binding.yaml, write a binding require-min-replicas-bind — policyName is require-min-replicas, validationActions is ["Deny"], and matchResources.namespaceSelector.matchLabels is policy-test: "yes". Apply both, then create the namespace poltest and attach the label policy-test=yes. Also create two samples — /root/poltest/cases/ok-two.yaml is a Deployment ok-two with replicas: 2, and /root/poltest/cases/bad-one.yaml is a Deployment bad-one with replicas: 1. Finally, send only ok-two.yaml as a server dry-run and save its output to /root/poltest/01-allow.txt.

The policy decides only the method of judging, and the binding decides where to apply it. So both must go up for a denial to happen. messageExpression is a CEL expression, so you concatenate strings and must convert numbers with string(...). What this step wants to show is the last line — a record of running only the samples that should pass cannot be evidence that the policy is alive. A server dry-run is kubectl create -n poltest -f <파일> --dry-run=server (with a file name in place of the placeholder). It skips only storage and passes admission as it is, so the denial message really comes back and no object remains. The denial message comes out on standard error, so add 2>&1 when capturing it to a file.

Once the expectations were written in a table, what was missing showed

Write the golden case table in /root/poltest/cases.txt. One line is one sample and the fields are split by | — four fields: 이름 | 매니페스트 경로 | allow 또는 deny | 기대 메시지 조각 (name, manifest path, allow or deny, and expected message fragment). Write the manifest path relative to the directory where the table file is, and lines starting with # and blank lines are comments. For a line whose expectation is allow, write - in the message field. For now, put in two lines — allow-two lists cases/ok-two.yaml as allow, and deny-one lists cases/bad-one.yaml as deny with the message fragment below the minimum 2 replicas, copied as it is from the actual denial message.

Do not make up the expected message; cut it from the actual denial output. First send bad-one once as a server dry-run and see what sentence comes back. Name samples so that you can tell what differs without opening the file. If you keep pairs made by copying a passing sample and changing only one place, the cause can be read from the name when a test breaks.

Build a runner that reads the table and runs it

Create /root/poltest/run-cases.sh. It takes the table file as an argument (if none, cases.txt next to the script), and for each line of the table sends that manifest with kubectl create -n poltest -f <파일> --dry-run=server (with a file name in place of the placeholder). If the command succeeds, the actual decision is allow, and if it fails, deny. If it equals the expectation, it prints one line PASS <이름>, and if not, one line FAIL <이름> <까닭> (with the name, and for FAIL also the reason). Resolve the manifest path relative to the directory where the table file is. On the last line, print total=<수> pass=<수> fail=<수> (each placeholder being a count), and if there is even one failure, end with a nonzero value. When you have built it, run it with bash run-cases.sh and confirm that both lines come out PASS.

When reading the table, use the form while IFS='|' read -r ... done < "$table". Add </dev/null so that the command called inside the loop does not swallow the table. You must trim the whitespace around fields for comparisons to match. The exit code is that of the last command as it goes out, so explicitly exit at the end. The grader runs this script with a different table - a runner that does not read the table and just prints fixed answers gets caught there.

Add samples that look plausible but break the rule

Create two more deny samples — /root/poltest/cases/bad-zero.yaml is a Deployment bad-zero with replicas: 0, and /root/poltest/cases/bad-default.yaml is a Deployment bad-default that does not write the replicas field at all. Add both to cases.txt under the names deny-zero and deny-default, and keep the expected message fragment the same below the minimum 2 replicas. The table is now four lines, and three of them expect deny. bash run-cases.sh must again end with everything PASS.

A Deployment that does not write replicas has its default filled in before admission. A server dry-run shows the policy the object that has passed through even that defaulting, so leaving it unwritten and writing 1 get the same decision. This is a place an offline engine easily misses. It is good to make deny samples by changing only one place from a passing sample.

I only polished the message and the test went red

Fix run-cases.sh so that a line whose expectation is deny becomes PASS only if the denial message contains the expected fragment (if the fragment is - or empty, it does not look at the message). Then create /root/poltest/policy-msgdrift.yaml — the same name and rule as policy.yaml, with only the messageExpression changed to a different sentence, and that sentence must not contain below the minimum 2 replicas. Apply it, run bash run-cases.sh, and save the whole output to /root/poltest/05-drift.txt (at least three FAIL lines must come out). Finally, reapply policy.yaml to restore the original message, and confirm that the test again ends with everything PASS.

The denial message is the only channel through which the policy speaks to developers. A pipeline that grabbed that sentence to make alerts or tickets silently stops on the day the wording changes. If you pin the message as an expected value, the person refactoring finds out on the spot. In a shell, the sturdiest way to check a substring is case "$out" in *"$frag"*). Even if you apply a policy, it is not reflected immediately - wait until the changed decision shows, and then run the test.

A resource the policy did not even look at was read as a pass

In /root/poltest/cases/sts-one.yaml, write a StatefulSet sts-one — replicas: 1 and serviceName: sts-one. Add a deny-sts-one line to cases.txt with a deny expectation (the same message fragment below the minimum 2 replicas), run bash run-cases.sh, and save the whole output to /root/poltest/06-falsepass.txt — that sample comes out as FAIL. The policy now looks only at deployments, so it does not judge this request at all, and because there is no denial, it looks like a pass. Now widen resources in policy.yaml to ["deployments", "statefulsets"], reapply it, and confirm that the test ends with all five lines PASS.

If the policy does not look at that resource, the decision is not a pass but "not applicable." But admission returns the two with the same face - the request succeeded. That is why putting in the table a sample that must be denied is the only evidence that the match scope has not drifted. A StatefulSet needs serviceName in addition to selector and template. Even after you widen the scope, it takes about a second to be reflected.

I loosened the rule by one notch and the test noticed first

Create /root/poltest/policy-loose.yaml — the same name, resources, and message as policy.yaml, with only the minimum lowered from 2 to 1 (the situation where someone asked "please let us also run a single-instance one" and a condition was loosened by one notch). Apply it, run bash run-cases.sh, and save the whole output to /root/poltest/07-regression.txt — at least two FAIL lines come out. Then reapply policy.yaml to restore it and confirm that the test again ends with everything PASS.

A policy is almost always fixed only in the direction of loosening. Each change looks reasonable at the time, and after half a year nobody knows what it was originally meant to block. If the test has a deny sample, red lights up on the day the rule is loosened. Also watch that the message wording stays the same at this point, so the sample with replicas 0 remains PASS - this is the moment the rule and the sentence go their separate ways.

Set the exit code so an empty test does not go green

Finally fix run-cases.sh so that it can be used as a pipeline gate. Extend the summary line to total=<수> pass=<수> fail=<수> deny=<수> result=<PASS|FAIL|INVALID> (where deny is the number of lines whose expectation is deny), and also leave the entire printed output as it is in report.txt in the directory where the table file is. The exit code convention has three values — 0 if everything passes, 1 if even one is out of line with its expectation, and 2 if the table has no samples at all or no deny expectations at all (result is INVALID). After fixing, run bash run-cases.sh to leave /root/poltest/report.txt, and also run it once with a table that has the deny expectations removed to confirm that 2 comes out.

A test that collects only passing samples stays green even if you delete the policy entirely. That is why "a state in which the test protects nothing" is moved out to a third exit code that is neither success nor failure. The gate treats only 0 as a pass and blocks both 1 and 2. If you put report.txt next to the table file, running with a different table does not overwrite the original result. The summary just adds two fields to the previous step's format, so leave the shape of the PASS and FAIL lines as it is.