The label rule was loosened and nobody noticed

Goal

With the Kyverno CLI's apply, test, and jp, you judge what a policy rejects, what it changes, and what it creates before putting it on a cluster, and then pin that judgment down with a test suite and a CI script.

Why it matters

Admission policies are quiet when they are wrong. A mistake that loosens a validation rule simply blocks no requests and raises no error, and a single anchor in a mutation rule changes whether it touches resources that already have a value. Discovering that difference in a production cluster is already too late. kyverno test writes down the expectations first, "this resource passes, that resource fails, this resource changes like this," and compares them with the engine's actual verdicts. So when the person who edited a policy makes an unintended loosening, the test lights the red light first. You fill ConfigMaps and API call results that are not in the cluster with a values file, and run the JMESPath of a condition separately with kyverno jp, verifying the policy repository like code.

Steps

In /root/kca-cli/validate/policy.yaml, write the ClusterPolicy require-team. The rule check-team selects Pods and requires that metadata.labels.team be a non-empty value, and its failureAction is Enforce. In /root/kca-cli/validate/resources.yaml, put the Pod good (label team=pay), the Pod bad with no label, and the Deployment web whose Pod template has no team label (all in namespace default). Run kyverno apply with --policy-report attached and save the standard output to /root/kca-cli/apply-report.yaml.
In /root/kca-cli/validate/kyverno-test.yaml, write a Test (apiVersion cli.kyverno.io/v1alpha1). policies is policy.yaml and resources is resources.yaml, and in results you expect pass for good, fail for bad, and fail for the Deployment web under the autogenerated rule name that appeared in the step 1 report. kyverno test /root/kca-cli/validate must pass all 3 cases.
Copy /root/kca-cli/validate wholesale to /root/kca-cli/regressed, and in the copy's policy.yaml change only the team key to the equality anchor =(team) (a common mistake that lets things pass when the label is missing). Save the entire output of kyverno test /root/kca-cli/regressed --remove-color to /root/kca-cli/regression.txt and also check the exit code. Do not look only at the summary line and the exit code; find the rows where the expectation and the actual result diverge in the RESULT and REASON columns of the table. The original validate must still pass as it is.
In /root/kca-cli/mutate/, put the ClusterPolicy add-managed-by (rule add-label, using an add anchor that adds the label managed-by: kyverno to Pods only when it is absent), resources.yaml (the Pod plain with no label, and the Pod owned that has the label managed-by: helm), patched.yaml with the expected appearance after mutation (the Pod with the label attached to plain), and kyverno-test.yaml. The test compares plain against patchedResources and expects pass, expects skip for owned, and kyverno test /root/kca-cli/mutate must pass.
In /root/kca-cli/generate/, put the ClusterPolicy ns-quota (rule gen-quota, which, when a Namespace is created, creates a ResourceQuota default-quota in that namespace with spec.hard.pods: "10", synchronize false), resources.yaml (the Namespace team-a), the expected generated resource generated.yaml, and kyverno-test.yaml (a generatedResource comparison for team-a, pass), and make kyverno test /root/kca-cli/generate pass.
In /root/kca-cli/context/, put the ClusterPolicy allowed-registries. The rule check-registry reads the ConfigMap platform/registry-config with the context reg for Pods, and with foreach rejects (Enforce) any container image that does not match one of reg.data.allowed (a comma-separated list of patterns). In resources.yaml, put the Pod internal, which has a single image registry.lab/web:1.0, and the Pod external, which adds a container sidecar with docker.io/busybox:1.36 to it. Since there is no cluster, fill reg.data.allowed with registry.lab/* using values.yaml (kind Values), connect it through variables in kyverno-test.yaml, and make internal pass and external fail pass.
In /root/kca-cli/jp/query.txt, write one JMESPath expression. It must take a Pod object as input and return the list of names of containers whose image does not start with registry.lab/. In /root/kca-cli/jp/external.yaml, put just the one Pod external from step 6, and save the output of kyverno jp query -i jp/external.yaml -q jp/query.txt to /root/kca-cli/jp/result.json.
Make /root/kca-cli/ci.sh an executable script. It takes the root directory as its first argument (/root/kca-cli if absent), runs kyverno test on each of the four directories validate, mutate, generate, and context under it, and must end with a non-zero value if even one fails. As you saw in step 3, even if the exit code is 0, if there is a row in the output table where the expectation and the actual result diverge (REASON is Want ...), it must be treated as a failure. It does not run regressed. The grader breaks the copy's policies and tests in several ways to see whether the script fails.

Notes

This image contains kyverno CLI 1.13.2. Check with kyverno version. This lab does not use a cluster.
To see only the test results, use kyverno test <디렉터리> --fail-only (the placeholder is the directory), and for detailed reasons use --detailed-results.
Common mistake: writing the expectation for a Deployment under the Pod rule name. Autogenerated rules have a different name.
Common mistake: trusting only the Test Summary line and the exit code. This version leaves rows where a fail expectation flipped to pass only in the table (you confirm this yourself in step 3).
Common mistake: wrapping a JMESPath string in backticks. Inside backticks is JSON, so unquoted letters become a syntax error.
Kyverno CLI test · Kyverno CLI apply · Kyverno CLI jp · Autogen rules

Ask the engine first, before putting the policy up

In /root/kca-cli/validate/policy.yaml, write the ClusterPolicy require-team. The rule check-team selects Pods and requires that metadata.labels.team be a non-empty value, and its failureAction is Enforce. In /root/kca-cli/validate/resources.yaml, put the Pod good (label team=pay), the Pod bad with no label, and the Deployment web whose Pod template has no team label (all in namespace default). Run kyverno apply with --policy-report attached and save the standard output to /root/kca-cli/apply-report.yaml.

The form is kyverno apply <정책> --resource <리소스> (the placeholders are the policy and the resource). Check in the report whether a Pod-targeted rule also applies to a Deployment and, if it does, what name the rule is reported under. If there is a violation, the exit code is not 0.

A test that writes down the expected results first

In /root/kca-cli/validate/kyverno-test.yaml, write a Test (apiVersion cli.kyverno.io/v1alpha1). policies is policy.yaml and resources is resources.yaml, and in results you expect pass for good, fail for bad, and fail for the Deployment web under the autogenerated rule name that appeared in the step 1 report. kyverno test /root/kca-cli/validate must pass all 3 cases.

For each results item you write policy, rule, resources, kind, and result. A rule autogenerated from a Pod rule has a prefix attached in front of its name.

The label rule was loosened, and the test noticed first

Copy /root/kca-cli/validate wholesale to /root/kca-cli/regressed, and in the copy's policy.yaml change only the team key to the equality anchor =(team) (a common mistake that lets things pass when the label is missing). Save the entire output of kyverno test /root/kca-cli/regressed --remove-color to /root/kca-cli/regression.txt and also check the exit code. Do not look only at the summary line and the exit code; find the rows where the expectation and the actual result diverge in the RESULT and REASON columns of the table. The original validate must still pass as it is.

=(key) checks the value only when the key is present. See which resource has a label map but no team. The CLI in this image (1.13.2), when a resource that was expected to fail is judged as pass, writes that mismatch in the REASON and yet emits the summary and exit code as passing (measured). You only find out whether a test tool counts every mismatch as a failure by deliberately breaking it like this.

A mutate test that also compares the changed result

In /root/kca-cli/mutate/, put the ClusterPolicy add-managed-by (rule add-label, using an add anchor that adds the label managed-by: kyverno to Pods only when it is absent), resources.yaml (the Pod plain with no label, and the Pod owned that has the label managed-by: helm), patched.yaml with the expected appearance after mutation (the Pod with the label attached to plain), and kyverno-test.yaml. The test compares plain against patchedResources and expects pass, expects skip for owned, and kyverno test /root/kca-cli/mutate must pass.

+(key): value inserts only when the key is absent. On a resource that already has the label the rule changes nothing, so the result may not be pass. patchedResources is a file name.

A generate test that compares the object to be created in advance

In /root/kca-cli/generate/, put the ClusterPolicy ns-quota (rule gen-quota, which, when a Namespace is created, creates a ResourceQuota default-quota in that namespace with spec.hard.pods: "10", synchronize false), resources.yaml (the Namespace team-a), the expected generated resource generated.yaml, and kyverno-test.yaml (a generatedResource comparison for team-a, pass), and make kyverno test /root/kca-cli/generate pass.

For the namespace of the object to generate, use the name of the request object as a variable. The expected generated resource must include even metadata.namespace for the comparison to match.

Fill the ConfigMap context without a cluster

In /root/kca-cli/context/, put the ClusterPolicy allowed-registries. The rule check-registry reads the ConfigMap platform/registry-config with the context reg for Pods, and with foreach rejects (Enforce) any container image that does not match one of reg.data.allowed (a comma-separated list of patterns). In resources.yaml, put the Pod internal, which has a single image registry.lab/web:1.0, and the Pod external, which adds a container sidecar with docker.io/busybox:1.36 to it. Since there is no cluster, fill reg.data.allowed with registry.lab/* using values.yaml (kind Values), connect it through variables in kyverno-test.yaml, and make internal pass and external fail pass.

The CLI cannot look up a context's configMap, so you give the variable values with per-rule values. There is a JMESPath function that turns a comma-separated string into a list. AnyNotIn understands wildcards in the value list.

Run the condition expression outside the policy first

In /root/kca-cli/jp/query.txt, write one JMESPath expression. It must take a Pod object as input and return the list of names of containers whose image does not start with registry.lab/. In /root/kca-cli/jp/external.yaml, put just the one Pod external from step 6, and save the output of kyverno jp query -i jp/external.yaml -q jp/query.txt to /root/kca-cli/jp/result.json.

Use the filter expression [?조건] (the placeholder is the condition) and the string function starts_with. In JMESPath, string literals use single quotes (backticks are JSON literals). The grader also runs the expression against other Pods.

The CI gatekeeper of the policy repository

Make /root/kca-cli/ci.sh an executable script. It takes the root directory as its first argument (/root/kca-cli if absent), runs kyverno test on each of the four directories validate, mutate, generate, and context under it, and must end with a non-zero value if even one fails. As you saw in step 3, even if the exit code is 0, if there is a row in the output table where the expectation and the actual result diverge (REASON is Want ...), it must be treated as a failure. It does not run regressed. The grader breaks the copy's policies and tests in several ways to see whether the script fails.

Capture the output in a variable or a file and check both the exit code and the mismatch phrase. Whether you stop at the first failure in the loop or run everything and then finish after collecting, only the resulting code needs to be correct. If color codes get mixed in, string searches miss, so use --remove-color.