The Deployment was created, but not a single pod exists

Goal

On a real Kyverno 1.19.1, you check what the authoring tools of Kyverno policies (preconditions, apiCall context, variable defaults, autogen, JSON patch, foreach, CEL, and background scanning) reject and change in real requests.

Why it matters

A policy's syntax being correct is different from it judging as intended. match selects only by resource kind and location, so to turn a rule on and off depending on the request content you need preconditions, and if you need information that is not in the request you need a context. If a variable does not resolve, the rule raises an error instead of a verdict, and in Enforce that error looks like a rejection, so you end up looking for the cause in the wrong place. Pod rules are by default automatically generated as rules for controllers, and if you turn that off, the Deployment appears to succeed and only the Pods quietly fail to appear. Mutation rules must be able to put in a different value per request or change only part of a list, and resources that were already in the cluster do not go through admission again, so they show up only through background scanning.

Steps

Create the namespace kca-pre and write and apply the ClusterPolicy limits-for-prod (background false) in /root/kca-write/01-limits.yaml. The rule memory-limit matches Pods in kca-pre, applies only to requests whose label tier is prod via preconditions (treating it as an empty string if the label is absent), and requires resources.limits.memory on every container (Enforce). Do not use a label selector in match.
Create the namespaces kca-prod (label env=prod) and kca-dev (label env=dev), and apply the ClusterPolicy team-in-prod-ns (background false) in /root/kca-write/02-team.yaml. The rule team-when-prod reads the context nsenv with apiCall from /api/v1/namespaces/{{request.namespace}} for Pods in the two namespaces (label env, none if absent) and rejects when env is prod and there is no team label (Enforce). Then temporarily change the env of kca-dev to prod, send a Pod without labels with a server dry-run, save the entire output to /root/kca-write/flip.txt, and set env back to dev.
In the namespace kca-var, apply the ClusterPolicy no-platform-team (background false, rule not-platform). It is a deny condition that rejects when the team label is platform, and at first you write the key as {{ request.object.metadata.labels.team }} without a default. Send a Pod without labels with a server dry-run and save the entire rejection output to /root/kca-write/missing.txt. Then add a default (an empty string) to the key in /root/kca-write/03-no-platform.yaml and apply again, so that a Pod without labels is accepted and team=platform is rejected.
Create the namespaces kca-auto and kca-noauto, and write and apply in /root/kca-write/04-probes.yaml the ClusterPolicies probes-auto and probes-noauto, each requiring a readinessProbe on Pods in its namespace (Enforce, background false, rule need-probe). Attach the annotation pod-policies.kyverno.io/autogen-controllers: none only to probes-noauto. Then run kubectl create deployment web --image=nginx:1.27-alpine in both namespaces and save the entire output on the kca-auto side to /root/kca-write/auto.txt.
Create the namespace kca-mut and write the ClusterPolicy requested-by (rule annotate-user) in /root/kca-write/05-requested-by.yaml. To Pods in kca-mut, add the annotation kca.io/requested-by with an add operation of patchesJson6902, and the value is {{request.userInfo.username}}. First apply without writing spec.background, save the entire rejection output to /root/kca-write/bg-denied.txt, and then apply with background set to false. Existing annotations must be preserved.
Write and apply the ClusterPolicy pull-policy (rule latest-always) in /root/kca-write/06-pull-policy.yaml. For Pods in kca-mut, use foreach to look, per container, at whether the image ends with :latest using element-level preconditions, and change only those containers to imagePullPolicy: Always with patchStrategicMerge. The imagePullPolicy of containers with a pinned tag must stay as it is.
Create the namespace kca-cel and apply the policies.kyverno.io/v1 ValidatingPolicy no-priv-esc in /root/kca-write/07-no-priv-esc.yaml. validationActions is [Deny], and matchConstraints is CREATE and UPDATE of core v1 pods with the namespaceSelector kubernetes.io/metadata.name: kca-cel. Put in the CEL variable escalating the list of names of containers whose securityContext.allowPrivilegeEscalation is not explicitly false, pass only if that list is empty, and show those names joined by commas in the rejection message (messageExpression).
In the namespace kca-bg, first create the Pod old-nolabel without labels and the Pod old-label with the label team=a, and then apply the ClusterPolicy audit-team (background true, Audit, rule team, requiring the team label on Pods in kca-bg) in /root/kca-write/08-audit-team.yaml. Wait until the results for both Pods appear in the PolicyReport, and then write in /root/kca-write/report.json the lists of Pod names for each result under the keys fail and pass.

Notes

Inside the VM there are k3s and Kyverno v1.19.1. In this version, kyverno.io/v1 ClusterPolicy emits a deprecation warning but works.
Testing a policy without saving: kubectl -n <ns> run t --image=nginx:1.27-alpine --dry-run=server (use -o yaml for the mutated result).
Checking policy readiness: kubectl get clusterpolicy <이름> -o jsonpath='{.status.conditionStatus.ready}' (the placeholder is the name), and for ValidatingPolicy use .status.conditionStatus.ready.
Common mistake: sending a request right after applying a policy. It takes a few seconds for the webhook to take effect.
Common mistake: stacking policies from several steps in one namespace. This lab separates the namespaces per step.
Preconditions · External Data Sources(apiCall) · Variables · Auto-Gen Rules · Mutate Rules · ValidatingPolicy · Reporting

Only prod Pods are required to have a limit

Create the namespace kca-pre and write and apply the ClusterPolicy limits-for-prod (background false) in /root/kca-write/01-limits.yaml. The rule memory-limit matches Pods in kca-pre, applies only to requests whose label tier is prod via preconditions (treating it as an empty string if the label is absent), and requires resources.limits.memory on every container (Enforce). Do not use a label selector in match.

The key of preconditions is a JMESPath variable. What happens when the variable does not resolve for a request without the label, you will see in step 3. The grader sends Pods with no tier, with dev, and with prod by server dry-run.

A single namespace label turns the rule on

Create the namespaces kca-prod (label env=prod) and kca-dev (label env=dev), and apply the ClusterPolicy team-in-prod-ns (background false) in /root/kca-write/02-team.yaml. The rule team-when-prod reads the context nsenv with apiCall from /api/v1/namespaces/{{request.namespace}} for Pods in the two namespaces (label env, none if absent) and rejects when env is prod and there is no team label (Enforce). Then temporarily change the env of kca-dev to prod, send a Pod without labels with a server dry-run, save the entire output to /root/kca-write/flip.txt, and set env back to dev.

If you hard-code the namespace name into the policy, the verdict cannot change without editing the policy. apiCall reads the API server's current value on every request. The dry-run is kubectl run ... --dry-run=server.

A Pod without the label was rejected for the wrong reason

In the namespace kca-var, apply the ClusterPolicy no-platform-team (background false, rule not-platform). It is a deny condition that rejects when the team label is platform, and at first you write the key as {{ request.object.metadata.labels.team }} without a default. Send a Pod without labels with a server dry-run and save the entire rejection output to /root/kca-write/missing.txt. Then add a default (an empty string) to the key in /root/kca-write/03-no-platform.yaml and apply again, so that a Pod without labels is accepted and team=platform is rejected.

If you read a missing key with JMESPath, variable substitution fails, and Kyverno treats that rule as an error that could not reach a verdict. Read the wording of the output to see what an error in an Enforce rule does to the request. The || operator supplies a default.

The Deployment was created but there are no Pods at all

Create the namespaces kca-auto and kca-noauto, and write and apply in /root/kca-write/04-probes.yaml the ClusterPolicies probes-auto and probes-noauto, each requiring a readinessProbe on Pods in its namespace (Enforce, background false, rule need-probe). Attach the annotation pod-policies.kyverno.io/autogen-controllers: none only to probes-noauto. Then run kubectl create deployment web --image=nginx:1.27-alpine in both namespaces and save the entire output on the kca-auto side to /root/kca-write/auto.txt.

Kyverno automatically expands a Pod rule into rules for resources that have a Pod template, such as Deployments and ReplicaSets. If it does not expand, the Deployment passes, and the rejection happens when the ReplicaSet controller creates the Pods. Look at the conditions and events of kubectl describe rs.

Stamp the requester's name onto the Pod

Create the namespace kca-mut and write the ClusterPolicy requested-by (rule annotate-user) in /root/kca-write/05-requested-by.yaml. To Pods in kca-mut, add the annotation kca.io/requested-by with an add operation of patchesJson6902, and the value is {{request.userInfo.username}}. First apply without writing spec.background, save the entire rejection output to /root/kca-write/bg-denied.txt, and then apply with background set to false. Existing annotations must be preserved.

In a JSON pointer, write / as ~1. The requester information exists only in the admission request, not in background processing that rescans already stored resources. If you add -o jsonpath to the server dry-run, you can see the mutated result.

Make only latest-tag containers pull fresh every time

Write and apply the ClusterPolicy pull-policy (rule latest-always) in /root/kca-write/06-pull-policy.yaml. For Pods in kca-mut, use foreach to look, per container, at whether the image ends with :latest using element-level preconditions, and change only those containers to imagePullPolicy: Always with patchStrategicMerge. The imagePullPolicy of containers with a pinned tag must stay as it is.

Inside foreach the current element is element. To change only a specific container in a list, match by name with the conditional anchor (name). JMESPath has an ends_with function.

One line of CEL blocked even the Deployment

Create the namespace kca-cel and apply the policies.kyverno.io/v1 ValidatingPolicy no-priv-esc in /root/kca-write/07-no-priv-esc.yaml. validationActions is [Deny], and matchConstraints is CREATE and UPDATE of core v1 pods with the namespaceSelector kubernetes.io/metadata.name: kca-cel. Put in the CEL variable escalating the list of names of containers whose securityContext.allowPrivilegeEscalation is not explicitly false, pass only if that list is empty, and show those names joined by commas in the rejection message (messageExpression).

Build the list with CEL's filter and map, and check a missing field first with has(). The ValidatingPolicy in this version also automatically generates Pod rules for resources that have a Pod template (status.autogen). The grader also sends a Deployment by dry-run.

Even Pods that existed before the policy showed up in the report

In the namespace kca-bg, first create the Pod old-nolabel without labels and the Pod old-label with the label team=a, and then apply the ClusterPolicy audit-team (background true, Audit, rule team, requiring the team label on Pods in kca-bg) in /root/kca-write/08-audit-team.yaml. Wait until the results for both Pods appear in the PolicyReport, and then write in /root/kca-write/report.json the lists of Pod names for each result under the keys fail and pass.

Audit only records without blocking requests. Already stored resources do not go through admission again, so they are evaluated only if background is on. The results gather in the namespace's PolicyReport (one per resource).