Why generate Fails Quietly

In one line

mutate modifies the object in admission and returns it. generate creates nothing in admission; it leaves an UpdateRequest, and then the background controller does the actual creation. Because of this difference, generate fails in its hardest-to-notice form: admission succeeds and only the resource does not appear.

Why this was needed

The reason mutate exists is that checking alone wears people out. If you enforce "attach a team label to every Pod" with validate alone, the same line has to go into hundreds of manifests, and the teams that miss it get their deployments blocked. If a value applies to the whole organization without exception, it is better for the system to insert it than to have people write it.

generate solves a different problem. There are facts you can know only at the moment a namespace is created. If you want to put a default-deny NetworkPolicy into a new namespace, you have to react to the namespace creation event. With GitOps it is hard to express the point in time "after the namespace appears," and if you leave it to people, they forget.

How it works

mutate has two syntaxes. patchStrategicMerge is Kubernetes' strategic merge patch, which merges arrays by a name key. It suits work that "keeps the existing and layers on top," such as adding one more sidecar to the container list. patchesJson6902 is the RFC 6902 JSON Patch, which points to the exact location with op/path/value. The path uses slash notation, and a point that often trips people up in practice is that if a slash appears inside a key, it must be escaped as ~1. To add a kca.io/owner annotation, the path becomes /metadata/annotations/kca.io~1owner. foreach mutate applies a patch to each element of a collection.

generate's syntax splits into data (you write the values directly in the policy) and clone (you copy a resource from another namespace). You cannot use the two together. On top of this comes synchronize. If true, the generated resource follows when the source changes, and if you edit the generated resource by hand, it is reverted.

The problem is that this convenience is not free. synchronize increases watching and writing by the number of target namespaces. In a cluster with five namespaces you feel nothing, but with hundreds it becomes a constant load on the background controller. And this controller is installed with least privilege. When you start to generate something that is not a standard resource, the person writing it has to add permissions for that resource, and without the permissions, admission succeeds and only the resource quietly does not appear.

That is why the diagnostic order is fixed. When the generated resource does not show up, do not start by looking into the policy YAML; look at the UpdateRequest first. If kubectl -n kyverno get updaterequests is empty, match did not hit, and if there is a request but no resource, it is a problem of the background controller's permissions or behavior. This one branch cuts the scope of the problem in half. You check permissions with kubectl auth can-i <동사> <리소스> --as system:serviceaccount:kyverno:kyverno-background-controller (the placeholders are the verb and the resource). And you must also remember that if you check for the generated resource right after creating the namespace, it may not be there yet. This is not a bug but a design.

Finally, let us point out the operational trade-off of mutate. Values inserted by mutate are not visible in Git. If you put them in Helm values, they are reviewed and can be rolled back by tag, but if you insert them with mutate, they are visible only in the cluster. Half a year later you end up in a state where nobody knows why this annotation is attached, and the fact that the manifest and the actual object differ keeps fighting with the GitOps tool's drift detection. The criterion is to leave only values that must be enforced across the whole organization to mutate, and to put values that teams must be able to change in the chart.

What it looks like in the field

The author's homelab runs GitOps with ArgoCD, which is up at 10.0.0.201. Here you get to see mutate and GitOps conflict directly, because the reconciliation loop is a tool that keeps comparing the manifest in Git with the cluster's actual state. The labels or sidecars Kyverno inserted are not in Git, so they show up in the diff, and if the tool tries to remove them, Kyverno puts them back. The fix is to exclude that path from the diff or to move it from mutate to a chart default, and if you put off this decision, a state in which the two controllers keep reverting each other goes on for a long time.

Another thing repeatedly confirmed on this cluster is the relationship between permissions and observation. Both the GPU Operator incident and the KubeVirt incident were of the form "the status display is normal but it does not actually work," and the quiet failure of generate belongs to exactly the same family. So when you deploy a policy that uses generate, it is better to build in a procedure that checks the background controller's permissions before the policy itself.

What you will do in the next lab

In /root/kca-mutate/, you write mutate rules that inject a label and a sidecar, a JSON Patch rule, a generate rule that creates a NetworkPolicy, and a clone rule that copies a ConfigMap. Then you actually create a Role, a RoleBinding, and a ServiceAccount so that the background controller can read the clone source, and verify them with auth can-i.