Setting Up Injection and Generation Rules and Their Permissions

Goal

You write the two mutate methods and the two generate methods as manifests, and build, directly in the cluster, the RBAC that generate needs in order to actually work, verifying even the permissions.

Why it matters

A generate rule is the feature that fails most quietly in Kyverno. Admission succeeds, so kubectl gives no error at all, and only the resource does not appear. There are usually two causes: match did not hit, or the background controller lacks permissions. If you create the permissions yourself in this lab and check them with auth can-i, then later in the field, when a generated resource does not show up, you will look at the UpdateRequest and permissions first instead of digging through the policy YAML. The habit of putting resource limits into sidecar injection is the same kind of preventive measure.

Steps

Create the /root/kca-mutate/ directory and write kind ClusterPolicy and metadata.name kca-add-team-label in mutate-labels.yaml. Create a rule add-managed-by, put Deployment in the kinds of match, and have mutate.patchStrategicMerge put kca.io/managed-by in metadata.labels with the value kyverno.
In the same file, add a rule inject-sidecar. Select the targets with kca.io/inject in match.any[0].resources.selector.matchLabels, and in mutate.patchStrategicMerge.spec.template.spec.containers[0] put the name log-collector, an image, and both memory and cpu under resources.limits.
In /root/kca-mutate/mutate-json6902.yaml, write kind ClusterPolicy and, with spec.rules[0].mutate.patchesJson6902, write an add operation that adds the annotation kca.io/owner. The path is under /metadata/annotations/, and you must escape the slash in the key.
In /root/kca-mutate/generate-netpol.yaml, write kind ClusterPolicy and metadata.name kca-generate-default-deny. The kinds of match is Namespace, generate.apiVersion is networking.k8s.io/v1, kind is NetworkPolicy, name is default-deny, namespace is a variable that receives the new namespace's name, synchronize is true, and put Ingress and Egress in generate.data.spec.policyTypes.
In /root/kca-mutate/generate-clone.yaml, write kind ClusterPolicy and metadata.name kca-clone-baseline. generate.kind is ConfigMap, name is kca-baseline, clone.namespace is kca-shared, clone.name is kca-baseline-config, synchronize is true, and do not write data.
Create the namespace kca-shared in the cluster and actually create the Role kca-clone-reader in it. Allow only get, list, and watch on configmaps of the core group, and do not include delete.
In the cluster, create the namespace kyverno and the ServiceAccount kyverno-background-controller in it, and actually create the RoleBinding kca-clone-reader in kca-shared to bind the Role kca-clone-reader to that service account.
Actually create the ConfigMap kca-baseline-config in kca-shared. The data is the two entries log-level=info and retention=7d. After that, use kubectl auth can-i to check that the service account can read configmaps in this namespace but cannot read secrets.

Notes

You can create it quickly with kubectl create role kca-clone-reader --verb=get,list,watch --resource=configmaps -n kca-shared.
The permission check is kubectl auth can-i get configmaps --as=system:serviceaccount:kyverno:kyverno-background-controller -n kca-shared.
Common mistake 1: using clone and data together in generate. Choose only one of them.
Common mistake 2: leaving the slash of the key as is in the JSON Patch path. It cannot be distinguished from the path separator.

Inject a label with patchStrategicMerge

Create the /root/kca-mutate/ directory and write kind ClusterPolicy and metadata.name kca-add-team-label in mutate-labels.yaml. Create a rule add-managed-by, put Deployment in the kinds of match, and have mutate.patchStrategicMerge put kca.io/managed-by in metadata.labels with the value kyverno.

If you draw the object's shape as it is inside the mutate block, only that part is merged. If a label key has a dot and a slash, wrap it in quotes in YAML.

Sidecar injection and resource limits

In the same file, add a rule inject-sidecar. Select the targets with kca.io/inject in match.any[0].resources.selector.matchLabels, and in mutate.patchStrategicMerge.spec.template.spec.containers[0] put the name log-collector, an image, and both memory and cpu under resources.limits.

You must put resource limits on the container you inject. Recall why the fact that mutate runs before validate matters here. And make sure the injection happens only for workloads selected by the label, not for all workloads.

Add an annotation with JSON Patch

In /root/kca-mutate/mutate-json6902.yaml, write kind ClusterPolicy and, with spec.rules[0].mutate.patchesJson6902, write an add operation that adds the annotation kca.io/owner. The path is under /metadata/annotations/, and you must escape the slash in the key.

JSON Pointer splits the path with slashes. If a slash is in the key name itself, a special escape is needed, and if you do not know that notation, the path points to the wrong place.

Generate a default-deny for every namespace

In /root/kca-mutate/generate-netpol.yaml, write kind ClusterPolicy and metadata.name kca-generate-default-deny. The kinds of match is Namespace, generate.apiVersion is networking.k8s.io/v1, kind is NetworkPolicy, name is default-deny, namespace is a variable that receives the new namespace's name, synchronize is true, and put Ingress and Egress in generate.data.spec.policyTypes.

generate has to state what to create (apiVersion/kind), under what name, and in which namespace. The target namespace is the namespace that was just created, so you receive it as a variable.

Copy with the clone method

In /root/kca-mutate/generate-clone.yaml, write kind ClusterPolicy and metadata.name kca-clone-baseline. generate.kind is ConfigMap, name is kca-baseline, clone.namespace is kca-shared, clone.name is kca-baseline-config, synchronize is true, and do not write data.

clone specifies the namespace and name of the source. Check that it cannot be used together with data, and which field is needed to make it follow changes to the source.

Create a Role that can read the clone source

Create the namespace kca-shared in the cluster and actually create the Role kca-clone-reader in it. Allow only get, list, and watch on configmaps of the core group, and do not include delete.

From here on it is the real cluster. Only reading is needed, so three verbs are enough, and the apiGroups of a core group resource is an empty string.

Bind it to the background controller

In the cluster, create the namespace kyverno and the ServiceAccount kyverno-background-controller in it, and actually create the RoleBinding kca-clone-reader in kca-shared to bind the Role kca-clone-reader to that service account.

The subject is the service account in the kyverno namespace. That namespace and service account do not exist in this lab environment either, so you have to create them too.

The source ConfigMap and permission verification

Actually create the ConfigMap kca-baseline-config in kca-shared. The data is the two entries log-level=info and retention=7d. After that, use kubectl auth can-i to check that the service account can read configmaps in this namespace but cannot read secrets.

The clone source has to actually exist for clone to work. And if you impersonate the service account with --as in kubectl auth can-i, you can check by hand that the permissions were attached correctly.