TT Lab
Get started
Learn Learning paths Courses

KCA — Kyverno Certified Associate

The policy was rejected, and the culprit was permissions

Continue in TT Lab

Goal

You read the components of an installed Kyverno, fix generate and cleanup rejections by adding controller permissions through an aggregated ClusterRole, and use real requests to tell where the resourceFilters and webhooks settings of the kyverno ConfigMap filter out requests.

Why it matters

Kyverno is not one program but a system in which four controllers, admission, background, cleanup, and reports, each run under their own ServiceAccount. The objects that generate rules create and that cleanup policies delete are handled with that controller's permissions, not the requester's, so without the permissions the policy is rejected in admission the moment you write it. The installation manifest is designed to keep default permissions to a minimum and to add label-aggregated ClusterRoles, so the basic operating practice is to add only the needed kinds and verbs instead of granting wildcard permissions. Conversely, when you want to take checks out, there are two places. resourceFilters is where Kyverno ignores a request after the API server has sent it, and the webhook namespaceSelector is where the API server does not send it at all. Either way, if you remove the default exclusions, you get incidents where Kyverno blocks itself or the system namespaces.

Steps

  1. In /root/kca-install/inventory.json, write version (the tag of the kyverno-admission-controller image), controllers (the Deployment names in the kyverno namespace mapped to the spec.replicas number), crd_count (the number of CRDs whose API group ends with kyverno.io), and webhook_configs (a sorted array of the names of the ValidatingWebhookConfigurations and MutatingWebhookConfigurations that have kyverno in their names).
  2. In /root/kca-install/02-gen-pdb.yaml, write and try to apply the ClusterPolicy gen-pdb. The rule default-pdb creates, when a Namespace with the label kca.io/pdb=true is created, a PodDisruptionBudget default-pdb (minAvailable 1, selector app=web) in it with synchronize true. Save the entire output of the apply command to /root/kca-install/gen-denied.txt.
  3. In /root/kca-install/03-role-bg.yaml, create the ClusterRole kca-bg-pdb, attach the label rbac.kyverno.io/aggregate-to-background-controller: "true", and allow only get, list, watch, create, update, and delete on poddisruptionbudgets in the policy group (no wildcards). Then apply gen-pdb again, create the namespace kca-pdb-a with the label kca.io/pdb=true, and confirm that the PDB is created. Write down the UID of the PDB that was created, delete the PDB, and write it together with the UID of the PDB that was created again in /root/kca-install/pdb.json as first_uid and second_uid.
  4. In /root/kca-install/04-clean-short.yaml, write and try to apply the ClusterCleanupPolicy clean-short. It deletes Pods with the label ttl=short in the namespace kca-clean every minute (*/1 * * * *). Save the entire output of the apply command to /root/kca-install/cleanup-denied.txt.
  5. Apply the ClusterRole kca-cleanup-pods (label rbac.kyverno.io/aggregate-to-cleanup-controller: "true", only get, list, watch, and delete on core pods) in /root/kca-install/05-role-cleanup.yaml, and apply clean-short again. In the namespace kca-clean, create the Pod short with the label ttl=short and the Pod long with ttl=long, and after the next top of the minute has passed, confirm that only short has disappeared, then write long_uid (the UID of the remaining long) and last_execution (the status.lastExecutionTime of clean-short) in /root/kca-install/cleanup.json.
  6. Create the namespaces kca-skip and kca-ctl, and apply the ClusterPolicy need-team (background false, rule team, requiring a team label on Pods in both namespaces, Enforce) in /root/kca-install/06-need-team.yaml. Then append [Pod,kca-skip,*] to resourceFilters of the ConfigMap kyverno in the kyverno namespace while keeping all existing entries. An unlabeled Pod in kca-skip must be accepted, and an unlabeled Pod in kca-ctl must still be rejected.
  7. Create the namespace kca-sel with the label kca.io/webhook=skip, add kca-sel to the target namespaces of need-team, and apply again. Then, in the namespaceSelector of the webhooks value of the ConfigMap kyverno, add the condition kca.io/webhook NotIn [skip] while keeping the existing kube-system and kyverno exclusion conditions. That condition must appear in the kyverno-resource-validating-webhook-cfg that Kyverno updates, an unlabeled Pod in kca-sel must be accepted, and kca-ctl must still be rejected. Do not put kca-sel in resourceFilters.
  8. In /root/kca-install/report.json, write background_role (the name of the ClusterRole that added generate permissions), cleanup_role (the name of the ClusterRole that added cleanup permissions), pdb_regenerated (whether the two UIDs of step 3 differ, a boolean), skip_by (the configuration key that filtered out kca-skip: resourceFilters or webhooks), sel_by (the configuration key that filtered out kca-sel), and default_exclusions (a sorted array of the default excluded namespace names remaining in the webhook selector).

Notes

What the installed Kyverno is made of

In /root/kca-install/inventory.json, write version (the tag of the kyverno-admission-controller image), controllers (the Deployment names in the kyverno namespace mapped to the spec.replicas number), crd_count (the number of CRDs whose API group ends with kyverno.io), and webhook_configs (a sorted array of the names of the ValidatingWebhookConfigurations and MutatingWebhookConfigurations that have kyverno in their names).

The group of a CRD is in spec.group. There are several groups, such as kyverno.io, policies.kyverno.io, and reports.kyverno.io. Kyverno may also create additional webhook registrations when you create policies, so read what is there now.

The policy is right, yet admission rejected it

In /root/kca-install/02-gen-pdb.yaml, write and try to apply the ClusterPolicy gen-pdb. The rule default-pdb creates, when a Namespace with the label kca.io/pdb=true is created, a PodDisruptionBudget default-pdb (minAvailable 1, selector app=web) in it with synchronize true. Save the entire output of the apply command to /root/kca-install/gen-denied.txt.

The target kind of a generate rule is created by the ServiceAccount of the background controller, not by the requester. Kyverno checks that account's permissions in advance when it receives a policy. Read the rejection message to see who lacks which verb.

Add permissions to the controller with a single label

In /root/kca-install/03-role-bg.yaml, create the ClusterRole kca-bg-pdb, attach the label rbac.kyverno.io/aggregate-to-background-controller: "true", and allow only get, list, watch, create, update, and delete on poddisruptionbudgets in the policy group (no wildcards). Then apply gen-pdb again, create the namespace kca-pdb-a with the label kca.io/pdb=true, and confirm that the PDB is created. Write down the UID of the PDB that was created, delete the PDB, and write it together with the UID of the PDB that was created again in /root/kca-install/pdb.json as first_uid and second_uid.

The kyverno:background-controller ClusterRole installed by Kyverno has an aggregationRule. You do not need to bind the new ClusterRole separately. A generated resource with synchronize turned on is recreated even if you delete it.

A cleanup policy with no permission to delete

In /root/kca-install/04-clean-short.yaml, write and try to apply the ClusterCleanupPolicy clean-short. It deletes Pods with the label ttl=short in the namespace kca-clean every minute (*/1 * * * *). Save the entire output of the apply command to /root/kca-install/cleanup-denied.txt.

It is the cleanup controller that runs cleanup policies. Which kinds that account in the default installation can delete can be seen in the kyverno:cleanup-controller ClusterRole.

Only the short-lived Pod disappeared on the dot

Apply the ClusterRole kca-cleanup-pods (label rbac.kyverno.io/aggregate-to-cleanup-controller: "true", only get, list, watch, and delete on core pods) in /root/kca-install/05-role-cleanup.yaml, and apply clean-short again. In the namespace kca-clean, create the Pod short with the label ttl=short and the Pod long with ttl=long, and after the next top of the minute has passed, confirm that only short has disappeared, then write long_uid (the UID of the remaining long) and last_execution (the status.lastExecutionTime of clean-short) in /root/kca-install/cleanup.json.

The cron schedule is in minutes, so you wait at most 1 minute. Pods that do not match the selector must remain as they are.

A request Kyverno received and pretended not to see

Create the namespaces kca-skip and kca-ctl, and apply the ClusterPolicy need-team (background false, rule team, requiring a team label on Pods in both namespaces, Enforce) in /root/kca-install/06-need-team.yaml. Then append [Pod,kca-skip,*] to resourceFilters of the ConfigMap kyverno in the kyverno namespace while keeping all existing entries. An unlabeled Pod in kca-skip must be accepted, and an unlabeled Pod in kca-ctl must still be rejected.

resourceFilters is a string of [종류,네임스페이스,이름] groups joined by spaces (the placeholders are the kind, namespace, and name). The defaults contain entries such as the kube-system and kyverno namespaces and Event, so if you replace it wholesale, Kyverno ends up inspecting even its own resources. Kyverno rereads this ConfigMap without a restart.

Make the API server not even ask

Create the namespace kca-sel with the label kca.io/webhook=skip, add kca-sel to the target namespaces of need-team, and apply again. Then, in the namespaceSelector of the webhooks value of the ConfigMap kyverno, add the condition kca.io/webhook NotIn [skip] while keeping the existing kube-system and kyverno exclusion conditions. That condition must appear in the kyverno-resource-validating-webhook-cfg that Kyverno updates, an unlabeled Pod in kca-sel must be accepted, and kca-ctl must still be rejected. Do not put kca-sel in resourceFilters.

The webhooks value is a JSON string. Kyverno copies that selector into the webhook registrations it manages, and the API server does not even send Kyverno requests from namespaces that do not match the selector. If you edit the registration object directly, Kyverno reverts it.

Report where you put the permissions and the filters

In /root/kca-install/report.json, write background_role (the name of the ClusterRole that added generate permissions), cleanup_role (the name of the ClusterRole that added cleanup permissions), pdb_regenerated (whether the two UIDs of step 3 differ, a boolean), skip_by (the configuration key that filtered out kca-skip: resourceFilters or webhooks), sel_by (the configuration key that filtered out kca-sel), and default_exclusions (a sorted array of the default excluded namespace names remaining in the webhook selector).

You write them based on the files left from the earlier steps and the current ConfigMap and webhook registrations. The grader recomputes the same values from the cluster.