The Promotion Procedure, and What Works Without Policy

Goal

You implement the procedure for promoting a policy from observation to enforcement with manifests and a script, and set up the same kind of control with only PSA labels and no policy engine, to check the boundary between the two approaches.

Why it matters

Using a policy tool well and operating policies well are different skills. What matters in operations is not the sophistication of the rules but the procedure for judging "is it OK to raise this rule to enforcement now." That judgment comes from reports, and the procedure has to be hardened into a script so that it persists even when people change. At the same time, you need the question from the other side. A standardized check such as the Pod security level is finished with two namespace label lines, so if you put even that into a policy engine, it only increases the components you have to operate. By setting up the two approaches side by side in this lab, that boundary becomes clear.

Steps

Create the /root/kca-ops/ directory and write kind ClusterPolicy and metadata.name kca-baseline in policy-baseline.yaml. Set spec.validationFailureAction to Audit and put in spec.validationFailureActionOverrides two entries that specify Enforce for kca-prod and Audit for kca-dev. There must also be at least one rule.
In the same file, set spec.webhookConfiguration.timeoutSeconds to 20 and spec.webhookConfiguration.failurePolicy to Ignore. Do not use the deprecated spec.webhookTimeoutSeconds and spec.failurePolicy.
In the same file, set spec.background to true, spec.admission to true, and spec.applyRules to All, and make two or more rules.
In /root/kca-ops/policyexception.yaml, write apiVersion kyverno.io/v2beta1, kind PolicyException, and metadata.name kca-allow-monitoring. Set spec.exceptions[0].policyName to kca-baseline, put the rule name you made in step 1 in ruleNames, and put kca-monitoring in spec.match.any[0].resources.namespaces and prometheus-* in names.
Make /root/kca-ops/promote.sh a shell script with execute permission. It must contain all of the following: local verification using kyverno apply and --resource, a PolicyReport check, a check of the webhook registration or UpdateRequests, a step that raises to Enforce, and handling that ends with a non-zero code on failure.
Create the namespace kca-prod in the cluster and attach the labels pod-security.kubernetes.io/enforce=restricted and pod-security.kubernetes.io/enforce-version. For the namespace kca-dev, attach only pod-security.kubernetes.io/warn=baseline and do not attach an enforce label.
Actually create the Deployment kca-hello in the namespace kca-prod. The Pod-level securityContext.runAsNonRoot must be true, securityContext.seccompProfile.type must be RuntimeDefault, the container-level securityContext.allowPrivilegeEscalation must be false, and capabilities.drop must contain ALL. And create the namespace kca-monitoring with the label pod-security.kubernetes.io/enforce=privileged.

Notes

You can attach both labels at once with kubectl label ns kca-prod pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/enforce-version=latest.
Check the script in advance with bash -n /root/kca-ops/promote.sh.
Common mistake 1: opening a PolicyException broadly to a whole namespace. Only when you narrow it down to the name pattern does an exception remain an exception.
Common mistake 2: not pinning the version in the PSA labels. A cluster upgrade changes the contents of the policy.

Per-namespace differential application

Create the /root/kca-ops/ directory and write kind ClusterPolicy and metadata.name kca-baseline in policy-baseline.yaml. Set spec.validationFailureAction to Audit and put in spec.validationFailureActionOverrides two entries that specify Enforce for kca-prod and Audit for kca-dev. There must also be at least one rule.

Keep the default for the whole policy in observation mode, and specify different behavior per namespace in the override array. The two fields action and namespaces make up one entry.

Migrate to webhookConfiguration

In the same file, set spec.webhookConfiguration.timeoutSeconds to 20 and spec.webhookConfiguration.failurePolicy to Ignore. Do not use the deprecated spec.webhookTimeoutSeconds and spec.failurePolicy.

The timeout and failure policy moved one block down from 1.13. You must not leave the fields at the old location, and think about which way to set the failure policy so that a policy in the observation stage does not stop the cluster.

background, admission, and applyRules

In the same file, set spec.background to true, spec.admission to true, and spec.applyRules to All, and make two or more rules.

The three flags decide, respectively, scanning existing resources, applying at admission, and the number of rules applied. The last field is the one that creates the problem of listing several rules and having the later ones not run.

A legitimate exception with PolicyException

In /root/kca-ops/policyexception.yaml, write apiVersion kyverno.io/v2beta1, kind PolicyException, and metadata.name kca-allow-monitoring. Set spec.exceptions[0].policyName to kca-baseline, put the rule name you made in step 1 in ruleNames, and put kca-monitoring in spec.match.any[0].resources.namespaces and prometheus-* in names.

An exception points to the policy name and the rule name together. Instead of writing only the namespace and excluding it wholesale, narrow it down to the name pattern.

The promotion checklist script

Make /root/kca-ops/promote.sh a shell script with execute permission. It must contain all of the following: local verification using kyverno apply and --resource, a PolicyReport check, a check of the webhook registration or UpdateRequests, a step that raises to Enforce, and handling that ends with a non-zero code on failure.

It can be used as a CI gate only when it contains local verification, a report check, diagnosis, and a non-zero exit on failure. Do not forget the execute permission either.

Make two tiers with PSA labels

Create the namespace kca-prod in the cluster and attach the labels pod-security.kubernetes.io/enforce=restricted and pod-security.kubernetes.io/enforce-version. For the namespace kca-dev, attach only pod-security.kubernetes.io/warn=baseline and do not attach an enforce label.

From here on it is the real cluster. Pod Security Admission works with namespace labels, and there are three modes: enforce, audit, and warn. Also think about why you pin the version label along with it.

A workload that passes restricted

Actually create the Deployment kca-hello in the namespace kca-prod. The Pod-level securityContext.runAsNonRoot must be true, securityContext.seccompProfile.type must be RuntimeDefault, the container-level securityContext.allowPrivilegeEscalation must be false, and capabilities.drop must contain ALL. And create the namespace kca-monitoring with the label pod-security.kubernetes.io/enforce=privileged.

The restricted level requires four things: running as non-root, forbidding privilege escalation, dropping all capabilities, and a seccomp profile. Also create the namespace that will be the target of the exception.