We left it on warn-only, then switched to enforce and every deploy stopped

Goal

You operate the Pod Security Admission (PSA) built into Kubernetes using only namespace labels. You walk through the order of raising from observation (warn, audit) to blocking (enforce) yourself, and before raising, you pull out in advance what will be blocked and turn it into a report.

Why it matters

PSA has nothing to install — it is already on in the API server, and all you do is hang three labels on a namespace. That is why it looks easy, but incidents always happen when you skip the order. A team that turns on enforce right away has every deployment blocked that day, and a team that leaves only warn on ends up in the same spot half a year later because nobody reads the warnings. What lies between those two failures is the order this lab covers — gather the current state with observation, pin the level version to separate cluster upgrades from policy changes, and raise only after pulling out the existing Pod violations right before the switch and making a list of what to fix. And only if you know that even after raising, Pods that are already running are not blocked, will you not mistake the day you changed the labels for completion.

Steps

In /root/psa/plain.yaml, write one Pod — name web, label app: web, container name web, image nginx:1.27, and not one line of securityContext. Create the namespace psa-legacy without PSA labels and apply this Pod. And in /root/psa/default.txt, write in one word the name of the enforce level that applies to a namespace with no labels at all.
Create the namespace psa-observe and hang only two labels on it, pod-security.kubernetes.io/warn=restricted and pod-security.kubernetes.io/audit=restricted (do not set enforce). Put in the same plain.yaml with kubectl create -f plain.yaml -n psa-observe, and save standard output and standard error together to /root/psa/warn.txt. That file must contain both the line saying the Pod was created and all four violation reasons.
Create the namespace psa-strict and hang pod-security.kubernetes.io/enforce=restricted on it. Put in the same plain.yaml with kubectl create -f plain.yaml -n psa-strict --dry-run=server and save the rejection message, including standard error, to /root/psa/denied.txt. Write a Pod that fixes all four things listed in that message to /root/psa/hardened.yaml (name web-hardened, image unchanged at nginx:1.27) and actually apply it to psa-strict.
Create the namespace psa-baseline and hang pod-security.kubernetes.io/enforce=baseline on it. In /root/psa/privileged.yaml, write a Pod that violates even baseline (name breaker, container name breaker, image busybox:1.36). And in /root/psa/gap.txt, write the result of putting the three manifests (plain.yaml, hardened.yaml, and privileged.yaml) into each of the two namespaces, one line each in the form <파일이름> baseline=<allow|deny> restricted=<allow|deny> (the file name first). Decide with --dry-run=server and do not actually create the Pods.
For each level label of the three namespaces, hang the paired version label as v1.30 — psa-strict and psa-baseline get pod-security.kubernetes.io/enforce-version, and psa-observe gets pod-security.kubernetes.io/warn-version and pod-security.kubernetes.io/audit-version. And create /root/psa/pinned.sh — when run with no arguments, it prints, one per line (sorted), only the names of namespaces that have an enforce, audit, or warn label but lack its paired -version label.
First put two more Pods in psa-legacy — one that is plain.yaml with only the name changed to api, and one that is hardened.yaml with only the name changed to batch. Then create a namespace just for decisions, psa-canary, and hang pod-security.kubernetes.io/enforce=restricted and pod-security.kubernetes.io/enforce-version=v1.30 on it. Save the output of kubectl label ns psa-legacy pod-security.kubernetes.io/enforce=restricted --overwrite --dry-run=server, including standard error, to /root/psa/preview.txt — the label must not actually get attached. Finally, create /root/psa/violators.sh <네임스페이스> (taking a namespace as its argument): it resubmits that namespace's Pods one by one to psa-canary (--dry-run=server) and prints, sorted one per line, only the names of those that violate restricted. Save the result of ./violators.sh psa-legacy to /root/psa/violators.txt.
This time, really hang pod-security.kubernetes.io/enforce=restricted and pod-security.kubernetes.io/enforce-version=v1.30 on psa-legacy. Then apply hardened.yaml to psa-legacy (it must pass), put in plain.yaml with kubectl create -n psa-legacy --dry-run=server, save the rejection message including standard error to /root/psa/enforced.txt, and then append the output of kubectl get pod -n psa-legacy after it in that file. Do not delete the web and api that were already running.
In /root/psa/targets.txt, write the names of the namespaces to check, one per line — four: psa-legacy, psa-observe, psa-strict, and psa-baseline. Create /root/psa/readiness.sh [목록파일] (an optional list file argument; if you omit the argument, it uses /root/psa/targets.txt). For each namespace in the list, it prints one line of judgment: <이름> ENFORCED if it is already enforce=restricted (the namespace name first), <이름> READY if not a single Pod violates, <이름> BLOCKED <개수> if some do (with the count), and <이름> MISSING if there is no such namespace. It must not actually change any labels. Save the execution result to /root/psa/report.txt.

Notes

Inside the lab Pod there is a real kube-apiserver v1.30.4 run by kwok. First run export KUBECONFIG=/root/.kube/config. Containers are not actually run, but admission really works.
Label format: pod-security.kubernetes.io/<enforce|audit|warn>=<privileged|baseline|restricted> and its paired pod-security.kubernetes.io/<모드>-version=<latest|v1.x> (where the placeholder is the mode).
To run only admission and leave no object, use kubectl create -f - --dry-run=server. kubectl label ns ... --dry-run=server warns you in advance about violations of the existing Pods in that namespace.
Common mistake: not knowing that warnings come out on standard error and leaving an empty file by applying only > 파일 (redirecting to a file).
Common mistake: a Pod with the same name already exists, so it never reaches admission and ends with AlreadyExists — for a judgment submission, send it with the name changed.
Common mistake: seeing that nothing happens on the day you raise the label and judging it safe. Existing Pods are blocked when they are redeployed.
Pod Security Admission · Pod Security Standards · Enforce Pod Security Standards with Namespace Labels · Admission Controllers Reference · Migrate from PodSecurityPolicy to PSA

A namespace with no labels at all blocks nothing

In /root/psa/plain.yaml, write one Pod — name web, label app: web, container name web, image nginx:1.27, and not one line of securityContext. Create the namespace psa-legacy without PSA labels and apply this Pod. And in /root/psa/default.txt, write in one word the name of the enforce level that applies to a namespace with no labels at all.

PodSecurity is an admission controller already turned on in the API server, and which level to apply is decided only by namespace labels. If there are no labels, the cluster default is used, and that default is the side that blocks nothing. Pod Security Standards has three levels — this is the name of the loosest of them.

With only warnings on, the same Pod was created along with warnings

Create the namespace psa-observe and hang only two labels on it, pod-security.kubernetes.io/warn=restricted and pod-security.kubernetes.io/audit=restricted (do not set enforce). Put in the same plain.yaml with kubectl create -f plain.yaml -n psa-observe, and save standard output and standard error together to /root/psa/warn.txt. That file must contain both the line saying the Pod was created and all four violation reasons.

PSA's three modes are independent of one another — only enforce blocks the request, warn returns a warning to the person who made the request, and audit leaves an annotation in the audit log. Warnings come out on standard error, so you must capture both, as in > 파일 2>&1 (redirecting to a file). The value of this step lies in "why turn it on if it can block nothing?"

Read the four reasons and fix the Pod to fit restricted

Create the namespace psa-strict and hang pod-security.kubernetes.io/enforce=restricted on it. Put in the same plain.yaml with kubectl create -f plain.yaml -n psa-strict --dry-run=server and save the rejection message, including standard error, to /root/psa/denied.txt. Write a Pod that fixes all four things listed in that message to /root/psa/hardened.yaml (name web-hardened, image unchanged at nginx:1.27) and actually apply it to psa-strict.

The rejection message tells you the places to fix as they are — you only need to sort out which belong to the Pod-level securityContext and which to the container level. Setting runAsNonRoot to true means the image must not run as root, so it is safer to also decide the UID to run as. For capabilities, the order is to drop them all and then add back only what is needed.

A Pod that baseline lets through and only restricted blocks

Create the namespace psa-baseline and hang pod-security.kubernetes.io/enforce=baseline on it. In /root/psa/privileged.yaml, write a Pod that violates even baseline (name breaker, container name breaker, image busybox:1.36). And in /root/psa/gap.txt, write the result of putting the three manifests (plain.yaml, hardened.yaml, and privileged.yaml) into each of the two namespaces, one line each in the form <파일이름> baseline=<allow|deny> restricted=<allow|deny> (the file name first). Decide with --dry-run=server and do not actually create the Pods.

baseline blocks only the well-known privilege escalations, and restricted adds hardening rules on top. What you cannot use in either is host namespaces and privileged containers. If a Pod with the same name already exists, it ends with AlreadyExists before reaching admission, so it is safer to change the name and namespace when you submit (you can extract it with kubectl create -f 파일 --dry-run=client -o json, fix it with jq, and pass it in again — where the placeholder stands for the file name).

A namespace left at latest stalls on cluster upgrade day

For each level label of the three namespaces, hang the paired version label as v1.30 — psa-strict and psa-baseline get pod-security.kubernetes.io/enforce-version, and psa-observe gets pod-security.kubernetes.io/warn-version and pod-security.kubernetes.io/audit-version. And create /root/psa/pinned.sh — when run with no arguments, it prints, one per line (sorted), only the names of namespaces that have an enforce, audit, or warn label but lack its paired -version label.

If you do not hang a version label, that spot runs at latest — meaning "the current version" of the level, so when you upgrade the cluster the rules go up with it, and a Pod that passed until yesterday is blocked today. The labels are contained whole in .items[].metadata.labels of kubectl get ns -o json, and in a namespace with no labels at all, that spot does not exist at all. The problem is to loop over the three modes by name and check for the presence of the paired label.

Pull out in advance what will be blocked before raising

First put two more Pods in psa-legacy — one that is plain.yaml with only the name changed to api, and one that is hardened.yaml with only the name changed to batch. Then create a namespace just for decisions, psa-canary, and hang pod-security.kubernetes.io/enforce=restricted and pod-security.kubernetes.io/enforce-version=v1.30 on it. Save the output of kubectl label ns psa-legacy pod-security.kubernetes.io/enforce=restricted --overwrite --dry-run=server, including standard error, to /root/psa/preview.txt — the label must not actually get attached. Finally, create /root/psa/violators.sh <네임스페이스> (taking a namespace as its argument): it resubmits that namespace's Pods one by one to psa-canary (--dry-run=server) and prints, sorted one per line, only the names of those that violate restricted. Save the result of ./violators.sh psa-legacy to /root/psa/violators.txt.

When there are many Pods, the advance warning shortens them to (and N other pods) — to know all the names, you must judge each Pod separately. Pods that are already running do not go through admission again, so the method is to resubmit the same spec to a namespace that enforces that level. From kubectl get pod <이름> -n <ns> -o json, keep only apiVersion, kind, and spec, and write metadata fresh (the placeholders are the Pod name and the namespace).

Even after raising, the Pods that were already running keep running

This time, really hang pod-security.kubernetes.io/enforce=restricted and pod-security.kubernetes.io/enforce-version=v1.30 on psa-legacy. Then apply hardened.yaml to psa-legacy (it must pass), put in plain.yaml with kubectl create -n psa-legacy --dry-run=server, save the rejection message including standard error to /root/psa/enforced.txt, and then append the output of kubectl get pod -n psa-legacy after it in that file. Do not delete the web and api that were already running.

PSA is an admission controller, so it judges only when a request comes in — it does not look again at objects already stored, so it does not evict violating Pods that are running. That is why the cluster right after raising to enforce is in a state where "only what comes in fresh is clean," and existing Pods are blocked only when they are redeployed. If you do not know this time lag, you will wrongly judge it safe when you see nothing happen on the day you raise it.

Build a report on which namespace can be raised first

In /root/psa/targets.txt, write the names of the namespaces to check, one per line — four: psa-legacy, psa-observe, psa-strict, and psa-baseline. Create /root/psa/readiness.sh [목록파일] (an optional list file argument; if you omit the argument, it uses /root/psa/targets.txt). For each namespace in the list, it prints one line of judgment: <이름> ENFORCED if it is already enforce=restricted (the namespace name first), <이름> READY if not a single Pod violates, <이름> BLOCKED <개수> if some do (with the count), and <이름> MISSING if there is no such namespace. It must not actually change any labels. Save the execution result to /root/psa/report.txt.

If you call the violators.sh from the previous step as it is, you only need to count lines for the count. The reason you must first filter out the ones that are already restricted is that if you reapply a label with the same value to such a namespace, the value does not change and no advance warning comes out at all. If you make the list file an argument, you can use the same script for other batches too.