A Log You Did Not Keep Becomes an Event That Never Happened

In one line

An audit policy is a document that decides not "what to record" but "what to throw away." Rules are evaluated in order from the top and only the first match is applied, so if you get the order wrong, quietly nothing is left.

Why this was needed

The first question of incident response is always the same. When, who, and what was done. In Kubernetes, the only data that can answer this question is the API server's audit log. But if you record everything, storage costs explode, and if you record sloppily, there is nothing at the very moment you need it. That is why you need a policy.

There are four levels.

Level	What is recorded
None	Nothing is recorded
Metadata	Only metadata such as requester, time, verb, and resource
Request	Metadata + request body
RequestResponse	Metadata + request body + response body

Here there is a trap that the CKS asks about repeatedly. If you apply RequestResponse to Secrets, the secret values go as they are into the audit log file. Audit logs are usually collected in a place whose access permissions are broader than the application's, so a measure meant to protect secrets ends up spreading them. So the standard practice is to record Secrets and ConfigMaps only up to Metadata.

How it works

The policy file is a Policy resource of audit.k8s.io/v1, and it evaluates the rules array from the top and applies only the first matching rule. So order is meaning. For example, if you want to exclude get requests for kube-system's Secrets, that level: None rule must be above the Secret-related rules. If you put it below, the upper rule matches first and the exclusion doesn't work at all.

You also use omitStages. A single request can produce events at four stages: RequestReceived, ResponseStarted, ResponseComplete, and Panic, and RequestReceived is almost always noise. If you put this stage in the top-level omitStages, the log volume drops noticeably.

After you create the policy file, you have to connect it to the API server. You specify the policy with --audit-policy-file and the output location with --audit-log-path, and set the rotation policy with --audit-log-maxage (retention days), --audit-log-maxbackup (number of files), and --audit-log-maxsize (MB). If you leave out these three, the disk fills up, and when the disk fills up, the API server stops.

Dangerous RBAC verbs get special treatment. escalate, bind, and impersonate are themselves privilege escalation attempts, so you leave them at RequestResponse and record even what they tried to change and how. For Pod changes, Request is about the right balance. If you keep the manifest, you can reconstruct what was deployed after a compromise, and you usually don't need the response body.

What it looks like in the field

There are areas the audit log can't answer, and runtime detection fills that gap. Falco observes kernel system calls and alerts on behavior that matches its rules. The most famous of the built-in rules catches the moment a shell starts inside a container, and its condition is evt.type=execve combined with whether it is in a container and whether the process name is in the list of shells. Behavior that doesn't go through the API doesn't appear in the audit log at all, so the two layers are not substitutes but tools that answer different questions.

You check what is actually running on a node with crictl. But you must respect a boundary. crictl bypasses the API server, so it is powerful for diagnosis but dangerous for changing state. The kubelet keeps watching the containers and sandboxes it created, so if a person deletes them with crictl rm, the kubelet's view and the actual state diverge and it stays in a strange intermediate state. A simple rule is this. If you want to know what Kubernetes is trying to do, use kubectl; if you want to know what the runtime actually did, use crictl. The point where the two views diverge is the location of the problem.

There are also cases where diagnosis becomes entirely meaningless. If crictl is attached to the wrong socket, it shows an empty list, and you reach the wrong conclusion that "there are no containers at all." The first step is to check what it is currently attached to with crictl info and compare whether runtime-endpoint in /etc/crictl.yaml matches the kubelet's --container-runtime-endpoint. If you don't specify the endpoint, crictl tries known socket candidates in turn, and a single command can take a dozen seconds or more.

There is also a set order for when you find a compromised Pod. Preserving evidence comes before immediate deletion. Collect the logs and state, cut that Pod's communication with a NetworkPolicy to isolate it, work out the compromise path and scope, and then replace it with a new image that fixes the vulnerability. If you delete it first, the forensic evidence is gone.

What you will do in the next lab

You build up an audit.k8s.io/v1 policy file one rule at a time. Starting from the catch-all rule at the very bottom, you reduce noise with omitStages, and you yourself build the order in which Secrets are kept only at Metadata, kube-system's Secret gets are None outright, dangerous RBAC verbs are RequestResponse, and Pod changes are Request. Finally, you document the audit log backend flags to attach to the API server.