The Path One Request Takes Through the apiserver

In one line

A request that enters the apiserver flows in the order authentication → authorization → admission → validation → storage. If you know what each stage judges and what it does not, most security questions get solved.

The six stages a single request passes through in the apiserver — authentication, authorization, Mutating admission, schema validation, Validating admission, and etcd storage. A 401 is a failure at stage 1 and a 403 is a failure at stage 2, and Mutating comes first because admission must fix first and check afterward for the order to make sense

Why this was needed

To answer "why was this request rejected," you need to know at which stage it was caught. A 401 is an authentication failure, a 403 is an authorization failure, and an admission webhook's rejection is yet another message. If you do not know the stages, you end up touching certificates when you should be fixing RBAC.

How it works

Stage 1: Authentication — "Who are you?"

The apiserver tries several authenticators in turn, and if even one succeeds, it proceeds with that identity. If all fail, the requester becomes system:anonymous (group system:unauthenticated).

Method	Where it is used
X.509 client certificate	The kubelet, controllers, and admin kubeconfig. CN is the username and O is the group
ServiceAccount token (JWT)	API calls from inside a Pod
OIDC token	Connecting human users to the company IdP
Webhook token authentication	Delegating to an external authentication system

One important fact: Kubernetes has no User object. A user is just a string produced by an authenticator, so you cannot "delete a user." You have to revoke the certificate or cut them off at the IdP.

--anonymous-auth=true does not turn an authentication failure into a rejection but grants an anonymous identity. What anonymous callers are allowed to do is decided at the authorization stage, so more than anonymous access itself, the permissions attached to anonymous are the real problem. In a default cluster, anonymous callers can see only public information such as /healthz and /version.

The system:masters group is special. Members of this group bypass all RBAC checks, and deleting the RoleBinding or ClusterRoleBinding does not revoke their permissions. Once O=system:masters is baked into a certificate, it is effectively a supreme privilege that cannot be revoked as long as the certificate is valid.

Stage 2: Authorization — "Are you allowed to do that?"

Authorization modules are also evaluated in order, and if even one allows, the request passes. You specify them with the --authorization-mode flag.

Node — restricts the kubelet to reading only what relates to its own node
RBAC — role-based. In practice this is the main body
Webhook / ABAC — delegation to an external system / file-based (legacy)
AlwaysAllow — allows everything. A disaster in production

RBAC has only allows and no denies. If there is no allow anywhere, it is a denial. So you cannot create something like "a Role that takes permissions away from this person"; you have to remove the binding.

Stage 3: Admission — "Does this content match policy?"

Even a request that passed authorization is checked for whether its content fits the rules. The order matters — Mutating first, Validating after. It must fix first and check afterward for the order to make sense.

The representative plugin that lives here is PodSecurity (PSA). External policy engines (OPA Gatekeeper, Kyverno) also join this stage through webhooks.

Admission webhooks have an operational pitfall. If failurePolicy is Fail, then when the webhook dies all Pod creation is rejected, and if it is Ignore, the policy is bypassed entirely. The judgment the author's blog reached is clear — "Keep Fail in production, but secure enough resources and replicas for the Gatekeeper Pods." It means solve an availability problem with engineering, not by weakening the policy.

Stage 4: etcd storage and encryption at rest

The apiserver serializes the object (protobuf by default) and then writes it to etcd through a transformer. If the transformer is identity, it is plaintext, and if it is aescbc, aesgcm, or kms, it is ciphertext.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - aescbc:
          keys: [{name: key1, secret: <base64>}]
      - identity: {}

The order of the providers array is everything. The first one is used for writing, and all providers are used for reading. So if you put identity first, you revert to plaintext storage, and if you put it last, it becomes "new writes are encrypted, and old plaintext can still be read." And even after you turn the setting on, existing Secrets remain in plaintext until they are rewritten — you need a full rewrite such as kubectl get secrets -A -o json | kubectl replace -f -.

The KMS v2 provider goes one step further. The DEK is created and cached locally, and the remote KMS manages only the KEK (envelope encryption). Key rotation does not require restarting the apiserver.

Stage 5: The kubelet's authentication and authorization

The kubelet has identities in two directions.

Outgoing — the client certificate it uses to connect to the apiserver. It has a username of the form system:node:<노드명> (the placeholder is the node name) and the system:nodes group, and the Node authorization module limits the scope with this identity.
Incoming — when someone calls the kubelet API (10250). Here authentication.anonymous.enabled and authorization.mode are decisive. With authorization.mode: AlwaysAllow, anyone who is authenticated can do anything, and with Webhook, it asks the apiserver. Webhook is the right answer.

The controller manager and token issuance

The controller manager runs the ServiceAccount controller and the token controller, and signs SA tokens with the key it receives through --service-account-private-key-file. If you obtain this key, you can issue tokens that impersonate any SA directly. This is because the apiserver only verifies with the corresponding public key.

Since v1.24, creating an SA no longer automatically generates a permanent token Secret, and the default is a time-limited token based on the TokenRequest API (kubectl create token <sa> --duration=3600s). It is a big security improvement. This also explains why --service-account-lookup=false is dangerous — if this value is false, even the token of an already deleted SA keeps passing as long as its signature is valid.

What it looks like in the field

The weak point of this system shows up directly in the author's homelab expansion log. To add a control plane, you need the CA private key. The new node must also become an entity that issues certificates. kubeadm does not have you carry this file with scp; with kubeadm init phase upload-certs --upload-certs it uploads it encrypted, as a Secret inside the cluster, and gives a 64-digit hexadecimal certificate-key as the key to that ciphertext.

And this Secret is automatically deleted after 2 hours. The design aims to minimize the time that the CA private key, the cluster's root of trust, exists inside the cluster, even in encrypted form. It is a good example of the principle "shrink the exposure window of sensitive material" implemented as a product feature.

One more thing. In that homelab, the control plane was not grown to 2 nodes but went straight to 3. The reason is clear — the etcd quorum is a majority, and the majority of 2 members is 2, so 2 members actually have a higher failure probability than 1 member. If either of the two dies, writes become impossible. It is a representative case where "more is better" is wrong in availability design.

What you will do in the next lab

In the next lab, you write the apiserver's list of dangerous flags yourself, write the EncryptionConfiguration YAML with the provider order right, and organize the kubelet check items. Then you attach a service account and a Role in each of two namespaces and prove with kubectl auth can-i that isolation actually works, and you observe and record what an anonymous user can do in this cluster right now.