Tokens, Roles, Admission, and API Versions That Disappear

In one line

A request a Pod sends to the API server passes through three gates in turn: authentication → authorization → admission. A service account token is checked for its signature, expiry, bound object, and audience; RBAC defines the allowed scope with rules that can only add; and an admission controller either modifies the spec (mutating) or rejects it (validating). And because API versions disappear according to fixed rules, you look at what the server is serving right now with kubectl api-resources and move your manifests with kubectl convert.

Why this was needed

The symptoms developers meet are always the same three: "401 Unauthorized," "forbidden," and "the YAML I wrote and the Pod that is running are different." The first is the result of authentication, the second of authorization, and the third of admission, and if you cannot tell the three apart, you keep getting 401 even after opening up RBAC, or you change the token and the forbidden stays the same. The fourth symptom comes after an upgrade — the kubectl apply that worked until yesterday fails with "no matches for kind." That is because the server no longer serves that API version, and this is not an incident but a schedule announced by the API deprecation policy.

How it works

Authentication — how is a service account token checked?

According to the service accounts documentation, a service account authenticates to the API server with a signed JWT. The client attaches an Authorization: Bearer <token> header, and the API server checks in this order — the token signature, whether it has expired, whether the object reference the token claims point to is still valid, whether the token is valid at this moment, and the audience claim. A token issued through the TokenRequest API is bound to the lifetime of a client such as a Pod, so the API server also checks whether that object still exists with the same unique ID. An old-style token held in a Secret is checked against the Secret.

The signing key is explained in the managing service accounts documentation. The token controller in kube-controller-manager signs tokens with the private key of --service-account-private-key-file, and kube-apiserver verifies them with the public key of --service-account-key-file. The route by which a token gets into a Pod is a projected volume that the ServiceAccount admission controller attaches (stable since 1.22, cannot be turned off).

- name: kube-api-access-<random-suffix>
  projected:
    sources:
    - serviceAccountToken:
        path: token
    - configMap:
        name: kube-root-ca.crt
        items: [{key: ca.crt, path: ca.crt}]
    - downwardAPI:
        items: [{fieldRef: {fieldPath: metadata.namespace}, path: namespace}]

The first of the three sources is the key one. The kubelet obtains a time-limited token through the TokenRequest API and injects it (default lifetime 1 hour), refreshes it before it expires, and the token is bound to that Pod and has kube-apiserver as its audience. The old way was a Secret-based token that never expired, and this mechanism replaced it. For a Pod that does not need a token, as in the configure service account documentation, you turn off the mount by putting automountServiceAccountToken: false in the ServiceAccount or the Pod spec (if both exist, the Pod spec wins). For external systems, use a projected token with audience: vault and expirationSeconds: 7200, as in the same document's example, and because the kubelet requests replacement once 80% of the TTL has passed or after 24 hours, the application has to reread the file periodically.

Authorization — RBAC only adds

The key sentence in the RBAC documentation is "permissions are purely additive, and there are no deny rules." A Role is a permission within a namespace, and a ClusterRole, because it is a resource that does not belong to a namespace, defines permissions on cluster-scoped resources or a set of permissions to reuse across several namespaces. A RoleBinding attaches a Role or ClusterRole to a subject (a user, a group, a service account) within a specific namespace, and a ClusterRoleBinding attaches it across the whole cluster.

The least privilege that the RBAC good practices describe is concrete — grant permissions at the namespace level where possible, use a RoleBinding instead of a ClusterRoleBinding to limit them to a specific namespace, avoid wildcards (since they amount to granting permission not only on the resources that exist now but on every resource type that will exist in the future), and use cluster-admin only when truly necessary. For a developer, this translates to: "if my Pod's service account needs only to read ConfigMaps, create a Role with only get and list in that namespace, and a RoleBinding."

Admission — this is where the spec is changed or rejected

According to the admission controllers documentation, an admission controller is code inside kube-apiserver that inspects the data of requests that create, delete, or modify objects. Read requests (get, list, watch) do not go through admission. It runs in two phases — first the mutating controllers change the data, then the validating controllers inspect it, and if even one rejects at either phase, the whole request is rejected. The list enabled by default in 1.37 includes CertificateApproval, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionPolicy, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, and ValidatingAdmissionWebhook.

Picking out only what developers meet every day, it looks like this.

Controller	Type	What it does to my spec
ServiceAccount	mutating + validating	Attaches the service account token volume
LimitRanger	mutating + validating	Rejects if the namespace's LimitRange is violated, and fills in defaults if no requests are written
DefaultStorageClass	mutating	Puts the default StorageClass into a PVC that has no storageClassName
NamespaceLifecycle	validating	Rejects creating objects in a namespace that is terminating or does not exist
PodSecurity	validating	Rejects Pods that violate the namespace's Pod Security labels
ResourceQuota	validating	Rejects if the namespace quota is exceeded

There are also four extension points. MutatingAdmissionWebhook and ValidatingAdmissionWebhook call external webhooks registered through the API; ValidatingAdmissionPolicy declares validation rules inside the API with CEL (Common Expression Language) without any external call; and MutatingAdmissionPolicy is the same approach for mutation. So if the Pod you see with kubectl get pod -o yaml differs from the YAML you submitted — a token volume appeared, requests were filled in, or a sidecar was attached — nobody edited it; it is what the mutating phase did. Which admission rejected a request shows up in the error message as the controller name.

API deprecation — disappearing by fixed rules

The API deprecation policy states that each API group is versioned independently, and that versions follow three tracks: alpha (v1alpha1), beta (v1beta1), and GA (v1). Rule 1: API elements may be removed only by bumping the version of the API group, and once an element is in a particular version, it is not dropped or greatly changed in that version. Rule 2: within a release, an object must be able to round-trip between versions without information loss. Rule 3: a version may not be deprecated in favor of a less stable one (GA can replace beta, but beta cannot replace GA). Rule 4a: lifetime is set by the stability level — a GA version may be marked deprecated but is not removed within a Kubernetes major version, a beta version is deprecated within 9 months or 3 minors after introduction, whichever is longer, and stops being served 9 months or 3 minors after deprecation, whichever is longer, and an alpha version may be removed in any release without notice.

The deprecated API migration guide lists, release by release, the versions that stopped being served. For example, v1.32 no longer serves the FlowSchema and PriorityLevelConfiguration of flowcontrol.apiserver.k8s.io/v1beta3, and you must move to v1, which has existed since v1.29. The migration procedure is in the same document.

1. Find. From 1.19 on, find the places that use deprecated APIs through client warnings, metrics, and audit information. You see what the server is serving right now with kubectl api-resources — with -o wide you can see the supported verbs as well, with --api-group=<그룹> (the placeholder is the group) only a specific group, and with --namespaced=false only cluster-scoped resources.
2. Test. Give the API server --runtime-config=<그룹>/<버전>=false (the placeholders are the group and the version) to turn off in advance a version that is about to be removed and see what breaks.
3. Migrate. Change controllers and integration code to call APIs that are not deprecated, and convert YAML with kubectl convert -f <파일> --output-version <그룹>/<버전> (the placeholders are the file, the group, and the version). For example, for an old Deployment it is --output-version apps/v1. The conversion may put in less-than-ideal defaults, so compare the result with the API reference. kubectl convert was once built into kubectl but is now a plugin that is not in the default installation, so you download it separately by the procedure in the installation documentation.

What it looks like in the field

You opened up RBAC but keep getting 401. 401 is an authentication failure. Authorization (RBAC) is a gate after authentication is passed, so no matter how much you widen the Role, 401 does not change. A typical case is when the token mounted in the Pod has expired and the application read it only once at startup and never again — that is why the documentation says to reread it periodically.

You did not write storageClassName on the PVC, yet some class is attached. The DefaultStorageClass admission put it in. If there is no default StorageClass, it does nothing, and if two or more are marked as default, it follows the separate rule the documentation defines. "A value I did not write is in there" is mostly the trace of a mutating admission.

What to check in the next quiz

The quiz asks about the items checked on a service account token and its default lifetime, where the signing keys are, RBAC's additive nature and the scope of bindings, the two phases of admission and read requests, the lifetime rules for beta APIs, and the use of kubectl convert.