Secrets — Where They Are Stored and Who Can Read Them

Summary

The base64 in a Kubernetes Secret is encoding, not encryption, and with the default settings it is stored in etcd in plaintext. Anyone who gets hold of the etcd disk gets hold of every secret.

Why this matters

When you first look at a Secret manifest, the values are unreadable strings, so they look encrypted. But reversing it takes a single command.

kubectl get secret app-db -o jsonpath='{.data.password}' | base64 -d
# pr0d-Db-Pass

There is no key and no secret. base64 is just a representation for safely carrying binary as text. The more important fact comes next. In a cluster with the default settings, the apiserver writes Secrets to etcd as plaintext, as they are. Someone who obtains an etcd snapshot file, a disk image, or a backup archive can simply read the password string inside it. You learned in the previous module how to take and move snapshots, so you should also know how sensitive that file is.

How it works

The response is built up in layers.

Layer 1 — Encryption at rest (EncryptionConfiguration). If you give the apiserver a configuration file, it encrypts Secrets before writing them to etcd.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <32바이트 base64 키>
      - identity: {}

Here the order is everything. The first provider in the list is used for writes, and for reads it tries them in order from the top. identity means "no encryption," so it must be last. If you put it first, from that moment you revert to plaintext storage. And even if you change the configuration, existing objects remain in plaintext until they are rewritten. You need a job that reads everything once and writes it back.

Layer 2 — Envelope encryption (KMS v2). Writing a key in a cluster configuration file ends up creating yet another secret. A KMS provider splits this problem into two layers. The data is encrypted with a data key (DEK), and that data key is in turn encrypted with the KMS's master key (KEK). The master key lives in a hardware security module or a cloud KMS outside the cluster, and the apiserver reaches it only through a plugin attached over a Unix socket. It is also a big advantage that you do not have to re-encrypt all the data when you rotate the key.

Layer 3 — Access control (RBAC). Encrypting storage is of no use against someone who can read through the API. Someone with get secrets permission in a namespace reads every secret in that namespace. It is very common for edit permission granted for convenience to be, in effect, permission to view production credentials. To narrow the scope, you pin the name with resourceNames.

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-password"]
    verbs: ["get"]

However, resourceNames does not work for list and watch. That is because a list request is one in which the target name cannot be known in advance. So name-level restriction is a tool for narrowing get, and the answer for list is not to grant it in the first place.

Layer 4 — Keeping the values outside the cluster. Tools such as External Secrets Operator keep the real values in a secrets manager and commit only references to the cluster. A SecretStore defines "where to fetch from," and an ExternalSecret defines "what to create under which name." Since there are no values in the manifest, you can commit it to git as it is. However, the operator ends up creating cluster Secrets anyway, so the RBAC and encryption-at-rest issues above remain.

Finally there is immutable: true. If you lock the value so it cannot be changed, changes by mistake are blocked, and the load drops too because the kubelet no longer has to watch for changes. In exchange, to change it you have to create it under a new name and move the references.

What it looks like in the field

First, response time is maturity. The question that measures the level of secret management is not "did we hide it well" but "assuming this value is now public, how many minutes does it take to revoke and replace it." If the answer is in hours, you should build the rotation path before buying more tools.

Second, the order of leak response. When you find a key in git history, the first action is not rewriting the history but revoking. A published value is collected by automated scanners within seconds, so no matter how much you erase the history, a value that has already been copied cannot be taken back. The right order is revoke → assess impact → clean up history.

Third, an environment variable is a delivery method, not a storage location. An environment variable injected into a container can be read on the same host through /proc/<PID>/environ, is inherited by child processes, and leaks into crash reports and logs. This is why file mounts are safer.

What to do in the next lab

You create a Secret and reverse the base64 yourself, then write the encryption-at-rest configuration and the KMS v2 envelope encryption configuration. You build the external secret store integration manifest without values, create a role that opens just one secret with resourceNames, and check its limits. You create an immutable Secret and see the modification get rejected, and in the end you query etcd directly and keep as evidence that the plaintext is visible, then sort out the mitigations.