Handling Secrets and Encryption at Rest

Goal

You create Secrets by type, see how the exposure paths differ by injection method, narrow the read scope with RBAC, and then write an encryption-at-rest configuration file yourself.

Why it matters

The values in a Secret manifest are in base64, so they look encrypted, but base64 is an encoding, not encryption. There is no key, and reversing it takes one command. By default, Secrets are stored in etcd in plaintext, so anyone who gets an etcd backup file reads every Secret. Only when you turn on EncryptionConfiguration are stored values encrypted, and here the identity provider must be last in the list. If it comes first, it goes back to plaintext storage.

The injection method matters too. Environment variables can be read through /proc/PID/environ, are inherited by child processes, and are carried out wholesale in crash reports and debug pages. A volume mount is safer, and narrowing the file permissions with defaultMode is the convention.

Last is RBAC. A person with get secrets permission in a namespace reads every Secret in that namespace. It is surprisingly little known that you can allow only a specific Secret with resourceNames.

Steps

1. Create the namespace cks-secrets and create an Opaque Secret db-cred in it. There are two keys: username (value app) and password (value pr0d-Db-Pass).
2. Create a Secret registry-cred of type kubernetes.io/dockerconfigjson in cks-secrets. The server is registry.cks.local, the user is ci, and the password is ci-token.
3. Create a Secret shop-tls of type kubernetes.io/tls in cks-secrets. Both tls.crt and tls.key must be non-empty.
4. Create a Pod env-app (container name app, image nginx:1.27-alpine) in cks-secrets. The container's envFrom[0].secretRef.name is db-cred.
5. Create a Pod vol-app (container name app) in cks-secrets. The volume name is db, secret.secretName is db-cred, secret.defaultMode is 0400 (256 in decimal), and the container mounts that volume at /etc/db with readOnly: true.
6. Create an Opaque Secret app-config in cks-secrets. It has one key, mode (value strict), and immutable is true.
7. Create a ServiceAccount app in cks-secrets, and create a Role db-cred-reader. apiGroups is the core group, resources is secrets, resourceNames is the single entry db-cred, and verbs is the single entry get. Bind it to the SA app with a RoleBinding app-db-cred. Then save the answers (yes/no) to the two questions, in order, as two lines in /root/cks-secrets/can-i.txt. The first line is whether the app SA can get secret/db-cred, and the second line is whether it can get secret/app-config.
8. Write the encryption-at-rest configuration in /root/cks-secrets/encryption-config.yaml. apiVersion: apiserver.config.k8s.io/v1, kind: EncryptionConfiguration, resources[0].resources is the single entry secrets, and providers has two entries, the first being aescbc (key name key1, secret being a base64-encoded 32-byte value) and the last being identity.

Notes

• kubectl create secret generic db-cred --from-literal=username=app --from-literal=password=pr0d-Db-Pass -n cks-secrets
• Generate a 32-byte key: head -c 32 /dev/urandom | base64
• kubectl auth can-i get secret/db-cred -n cks-secrets --as=system:serviceaccount:cks-secrets:app
• Common mistake 1: wrapping defaultMode: 0400 in quotes makes it a string and it is rejected. Write it as a number.
• Common mistake 2: if you put identity first in the EncryptionConfiguration, newly written Secrets are stored in plaintext.
• Common mistake 3: even after you turn the configuration on, existing Secrets are plaintext until they are rewritten. You must update them all with kubectl get secrets -A -o json | kubectl replace -f -.

Create an Opaque Secret

Create the namespace cks-secrets and create an Opaque Secret db-cred in it. There are two keys: username (value app) and password (value pr0d-Db-Pass).

kubectl create secret generic --from-literal= is the fastest. If you don't specify a type, it is Opaque.

A registry credential Secret

Create a Secret registry-cred of type kubernetes.io/dockerconfigjson in cks-secrets. The server is registry.cks.local, the user is ci, and the password is ci-token.

For a Secret with a defined type, the data key names are defined too. The dockerconfigjson type uses only the single key .dockerconfigjson.

A TLS Secret

Create a Secret shop-tls of type kubernetes.io/tls in cks-secrets. Both tls.crt and tls.key must be non-empty.

The API server accepts it only if both keys tls.crt and tls.key are present.

Inject as environment variables

Create a Pod env-app (container name app, image nginx:1.27-alpine) in cks-secrets. The container's envFrom[0].secretRef.name is db-cred.

The secretRef of envFrom spreads all of a Secret's keys into environment variables. It is convenient, but remember that it stays in the process environment as is.

Mount as a volume and narrow the permissions

Create a Pod vol-app (container name app) in cks-secrets. The volume name is db, secret.secretName is db-cred, secret.defaultMode is 0400 (256 in decimal), and the container mounts that volume at /etc/db with readOnly: true.

If you write defaultMode in octal in YAML, it is stored in decimal. 0400 is owner read-only.

An immutable Secret

Create an Opaque Secret app-config in cks-secrets. It has one key, mode (value strict), and immutable is true.

If you turn on immutable: true, the data can't be modified and you must delete and recreate it. It also reduces the kubelet's watch load.

Restrict reading to a specific Secret

Create a ServiceAccount app in cks-secrets, and create a Role db-cred-reader. apiGroups is the core group, resources is secrets, resourceNames is the single entry db-cred, and verbs is the single entry get. Bind it to the SA app with a RoleBinding app-db-cred. Then save the answers (yes/no) to the two questions, in order, as two lines in /root/cks-secrets/can-i.txt. The first line is whether the app SA can get secret/db-cred, and the second line is whether it can get secret/app-config.

If you pin the name with a Role's resourceNames, only that Secret can be read. Check the result by asking auth can-i in the TYPE/NAME format.

An encryption-at-rest configuration file

Write the encryption-at-rest configuration in /root/cks-secrets/encryption-config.yaml. apiVersion: apiserver.config.k8s.io/v1, kind: EncryptionConfiguration, resources[0].resources is the single entry secrets, and providers has two entries, the first being aescbc (key name key1, secret being a base64-encoded 32-byte value) and the last being identity.

In the provider list, order carries meaning. For writing, the first one is used, and for reading, they are tried in order. If you put identity in the wrong place, it goes back to plaintext storage.