Seeing the Plaintext in etcd Yourself, and Stopping It

Goal

You open etcd directly to check how a Secret is actually stored, and write each of the four layers of response, encryption at rest, envelope encryption, access control, and an external store, as a manifest.

Why it matters

The most widespread misconception is that "a Secret is safe because it is base64." base64 is a representation with no key, so reversing it takes a single command, and more important is the fact that with the default settings the apiserver writes Secrets to etcd as plaintext, as they are. The etcd snapshot you learned about in the previous module is therefore also the most sensitive file in the cluster. In this lab you confirm that fact with your own hands rather than taking someone's word for it. The response is not a single layer. EncryptionConfiguration stops someone who obtains the disk, and KMS envelope encryption takes even that key out of the cluster. But encrypting storage is of no use against someone who can read through the API, so RBAC is needed, and although resourceNames can narrow things down to the name level, you also need to know the limit that it does not work for list/watch. Finally, through the external secrets approach that keeps the values entirely outside the cluster, the core of this lab is distinguishing what each of the four layers blocks and what it cannot.

Steps

Create the namespace secret-lab and in it create the Secret db-password. The type is Opaque and the value of the key password is labhub-Pr0d-2026. Also create one more Secret, other-secret, as a control group for comparing permissions later.
Save the value of db-password's data.password, as is, to /root/ops/secrets/out/encoded.txt, and the decoded plaintext to /root/ops/secrets/out/decoded.txt (the decoded result must be labhub-Pr0d-2026 and the two files must correspond to each other). And in /root/ops/secrets/out/base64-note.txt, write in one line the conclusion that base64 is only encoding, not encryption.
Write /root/ops/secrets/encryption-config.yaml. apiVersion is apiserver.config.k8s.io/v1, kind is EncryptionConfiguration, and resources[0].resources must contain secrets. The first of providers is a real encryption provider (one of aescbc, secretbox, and aesgcm) and must have keys[0].name, and the last must be identity.
Write /root/ops/secrets/kms-config.yaml. kind is EncryptionConfiguration and providers[0] must be kms. kms.apiVersion is v2, kms.endpoint is a socket path starting with unix://, and kms.name must also be present. And in /root/ops/secrets/out/envelope-note.txt, write an explanation of the two-layer structure of envelope encryption (data key and master key).
In /root/ops/secrets/external-secret.yaml, write two documents. One is kind: SecretStore and the other is kind: ExternalSecret. The ExternalSecret must have all of spec.refreshInterval, spec.secretStoreRef.name, spec.target.name, spec.target.creationPolicy, and spec.data[0].remoteRef.key. The real password labhub-Pr0d-2026 must not be in this file.
In secret-lab, create the ServiceAccount db-client and the Role db-secret-reader. The rule must be, for the secrets resource, only a single resourceNames entry db-password and a single verbs entry get. Bind this role to db-client with a RoleBinding. As a result, db-client must read db-password, must not read other-secret, and must not be able to list secrets (list) either. And in /root/ops/secrets/out/resourcenames-note.txt, write that resourceNames does not work for list/watch.
In secret-lab, create the Secret pinned-config. It is immutable: true and the value of the key build is 2026-08-20. Then try to change this value and save the rejection message to /root/ops/secrets/out/immutable-error.txt. Even after that, the build value must still be 2026-08-20.
Save the output of querying the key /registry/secrets/secret-lab/db-password from etcd to /root/ops/secrets/out/etcd-secret.txt. The plaintext labhub-Pr0d-2026 must be visible in this file. Then create /root/ops/secrets/out/secrets-audit.json. total_secrets must match the actual number of Secrets in secret-lab, encrypted_at_rest is false, and plaintext_visible_in_etcd is true. Write 3 or more mitigations in the mitigations array, and it must include encryption at rest (EncryptionConfiguration or "encryption") and access control (RBAC or "permissions").

Reference

The etcd in this lab listens at 127.0.0.1:2379 without client TLS. You can query it with etcdctl --endpoints=127.0.0.1:2379 get /registry/secrets/secret-lab/db-password.
Kubernetes stores objects in etcd as protobuf, not JSON. So it is normal for the output to look like garbled binary, and human-readable strings are mixed into it. The password is in there in plaintext, not base64 — that is the point of this step. To look at it comfortably, skim it with | strings or | hexdump -C, and save the query output to the file as it is (if the null bytes bother you, strip them with | tr -d '\0').
For the encryption key in step 3, use a 32-byte random value encoded in base64. You can make one with head -c 32 /dev/urandom | base64.
For the modification attempt in step 7, either kubectl patch or kubectl apply works. The error message goes to standard error, so you have to capture it with 2>&1 for it to remain in the file.
It is safest to count the number of secrets in step 8 with kubectl get secrets -n secret-lab -o json | jq '.items | length'. Counting by hand easily goes wrong.
Common mistake 1: putting identity at the very front of the list in step 3. What is used for writes is the first provider, so you revert to plaintext storage at that moment.
Common mistake 2: in step 6, having verbs include list as well. resourceNames cannot narrow list requests, so every secret is exposed.
Common mistake 3: writing the real value in the manifest of step 5. That there is no value is the reason the external secrets approach exists.
The lab Pod comes up fresh for each lab, so the cluster state created in the earlier lab does not remain. Create the namespace and Secrets yourself within this lab. This is exactly why operational procedures must be left in runbooks and manifests rather than in memory.

Create two Secrets for the lab

Create the namespace secret-lab and in it create the Secret db-password. The type is Opaque and the value of the key password is labhub-Pr0d-2026. Also create one more Secret, other-secret, as a control group for comparing permissions later.

Check what the default type is. You create two because you need a control group for comparing the permission scope later.

Reverse the base64 yourself

Save the value of db-password's data.password, as is, to /root/ops/secrets/out/encoded.txt, and the decoded plaintext to /root/ops/secrets/out/decoded.txt (the decoded result must be labhub-Pr0d-2026 and the two files must correspond to each other). And in /root/ops/secrets/out/base64-note.txt, write in one line the conclusion that base64 is only encoding, not encryption.

This step is for confirming the difference between encoding and encryption by hand. Also check that the reversed value and the encoded value correspond to each other.

Write the encryption-at-rest configuration

Write /root/ops/secrets/encryption-config.yaml. apiVersion is apiserver.config.k8s.io/v1, kind is EncryptionConfiguration, and resources[0].resources must contain secrets. The first of providers is a real encryption provider (one of aescbc, secretbox, and aesgcm) and must have keys[0].name, and the last must be identity.

What is used for writes is the first in the list. Think about what must remain at the end so that you can keep reading the existing plaintext. A key needs a name.

Write the KMS v2 envelope encryption configuration

Write /root/ops/secrets/kms-config.yaml. kind is EncryptionConfiguration and providers[0] must be kms. kms.apiVersion is v2, kms.endpoint is a socket path starting with unix://, and kms.name must also be present. And in /root/ops/secrets/out/envelope-note.txt, write an explanation of the two-layer structure of envelope encryption (data key and master key).

The plugin attaches through a socket, not the network. Leave a note on what the two-layer structure is.

Write the external secret store integration manifest

In /root/ops/secrets/external-secret.yaml, write two documents. One is kind: SecretStore and the other is kind: ExternalSecret. The ExternalSecret must have all of spec.refreshInterval, spec.secretStoreRef.name, spec.target.name, spec.target.creationPolicy, and spec.data[0].remoteRef.key. The real password labhub-Pr0d-2026 must not be in this file.

You need two documents. One defines where to fetch from, and the other defines what to create under which name. If the real password ends up in the manifest, the reason for using it disappears.

Create a role that opens just one secret

In secret-lab, create the ServiceAccount db-client and the Role db-secret-reader. The rule must be, for the secrets resource, only a single resourceNames entry db-password and a single verbs entry get. Bind this role to db-client with a RoleBinding. As a result, db-client must read db-password, must not read other-secret, and must not be able to list secrets (list) either. And in /root/ops/secrets/out/resourcenames-note.txt, write that resourceNames does not work for list/watch.

There is a field that narrows the scope by name. But also sort out that there are verbs for which that field does not work.

Create an immutable Secret and confirm the modification is rejected

In secret-lab, create the Secret pinned-config. It is immutable: true and the value of the key build is 2026-08-20. Then try to change this value and save the rejection message to /root/ops/secrets/out/immutable-error.txt. Even after that, the build value must still be 2026-08-20.

After locking it, actually try to change it and keep the rejection message. The error goes to standard error.

Pull the plaintext out of etcd and keep it as evidence

Save the output of querying the key /registry/secrets/secret-lab/db-password from etcd to /root/ops/secrets/out/etcd-secret.txt. The plaintext labhub-Pr0d-2026 must be visible in this file. Then create /root/ops/secrets/out/secrets-audit.json. total_secrets must match the actual number of Secrets in secret-lab, encrypted_at_rest is false, and plaintext_visible_in_etcd is true. Write 3 or more mitigations in the mitigations array, and it must include encryption at rest (EncryptionConfiguration or "encryption") and access control (RBAC or "permissions").

Recall the key path rule for Kubernetes objects. The value is protobuf so it looks garbled, but the strings read as they are.