A Token Baked Into an Image Cannot Be Taken Back

In one line

An environment variable is a delivery method, not a storage location. And a secret put in during an image build does not go away when you delete it. If you pushed it to a registry, you must treat it as already leaked and revoke it.

Why this was needed

It is easy to think that loading a .env file and then deleting it is safe. But /proc/<pid>/environ can be read for as long as the process lives. It is a place that a sidecar on the same host, a debugging shell, a crash dump, and a monitoring agent can all read. An environment variable is only a channel for handing a value to a process, not a device that hides the value.

The image side is worse. A token put in with ENV stays permanently in the image configuration.

docker inspect -f '{{json .Config.Env}}' bad:v1
["NPM_TOKEN=npm_9fA3...", ...]

Receiving it via ARG and deleting it with rm ~/.npmrc is the same story. The file remains in a lower layer and also remains in the build history.

docker history --no-trunc bad:v2 | grep -o 'NPM_TOKEN=[^ ]*'

There is only one conclusion to draw here. If you pushed it to a registry, you must treat that token as already leaked and revoke it. Deleting the image cannot undo it.

How it works

If you truly need a secret during the build, use BuildKit's secret mount.

RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci

This mount is a tmpfs, so it disappears when that RUN finishes and does not remain in a layer. This is fundamentally different from passing it as a build argument.

The base64 in a Kubernetes Secret is not encryption. It is only encoding, and real protection begins only when you separately turn on etcd encryption or an external secret store.

The order of actions for a secret you have already committed is fixed: revoke, investigate the impact, then clean up the history. You must not change the order. And history rewriting is not an action that undoes the leak; it is hygiene work that reduces recurrence. This is because the secret has already been copied to forks, local clones, PR references, and CI caches.

What it looks like in the field

Rotation is a design property, not an operating procedure. If the code assumes "there is exactly one valid key," rotation will surely fail. This is because the moment you deploy the new key overlaps with the moment clients still using the old key remain. So you must design it so that signing uses the one current key, and verification uses the whole set of valid keys. Without this principle, a rotation plan will not be carried out.

Detection tools also have three limits. First, a pre-commit hook is bypassed with a single --no-verify. So a hook is a convenience and the control is the server-side check. Second, the rules work well only on values with a distinct shape, such as AKIA and sk_live_. Tokens in an internal system's own format are not caught easily. Third, a scanner looks only at code repositories. It does not see wikis, issue attachments, or chat logs.

Finally, there is one question to put to your team. Assuming this credential has just been made public, how many minutes does it take to revoke it, replace it, and get the service back to normal? If you cannot answer this question, you do not yet have a secret management system.

A file mount must not be left alone either

Moving from environment variables to files is an improvement, but a file does not become safe by itself. A few things remain to check after moving.

Permissions and owner. If the file is placed so that other users can read it, the move has no meaning. Even if it looks fine because there is only one user in the container, if a sidecar mounts the same volume, it can read it too. Specify permissions when mounting, and do not mount that volume in the sidecar.

Does it remain on disk? If the volume holding the secret is memory-backed, it is not written to the node disk, but if it is an ordinary volume, it stays. The scope of this judgment extends to whether the disk is wiped when the node is retired.

Logs and dumps. As seen earlier, if the code that read the value puts it in an error message, the result is the same whether it is a file or an environment variable. And the memory dump a process leaves as it dies contains the loaded value as it is. You must also check whether dump collection is turned on, where the dumps accumulate, and who can read them.

What happens when it is updated. One of the values of a file mount is that it is updated without a restart, but if the application does not reread that file, it is of no use. This is the same as the earlier story about configuration injection, and it matters more for secrets. If the old key has been revoked but the process still holds the old value, from that moment every request fails.

In summary, secret management is not a problem of choosing where to store it but a problem of designing a lifecycle. More than where you put it, what decides real safety is who can read it, when it changes, and how you notice that it changed.

What you will do in the next lab

You confirm that a secret passed as an environment variable is visible as it is in docker inspect, then move the same value to a file mount and confirm it disappears from inspect. Then you change the value without restarting the container and measure the difference in rotation behavior between the two methods yourself, and finally write a simple secret detection script.