TT Lab
Get started
Learn Learning paths Courses

Policy as Code

Image reference policy: tags, digests, and what you layer on top

Continue in TT Lab

In one sentence

A tag is a name tag that can be moved, and a digest is a hash of the content and cannot be moved, and almost all of image policy is a corollary of this one sentence.

Why it was needed

There is a kind of deployment incident that is the hardest to reproduce. What ran well yesterday runs differently today. The manifest is the same. The code is the same. But it is different. When the cause is the image, it is usually because the same tag came to point at different content.

The official Kubernetes documentation states this property briefly. A tag can be moved to point to a different image, but a digest is fixed. A digest is a hash of the image content and is immutable. So app:v1.2.3 is not a promise but a convention. A registry does not stop you from pushing a new image onto the same tag (unless it is a registry with immutable tags turned on).

latest is the extreme of this problem. If you do not write a tag, Kubernetes treats it as meaning latest. And one more fact is attached. If you omit the tag or leave it at :latest and do not write imagePullPolicy, that value automatically becomes Always. That is, every time the Pod restarts, it pulls whatever the registry gives at that moment. That is why yesterday and today differ.

How it works

An image reference has three pieces.

registry.example.com/team/app:v1.2.3@sha256:1ff6c18f...
└── 레지스트리 ──┘└ 이름 ┘└ 태그 ┘└─── 다이제스트 ───┘

What happens if you write all three? The documentation answers precisely. If you write both a tag and a digest, only the digest is used when pulling. That gives the form used in practice — you leave the tag for people to read, and pin the real identity by digest.

The default of imagePullPolicy is also decided by the shape of the reference.

Reference When imagePullPolicy is not written
A digest is written IfNotPresent
The tag is :latest Always
No tag Always
Any other tag IfNotPresent

What a digest guarantees and what it does not. There is a lot of misunderstanding here.

It guarantees one thing. Content identity. If you pull the same digest twice, you get the same bytes. Whichever registry you pull from, and however many years later, it is the same.

It does not guarantee three things.

What fills in the latter two is signatures (who vouches) and provenance attestations (how it was made). The provenance specification of SLSA defines the standard format for the latter — it leaves the builder, the source repository and commit, and the build parameters as a signed document. Digest pinning is the floor on which these upper layers stand. This is because what points to what to sign is, in the end, a digest. This lab environment has no signing tool such as cosign, so signatures and provenance attestations are covered only as concepts.

Two places to apply policy.

The inside (admission). It looks at the image string in the Pod spec. Allowed registry prefixes, no latest, requiring a digest. There is an important pitfall here. An image string check is a string check, so it is easily bypassed. If you put registry.example.com in the allowlist and do only a prefix check, registry.example.com.evil.net/x passes. You must cut exactly at the host boundary (the part before /) and compare. Also, containers are not only in containers — policies that leave out initContainers and ephemeralContainers are common.

The outside (build and repository). FROM in a Dockerfile, the image values in manifests and Helm charts, and the image list in build outputs. Admission is the last line of defense, and this is a place people can fix. If FROM alpine:3.20 is in the repository, a different base can come in every time you build. If you put the same rule on both sides, developers find out in the PR, and the cluster blocks what might have leaked through.

What you see in the field

First, the day a tag was overwritten. CI pushed app:v1.4.0 twice. The Pods in the cluster did not change, but only the Pods that restarted that day came up with the new content. It became a state where different code ran in each Pod of the same Deployment, and the cause is visible only by comparing digests. If you extract status.containerStatuses[].imageID with kubectl get pod -o jsonpath, you get the digest actually pulled.

Second, a rollback that cannot roll back. A team that deployed only by tag decided "let's go back to the previous version," but that tag had already been overwritten. The thing to go back to was gone. If they had deployed by digest, the address to go back to would still remain.

Third, the cost of digest pinning. Pinning is not free. Base image security patches do not follow automatically, so there must also be a procedure for periodically raising the pinned values. Without that procedure, a few months later you end up in a state of "perfectly reproducible but entirely vulnerable." Pinning and updating are a pair.

Fourth, holes in the allowlist. Incidents repeat in which a policy written with prefix comparison let through an external host with a similar name. When writing the rule, always put both samples that must pass and samples that must be blocked, and in particular put in attack samples with similar names.

References

What you will do in the next lab

You pick apart image references yourself and check them with policy. You pull the manifest and digest out of an offline image bundle to confirm what the tag and the digest each point to, and copy the same content under a different tag to see that the digest stays the same. Then you write three rules, allowed registries, no latest, and requiring a digest, and put in a sample that slips past the prefix comparison with a similarly named host and a sample whose violation is only in initContainers, finding and plugging the holes in the rules yourself.