Three ways to bring images into an air-gapped cluster

In one line

A node in an air-gapped cluster cannot get images by itself. Someone has to put the image into the runtime store, and even after it is put in, the Pod's pull policy must not go out to ask the registry. For this job, k3s offers three routes: a private registry, a per-node image archive, and an embedded registry mirror.

Why this was needed

Because of the customer's security policy, the production cluster does not reach the internet. When you apply the manifest the day before delivery, the Pod goes through ErrImagePull and stays in ImagePullBackOff. It is a state you have never seen in the connected development environment. There, the kubelet fetched the image on its own. In an air-gapped network, that "on its own" does not exist. The FDE has two jobs. On the connected side, decide what to take and carry it over, and on the receiving side, prove that it is the same as before it was carried, and then put it into the runtime.

How it works

A Pod looks at the node's runtime store, not the registry. When it starts a container, the kubelet follows the pull policy. According to the Kubernetes documentation, IfNotPresent pulls only if it is not local, Never does not go to pull, and Always makes the runtime contact the registry every time a container starts, resolve the tag to a digest, and pull only the layers it does not have. If you do not write a policy, it becomes Always when the tag is :latest or absent, and IfNotPresent for any other tag. When a pull fails, the kubelet retries at exponentially growing intervals, and the interval stops at 300 seconds. That is why even right after an import, a Pod can sit still for several minutes.

The three routes of k3s. The air-gap installation documentation of k3s divides the ways to bring in images into three.

사설 레지스트리       /etc/rancher/k3s/registries.yaml 로 미러·인증을 설정. 바꾸면 노드마다 k3s 재시작
노드에 직접 배포      /var/lib/rancher/k3s/agent/images/ 에 이미지 tar 를 둔다
내장 레지스트리 미러  한 노드의 containerd 저장소에 있는 이미지를 다른 노드가 받아 간다

The private registry documentation has a sentence that is easy to miss. containerd has an implicit default endpoint for all registries, and even if you write another endpoint in registries.yaml, that default endpoint is always tried as a last resort. Also, an image name that does not specify a registry is regarded as docker.io for historical reasons. This is what decides where a manifest written short like nginx:1.25 goes to ask in an air-gapped network.

When is the image directory read? The air-gap documentation says that k3s imports the archives in this directory every time it starts. It is to always restore them even if an image was deleted or cleaned up, and in exchange the kubelet does not start until all archives have been processed, so startup gets slower. That is why, from the May 2025 releases (v1.33.1+k3s1, v1.32.5+k3s1, and so on), you can use conditional import, which skips archives whose size and modification time are the same, by putting a .cache.json in the directory. In this case, a deleted image has to be imported again by hand with ctr image import or you have to touch the archive. Meanwhile, the image import documentation says that the feature of importing even a tar put in while running is from the January 2025 releases (v1.32.0+k3s1, v1.31.5+k3s1, v1.30.9+k3s1, v1.29.13+k3s1), and that before that it imported only at boot. That is why you must first check the k3s version of the customer cluster. Do not get confused either by the fact that if you put a text file listing image names, one per line, in the same directory, it conversely becomes a feature for pre-pulling online.

An import ends with a hash. Before carrying it over, you record the archive's sha256, and on the receiving side you compare with the same command before putting it in. After putting it in, you check whether the image ID (the config hash) that the runtime reports is the same as the config hash inside the archive. Only when these two connect can you say "what is running in the customer cluster is the very thing we verified".

What it looks like in the field

We measured it this way on the lab VM (measured, v1.33.3+k3s1, containerd v2.0.5-k3s2). When we blocked outbound traffic with REJECT, the curl status code for https://registry.k8s.io/v2/ changed from 401 to 000, and a new Pod went through ErrImagePull to ImagePullBackOff within 2 seconds of being created. The cause line of the event begins with failed to resolve reference and says that the request Head "https://registry.k8s.io/v2/e2e-test-images/busybox/manifests/1.36.1-1" ended with connection refused. Before even receiving the layers, it was already blocked at the first request that asks the registry in order to resolve the tag to a digest. When we put the busybox archive (4.5MB) in with k3s ctr -n k8s.io images import, saved and the manifest digest were printed in the output. However, this output has progress indicators mixed in, so the image name saved to a file remained in a form with the hyphen and the tag missing. It is safer to reconcile an import record by the digest rather than the name. When we copied the nginx archive (17MB) into the image directory of the running k3s, Importing images from and Imported images ... in 1.094242885s were printed in the k3s log, and it appeared in the runtime store without a restart. When we made an imagePullPolicy: Always Pod with the busybox we had imported, even though the image was in the store, ErrImagePull occurred from the very same HEAD request failure.

The second most common incident happens after the import is finished. Someone wrote imagePullPolicy: Always in the manifest, or left the tag as latest. The image is in the store, but the kubelet goes out to ask the registry and fails on the blocked network. At this point, the container imagePullPolicy of a Pod cannot be changed after it is created, so you have to create the Pod again. If you briefly lift the block to get past it, it works in a connected environment, but the customer's air-gapped network has no such option.

There is also a trap when imitating an air-gapped network. The VM in this lab is operated through the grading connection coming in from outside (8899), so if you block outbound traffic without keeping connections already established, the loopback, and the cluster's internal ranges alive, the grading and the API are cut off together. You have to gather the rules into a single chain so that it ends up the same no matter how many times you run it, and only then is it safe for the person who takes over to run it again.

What really matters in practice

• Write the import list not by tag but by the whole image name (including the registry). A short name resolves to docker.io.
• Record the archive hash before carrying it, compare it before putting it in, and compare it with the runtime image ID after putting it in.
• The containerd namespace Kubernetes looks at is k8s.io. An image put in another namespace does not exist for the Pod.
• Check the manifest's pull policy and tag before the import. Always and latest reserve a failure in an air-gapped network.
• Whether an image directory import works even while running depends on the k3s version. Leave the version in the record as well.
• Do not copy an import record by hand; build it by extracting from the runtime and the Pods.

What you will do in the next lab

On a connected k3s VM, you fetch two images as archives and record the hashes, and block outbound traffic with iptables to make an air-gapped network. You leave as evidence that, while it is blocked, a Pod with a new image fails to pull, then import by two routes, k3s ctr images import and the image directory, and start the Pods. You confirm that an Always Pod fails with an image you have imported, fix the policy, and then leave the import record reconciled against the runtime.

References: K3s Air-Gap Install · K3s Import Images · K3s Private Registry Configuration · Kubernetes Images(imagePullPolicy)