Keep the name, change where it comes from

In one line

There are two ways to get images into an air-gapped Kubernetes: put an archive into each node directly, or stand up an internal registry and tell the runtime about the mirror. A mirror changes none of the image names in your manifests. In exchange, the configuration file differs per runtime (registries.conf for podman, hosts.toml for containerd, registries.yaml for k3s), and every one of them must be told about the private CA separately.

Why this was needed

In an air-gapped cluster with ten nodes, you cannot copy a tar into every node each time you roll out a new version. So you set up an internal registry. But the common first attempt is to change the image names in manifests and charts to registry.corp.internal/.... Then you diverge from the charts in the outside repository, you have to redo the renaming every time upstream releases a new version, and a manifest pinned by digest breaks if the digest changes in the course of moving. A mirror avoids this problem by telling the runtime only "registry.k8s.io is pulled from here".

How it works

The internal registry. The CNCF distribution (the old docker registry) keeps images in storage.filesystem.rootdirectory of its configuration file, sets the address with http.addr, and sets HTTPS with certificate and key of http.tls. If you give proxy.remoteurl, it becomes a pull-through cache, but in that mode push does not work — in an air-gapped network there is no upstream, so you keep it as a regular registry and push the imports. On Ubuntu noble it comes as the docker-registry package (configuration /etc/docker/registry/config.yml, service docker-registry.service).

Moving while keeping digests. skopeo copy moves directly from registry to registry. --all moves the whole multi-architecture list, and --preserve-digests makes it fail if the digests cannot be preserved. If you move only one architecture, the digest of the list changes, and a manifest pinned as image@sha256:... cannot find the image on the mirror. skopeo sync moves many images at once from a YAML list.

podman: registries.conf. In containers-registries.conf(5), a [[registry]] states with prefix which names it applies to and with location the original location, and the location of [[registry.mirror]] is where to try first. With pull-from-mirror you can choose which of tags and digests to pull from the mirror. Fragment files placed in /etc/containers/registries.conf.d/ are read in alphabetical order and joined. The private CA goes at /etc/containers/certs.d/<호스트:포트>/ca.crt (the placeholder is the host and port).

containerd: hosts.toml. According to the containerd documentation, in /etc/containerd/certs.d/<레지스트리>/hosts.toml (the placeholder is the registry), the directory name is the original registry, server is the original address, and [host."..."] is the mirror to try first, where you write capabilities (pull, resolve, push) and ca. It tries the listed hosts in turn, and if all fail it falls back to server. In an air-gapped network, that fallback goes to the blocked outside, so the error message can be confusing. The location of the setting (config_path) that makes CRI look at this directory differs between containerd 1.x and 2.x, and ctr images pull --hosts-dir lets you test the same format in a single command.

k3s: registries.yaml. According to the k3s documentation, in /etc/rancher/k3s/registries.yaml you write the original registry and its endpoint under mirrors, and the tls (ca_file and so on) and authentication of the mirror address under configs, and after changing it you must restart k3s on each node. The documentation also says the default endpoint is always tried as a last resort. The k3s in this lab (v1.33.3+k3s1) uses containerd 2.0.

What it looks like in the field

I measured this in this lab VM. The list for registry.k8s.io/e2e-test-images/busybox:1.36.1-1 contains five Linux architectures and two Windows images, so when I moved two images with --all, the registry storage became 893MB (the Linux layers are only a few MB). When I put it on the VM root disk (2.4GiB), the disk filled up at the first image — the storage location of a mirror must be chosen by looking at the import volume. The digest of pause, moved as a whole list, was sha256:ee6521f2…, the same as outside, and a copy moved once more without --all changed to sha256:7c38f247…. k3s read registries.yaml and created /var/lib/rancher/k3s/agent/etc/containerd/certs.d/registry.k8s.io/hosts.toml with a header saying # File generated by k3s. DO NOT EDIT. — the same format as a hand-written hosts.toml. When I left the registry key owned by root, docker-registry.service died with status=1/FAILURE.

With a mirror, the imagePullPolicy: Always Pod that failed in the fde-delivery module comes up. Always asks the registry for the digest of the tag every time, and now the internal mirror accepts that question. Conversely, if you call for an image the mirror has never had, the runtime cannot find it on the mirror, falls back to the original registry, is blocked, and fails — even if the original registry address is printed in the error line, the cause is "it was not on the import list".

What you will do in the next lab

On the k3s VM, you stand up an internal registry over HTTPS with a private CA, and while connected you move two images keeping their digests. After blocking outbound traffic, you pull under the original name by three routes: podman (registries.conf), containerd (a hand-written hosts.toml), and k3s (registries.yaml). You bring up an Always Pod, and leave an import record checked against the registry and runtime digests.

Reference documents: distribution Configuration · Registry as a pull through cache · skopeo-copy(1) · containers-registries.conf(5) · containerd Registry Configuration (hosts.md) · K3s Private Registry Configuration