Pull images by their original names from an internal mirror

This lab runs on a real k3s

A single k3s (v1.33.3+k3s1) is running inside the VM, with skopeo, podman, and Ubuntu's docker-registry package (the CNCF distribution registry) installed. It takes a few minutes to come up at first. At first the VM can reach the internet (80/443) — up to step 3 is the connected import preparation, and in step 4 you block outbound traffic to turn it into an air-gapped network. Grading is done by an agent that comes in from outside to port 8899 of the VM, so blocking already-established connections and loopback cuts off grading.

Goal

Stand up an internal registry over HTTPS with a private CA and move images as they are by digest, then, with the outside blocked, make podman, containerd, and Kubernetes pull from the internal mirror without changing the image names.

Why it matters

If in an air-gapped network you change every image name in manifests to the internal registry address, the charts and manifests diverge from the outside and have to be fixed again every time you receive the next version. A mirror leaves the names as they are and tells the runtime only "that registry is pulled from here". The trap is that the configuration file differs per runtime — registries.conf for podman, hosts.toml for containerd, registries.yaml for k3s. And every one of them must be told about the private CA separately.

Estimated time is 60 minutes. The VM disappears when the session ends, so keep separately, before ending, any files you want to retain.

Steps

In /root/registry/pki, create a root (ca.crt and ca.key, CN Airgap Internal Root CA) and a registry.airgap.internal certificate (registry.crt and registry.key, with a SAN), and add 127.0.0.1 registry.airgap.internal to /etc/hosts.
With /etc/docker/registry/config.yml, make the internal registry serve HTTPS at 127.0.0.1:5000 with that certificate, and enable the service so that it also starts at boot. The registry service account must be able to read the key.
While connected, move registry.k8s.io/e2e-test-images/busybox:1.36.1-1 and registry.k8s.io/pause:3.10 to the same paths in the internal registry (registry.airgap.internal:5000/e2e-test-images/busybox:1.36.1-1 and .../pause:3.10), preserving their digests, and write one line each of 원래이름 다이제스트 (the placeholders are the original name and the digest) in /root/registry/digests.txt.
Block outbound traffic with /root/registry/egress.sh, which leaves only one set of rules even if run again (the chain AIRGAP-EGRESS, jumped to once each from OUTPUT and FORWARD, keeping loopback, established connections, 10.42.0.0/16, and 10.43.0.0/16).
podman: with /etc/containers/registries.conf.d/50-airgap-mirror.conf, set the mirror for registry.k8s.io to the internal registry and tell it the private CA, then pull with the original name registry.k8s.io/e2e-test-images/busybox:1.36.1-1.
containerd: write /root/registry/certs.d/registry.k8s.io/hosts.toml by hand and pull with k3s ctr -n airgap-ctr images pull --hosts-dir /root/registry/certs.d registry.k8s.io/pause:3.10.
Kubernetes: with /etc/rancher/k3s/registries.yaml, make k3s use the internal mirror, restart k3s, and then bring the Pod mirror-busybox (manifest /root/registry/mirror-busybox.yaml), with the original name and imagePullPolicy: Always, to Running in the namespace airgap.
Leave an import record in /root/registry/mirror-record.json — for each item of the images array, ref, mirror_ref, digest, and pulled_by (whichever of podman, ctr, and kubelet actually pulled it), and at the top level egress_blocked.

Notes

Certificate: the same procedure as in the private CA module (a SAN is required).
Registry response: curl --cacert /root/registry/pki/ca.crt https://registry.airgap.internal:5000/v2/_catalog
Digest: skopeo inspect --format '{{.Digest}}' docker://<이미지> (the placeholder is the image) · moving while preserving: skopeo copy --all --preserve-digests ...
The registry CA location for podman and skopeo: /etc/containers/certs.d/<호스트:포트>/ca.crt (the placeholder is the host and port)
The containerd configuration k3s created: /var/lib/rancher/k3s/agent/etc/containerd/
Common mistake 1: leaving the registry key readable only by root. The service runs as a dedicated account, cannot read the key, and dies — journalctl -u docker-registry.
Common mistake 2: moving without --all. The multi-architecture list shrinks to one architecture so the digest changes, and a manifest pinned by digest cannot find the image on the mirror.

Certificate for the internal registry

In /root/registry/pki, create a root (ca.crt and ca.key) and a registry.airgap.internal certificate (registry.crt and registry.key, with a SAN), and add the name to /etc/hosts.

The root is self-signed with CA:TRUE, and the server certificate is signed with the root while giving the SAN through an extension file. It is the same procedure as in the private CA module.

The internal registry over HTTPS

With /etc/docker/registry/config.yml, make the registry serve HTTPS at 127.0.0.1:5000 with that certificate, and enable the service so that it also starts at boot.

In the http section of the configuration, give the address and tls (the certificate and key paths). The service runs as a dedicated account, so the key must be placed where, and with permissions such that, that account can read it. If it does not start, look first at journalctl -u docker-registry.

Move while preserving digests

Move the two images to the same paths in the internal registry preserving their digests, and write one line each of 원래이름 다이제스트 (the placeholders are the original name and the digest) in /root/registry/digests.txt.

skopeo copy moves directly from registry to registry. Only if you move the whole multi-architecture list is the digest the same as outside. If you put the internal registry's CA at /etc/containers/certs.d/host:port/ca.crt, skopeo and podman both use it.

Block outbound traffic to make an air-gapped network

Block outbound traffic with /root/registry/egress.sh, which leaves only one set of rules even if run again.

If you create a new chain and jump to it at the very front of OUTPUT and FORWARD, then a flush-and-refill approach gives the same shape however many times you run it. Keep the grading agent and the Kubernetes ranges open first.

podman: a mirror in registries.conf

With /etc/containers/registries.conf.d/50-airgap-mirror.conf, set the mirror for registry.k8s.io to the internal registry, and pull busybox with the original name.

In registries.conf, a [[registry]] states with prefix which names it applies to, and a [[registry.mirror]] states the location to try first. Do not change the name. The private CA uses the certs.d you placed in step 3 as it is.

containerd: hosts.toml by hand

Write /root/registry/certs.d/registry.k8s.io/hosts.toml and pull with k3s ctr -n airgap-ctr images pull --hosts-dir /root/registry/certs.d registry.k8s.io/pause:3.10.

The directory name is the original registry, server inside the file is the original address, and [host."..."] is the mirror to try first. For the mirror, write the pull and resolve capabilities and the CA path.

Kubernetes: registries.yaml and an Always Pod

With /etc/rancher/k3s/registries.yaml, make k3s use the internal mirror and restart it, then bring the Pod mirror-busybox (manifest /root/registry/mirror-busybox.yaml), with imagePullPolicy: Always, to Running in the namespace airgap.

Under mirrors write the original registry and endpoint, and under configs write the tls ca_file of the mirror address. As the k3s documentation says, you must restart k3s after changing it. Leave the image name as it originally was — even with Always, it comes up if the tag can be resolved on the mirror.

Check the import record against the registry and the runtimes

In /root/registry/mirror-record.json, leave ref, mirror_ref, digest, and pulled_by for each image, and egress_blocked at the top level.

Get the digest by asking the internal registry, and for pulled_by write whichever of podman, ctr, and kubelet actually pulled that image. Do not copy the values by hand; collect them from command output.