TT Lab
Get started
Learn Learning paths Courses

Building clusters with Kubespray and Terraform

What to carry into the air gap — files, images and Python

Continue in TT Lab

Goal

You make the list of files and images that kubespray downloads during installation, and using the inventory's offline variables, change it to point to the internal mirrors (registry.lab.internal:5000, http://files.lab.internal) by changing the list. You make the mapping table to hand to the person who will fill the mirrors, and an import bundle that includes the control node's Ansible. You do not create an actual air-gapped network or mirrors.

Why it matters

Most failures of air-gapped installation are "something left out." If you find out only after bringing things in through the import review that one file is missing, you have to go out and come back, and that round trip takes days. So you should have the installation tool itself say what it downloads (generate_list.sh), and while still outside, check whether that list is really for the version you will install this time and whether every address points to the internal network. The list version trap you meet in this module is one that I actually experienced while building this course — I wrote the version in the inventory but the list came out with a different version.

Steps

  1. In /opt/ks/kubespray/contrib/offline, run ./generate_list.sh with no arguments and copy the resulting temp/files.list and temp/images.list to /root/ks/offline/default/. Write to /root/ks/offline/default/count.json files and images (the number of lines in the two lists) and registries (the count by first path segment of images.list, for example {"quay.io": 3}).
  2. The inventory's kube_version is 1.35.8. Run ./generate_list.sh -i /root/ks/inventory/lab/inventory.ini once, and once more with -e kube_version=1.35.8 added, and check the kube-apiserver tag in images.list each time. Write to /root/ks/offline/trap.json with_inventory (the kube-apiserver tag of the first run), with_extra_var (the tag of the second run), and ignored_group (among the group_vars groups in which the inventory's kube_version is written, the name of the group that was not applied to the list playbook).
  3. Edit /root/ks/inventory/lab/group_vars/all/offline.yml to set registry_host: "registry.lab.internal:5000" and files_repo: "http://files.lab.internal", and, as in the air-gapped section of the kubespray documentation, set kube_image_repo, gcr_image_repo, docker_image_repo, quay_image_repo, and github_image_repo to {{ registry_host }}, and github_url, dl_k8s_io_url, storage_googleapis_url, and get_helm_url to {{ files_repo }}/<원래 도메인> (the placeholder is the original domain). The file must be in group_vars/all.
  4. Make the list again with ./generate_list.sh -i /root/ks/inventory/lab/inventory.ini -e kube_version=1.35.8 and copy the two files to /root/ks/offline/mirror/. Every line of files.list must start with http://files.lab.internal/, every line of images.list must start with registry.lab.internal:5000/, and the number of lines must equal that of step 1.
  5. Make the mapping table to hand to the person who fills the mirrors. Pair the original 1.35.8 list, made without applying the offline variables, with the mirror list from step 4 line by line, and write 원본<TAB>미러 (the placeholders are the original and the mirror, separated by a tab) one per line (every image in images.list) to /root/ks/offline/image-map.tsv.
  6. Add containerd_registries_mirrors to /root/ks/inventory/lab/group_vars/all/offline.yml so that the prefix registry.lab.internal:5000 uses the host http://registry.lab.internal:5000 for pull and resolve and skips TLS verification (skip_verify: true). ansible-inventory --host node1 must resolve this list as it is.
  7. Download the Python packages with kubespray's requirements.txt into /root/ks/offline/pypi/ (pip download), and check with pip install --dry-run --no-index --find-links /root/ks/offline/pypi -r requirements.txt whether an installation that does not use the internet resolves with that directory alone. Write to /root/ks/offline/pypi.json wheels (the number of downloaded files) and resolved (whether the dry-run succeeded, a boolean).

Notes

Start with the list of what it downloads

In /opt/ks/kubespray/contrib/offline, run ./generate_list.sh with no arguments and copy the resulting temp/files.list and temp/images.list to /root/ks/offline/default/. Write to /root/ks/offline/default/count.json files and images (the number of lines in the two lists) and registries (the count by first path segment of images.list, for example {"quay.io": 3}).

generate_list.sh extracts download_url and the image repos and tags from roles/kubespray_defaults/defaults/main/download.yml to make a template, and fills in the variables with a small playbook. It contains even those of CNIs and add-ons that are not turned on, so it is more generous than an actual installation. The first segment of an image list entry is the original registry you have to mirror.

The list quietly comes out with a different version

The inventory's kube_version is 1.35.8. Run ./generate_list.sh -i /root/ks/inventory/lab/inventory.ini once, and once more with -e kube_version=1.35.8 added, and check the kube-apiserver tag in images.list each time. Write to /root/ks/offline/trap.json with_inventory (the kube-apiserver tag of the first run), with_extra_var (the tag of the second run), and ignored_group (among the group_vars groups in which the inventory's kube_version is written, the name of the group that was not applied to the list playbook).

generate_list.yml runs with hosts: localhost. localhost is not in any group of the inventory, so it receives only the group_vars of all. If the list comes out with the wrong version, then with the imported files the installation stops the moment it starts with 'file not found' — and that in an air-gapped site.

Make the inventory point to the internal mirrors

Edit /root/ks/inventory/lab/group_vars/all/offline.yml to set registry_host: "registry.lab.internal:5000" and files_repo: "http://files.lab.internal", and, as in the air-gapped section of the kubespray documentation, set kube_image_repo, gcr_image_repo, docker_image_repo, quay_image_repo, and github_image_repo to {{ registry_host }}, and github_url, dl_k8s_io_url, storage_googleapis_url, and get_helm_url to {{ files_repo }}/<원래 도메인> (the placeholder is the original domain). The file must be in group_vars/all.

If, as the documentation's tip says, you put the original domain as the first directory under files_repo (files_repo/github.com/...), you can move URLs mechanically when filling the mirror. There are two reasons to put the variables in all — nodes that have only etcd also have to download, and the list playbook (localhost) also has to read these values.

Did the lists become entirely internal addresses

Make the list again with ./generate_list.sh -i /root/ks/inventory/lab/inventory.ini -e kube_version=1.35.8 and copy the two files to /root/ks/offline/mirror/. Every line of files.list must start with http://files.lab.internal/, every line of images.list must start with registry.lab.internal:5000/, and the number of lines must equal that of step 1.

If even one line still has the original domain, that file or image cannot be downloaded in the air-gapped network. If there are remaining lines, the domain in that line tells you which variable is missing.

Pairing the originals and the mirror

Make the mapping table to hand to the person who fills the mirrors. Pair the original 1.35.8 list, made without applying the offline variables, with the mirror list from step 4 line by line, and write 원본<TAB>미러 (the placeholders are the original and the mirror, separated by a tab) one per line (every image in images.list) to /root/ks/offline/image-map.tsv.

The offline variables are in group_vars/all, so make the original list with only -e kube_version=... and no inventory. The two lists come from the same template and have the same order. The key of the mirror design is that only the registry address changes and the path stays the same — if you change the path on the mirror side, this correspondence breaks.

Make nodes trust the internal registry

Add containerd_registries_mirrors to /root/ks/inventory/lab/group_vars/all/offline.yml so that the prefix registry.lab.internal:5000 uses the host http://registry.lab.internal:5000 for pull and resolve and skips TLS verification (skip_verify: true). ansible-inventory --host node1 must resolve this list as it is.

The kubespray documentation says the configuration method differs between containerd 2 and 1.7. In 2, containerd_registries_mirrors becomes /etc/containerd/certs.d//hosts.toml. It is a lab, so TLS is turned off, but in production the principle is to use certificates signed by the internal CA and not to turn on skip_verify.

Ansible must be brought in too

Download the Python packages with kubespray's requirements.txt into /root/ks/offline/pypi/ (pip download), and check with pip install --dry-run --no-index --find-links /root/ks/offline/pypi -r requirements.txt whether an installation that does not use the internet resolves with that directory alone. Write to /root/ks/offline/pypi.json wheels (the number of downloaded files) and resolved (whether the dry-run succeeded, a boolean).

In an air-gapped network you have to take the control node's Ansible as well. The kubespray documentation labels Python packages 'optional', which means the case in which the OS provides the same version, but this version requires an exact version such as ansible==12.3.0, so you usually have to bring it yourself. The Python version and the CPU architecture must be the same when you download and when you install — they are written in the wheel file names.