Closed-network k0s multi-node actual measurement 1: How to prove that the internet is blocked, and 1 node

Status: Actual measurement (2026-10-31). These are values measured by setting up k0s v1.37.1+k0s.0 inside a TT Lab practice VM (Ubuntu 24.04 on KubeVirt). The 1-node scenario is the value from a run executed from start to finish using the fixed correct answer key, and sealing and importing were repeated in the exact same manner across all 1-, 2-, and 3-node scenarios. Node addresses have been replaced with example ranges (192.0.2.0/24). Writing convention: Actual measured data is the default; unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. Starting with the Conclusion
• 2. How the Closed Network Was Built
  • 2.1 Trap: The chain name fwd is a reserved word
  • 2.2 Trap: "Allowing only outgoing traffic to the internal network NIC" kills the k0s CNI
• 3. What to Bring In
• 4. 1 Node: k0s install controller --single
  • 4.1 When to Use
  • 4.2 Actual Measurement
  • 4.3 Limitations of --single
• 5. Proof That It Was a Closed Network
• 6. Summary

1. Starting with the Conclusion

Servers in places like public institutions, financial companies, and factories often cannot reach the internet. To install Kubernetes in such environments, you must bring in the necessary files from the outside (importing). This much is widely known, but doing it in practice leaves trickier questions: Did it really not receive anything from the outside during installation? If you practice in a lab that can reach the outside, you won't even notice if someone secretly downloads a missed image from the internet. That loophole reveals itself only on the day you enter the actual site.

• A closed network is harder to prove than to build. This test verified it in three layers: external HTTPS is rejected in 0.014 seconds (curl rc=7), unimported images result in ErrImagePull , and after inserting a log rule before the sealing rules, there were 0 outbound TCP connections made by the bastion server while k0sctl performed installation and backup.
• k0s requires few things to be imported. A single binary (271,493,593 bytes) and a single bundle containing 13 system images (417,884,160 bytes) are enough to set up a node. It took 68 to 72 seconds for the bastion server to receive them, verify the hashes, and even build the app image bundle.
• For 1 node, it took 144 seconds from installation to node Ready, and it took an additional 49 seconds for 4 system pods to become Ready. However, the node set up with --single cannot have nodes attached to it later.
• We fell into two traps in the sealing script. The nftables chain name fwd was a reserved word, causing the entire table to be rejected, and the rule written as "Allow only outgoing traffic to the corporate network NIC" killed the k0s CNI. The methods and reasons for fixing both are described in Section 2.

How workers behave when the controller dies in a 2-node setup is covered in Part 2, and 3-node HA along with Rook-Ceph and CloudNativePG running on top of it are covered in Part 3.

2. How the Closed Network Was Built

Since we did not have separate closed network equipment for experiments, we blocked external connections using nftables inside a VM. There are two reasons why we did not block it using the network policies of the outer cluster wrapping the VM: that policy did not have a mode like "no internet", and learners needed to be able to read and fix the sealing table themselves for it to be a practical exercise. Instead, there is a clear limitation. Since root inside the VM can delete this table, the sealing is a learning and testing mechanism, not a security boundary. During the test, no one deleted it, and we re-measured at each step whether the sealing table remained intact and whether it was still blocked.

The configuration is as follows.

• Only one bastion server uses the internet. It receives files from the official release, compares them with the sha256sums.txt value uploaded along with the release (deletes and re-downloads if they differ, and stops if they differ three times), creates an image bundle, and then seals itself as well. This is because, after importing is finished, the bastion server must also be unable to reach the outside to prove that nothing was "secretly downloaded during installation".
• Nodes are sealed from the first boot. Not a single package was added to the OS image. Necessary files are passed to the corporate network by the bastion server (via scp and k0sctl's SSH). Learners verify them once more on the node using sha256sum -c . The bastion server's verification checks "whether the item from the internet is the official file," and the node's verification checks "whether it was altered during transit."
• Each node has two NICs. They are the NIC with the default route (enp1s0, the path leading outside) and the corporate network NIC used exclusively between nodes (eth1). This configuration triggers the second trap in Section 2 and the join failure in Part 2.

The core of the sealing script is as follows. Only items to keep are written, and other new connections are reject .

DNS=$(awk '/^nameserver/{print $2}' /run/systemd/resolve/resolv.conf | paste -sd, -)
VXLAN_PEERS=...   # 시험 환경의 세션망 상대(바깥 클러스터의 주소라 생략)
nft -f - <<NFT
table inet airgap {
  chain airgap_out {                       # 이름에 fwd 를 쓰지 않는다(2.1)
    type filter hook output priority 0; policy accept;
    oifname "lo" accept
    ct state established,related accept
    oifname "eth1" accept                  # 사내망 NIC
    fib daddr type local accept            # 목적지가 이 노드 자신의 주소(2.2)
    ip daddr 192.0.2.0/24 accept        # 사내망 대역
    ip daddr $VXLAN_PEERS udp dport 4789 accept   # 시험 환경의 세션망을 실어 나르는 VXLAN
    ip daddr { $DNS } meta l4proto { tcp, udp } th dport 53 accept
    udp dport 67 accept                    # DHCP
    ip daddr { 172.20.0.0/16, 172.21.0.0/16 } accept   # k0s 파드·서비스 대역
    counter reject
  }
  chain airgap_fwd {
    type filter hook forward priority 0; policy accept;
    ct state established,related accept
    oifname "eth1" accept
    ip daddr { 172.20.0.0/16, 172.21.0.0/16, 192.0.2.0/24 } accept
    counter reject
  }
}
NFT

Reasons are noted for a few items.

• It is reject, not drop. drop makes the sender wait until a timeout. reject immediately returns a refusal, so blocked attempts fail in just 0.014 seconds, and where and what was blocked is immediately recorded in logs and counters. Since firewalls in actual sites are often drop, it is worth remembering that the same attempt could hang for tens of seconds at an actual site.
• DNS is left open. The corporate network in the test environment is a network where VMs are connected via VXLAN, and it re-resolves neighboring external addresses via DNS every 30 seconds. If DNS is blocked, the corporate network itself collapses. Therefore, this network is a network where "names resolve but connections are refused" (Actual measurement: getent hosts github.com returns an address, while curl returns rc=7). Since actual sites usually have an internal corporate DNS, we considered that this difference does not change the test results.
• Sealing is re-applied on every boot via a systemd unit (Type=oneshot, RemainAfterExit=yes). The entire behavior after reboot was not measured (Unmeasured).

2.1 Trap: Chain name fwd is a reserved word

The initial sealing table named the chain out·fwd . nft -f rejected it like this (Ubuntu 24.04, nftables 1.0.9).

Error: syntax error, unexpected fwd, expecting string or last

And subsequent lines were followed by errors such as unexpected type, unexpected policy . Because a single initial error caused the parser to lose its way, looking only at the lower part of the output makes the type filter hook forward line appear incorrect, making it difficult to find the cause. You must look at the topmost error.

The cause lies in the fact that fwd is a statement keyword in nftables. The nft manual's FWD STATEMENT section explains that fwd to device is a statement that sends packets to another interface (used only in ingress and egress hooks of the netdev family), and the lexical rules of the same manual state that "Identifiers that coincide with keywords must be enclosed in double quotes." Instead of quotes, we changed the name to airgap_out·airgap_fwd . Whether "fwd" enclosed in quotes is accepted in this version was not tested (Unmeasured).

There is another reason why the damage was severe. The sealing ran inside the VM's first boot recipe (cloud-init, set -e), and when it failed there, the entire recipe stopped, so the "VM Ready" marker was never printed. The testing tool announced failure only after waiting 1,200 seconds. Now, we check the syntax first using nft -c -f(check only, do not apply) before inserting it.

2.2 Trap: "Allowing only outgoing traffic to the internal network NIC" kills the k0s CNI

The initial sealing table left only oifname "eth1" accept . We thought it would be sufficient to allow outgoing traffic to the corporate network NIC and reject the rest. However, when we installed 1 node, the kube-system pods ultimately did not become Ready. Tracing it revealed the following chain of events.

1. k0s's default CNI, kube-router, runs in the host network and calls the API via kubernetes the Service address (172.21.0.1:443).
2. kube-proxy DNATs this address to the node's own API address (e.g., 192.0.2.3:6443).
3. However, the log recorded in the OUTPUT filter hook was OUT=enp1s0 SRC=<NAT 주소> DST=192.0.2.3 DPT=6443 . The destination had changed to the corporate network address, but the outgoing NIC appeared to be the default route NIC. Therefore, it was rejected without hitting oifname "eth1" .
4. kube-router left failed to synchronize cache: 1m0s timeout and restarted every minute, failing to write the CNI configuration file, causing all pods to get stuck in no IP ranges specified .

We could not verify up to the kernel's route recalculation order why the NIC appeared that way in step 3. We presume that the exit initially determined by the Service address route (default route) remained at the filter hook stage (Estimated). The fix is definite. By adding the rule fib daddr type local accept, meaning if the destination is this node's own address, allow it regardless of the NIC, kube-router connected and all 4 system pods became Ready. For cases going to other nodes' APIs, ip daddr 192.0.2.0/24 accept is also left in place.

The lesson is this: Closed network rules should be written based on "where it goes," not "which NIC it goes out of." On Kubernetes nodes, packet appearances change across hooks due to DNAT.

3. What to Bring In

The version was fixed to the latest stable release as of the test day (2026-10-31).

The 13 items in the airgap bundle matched the official list (airgap-images-linux-amd64.txt): pause, konnectivity agent, kube-proxy, coredns, metrics-server, pushgateway, CNI plugin, kube-router, the Calico trio, envoy for NLLB, and traefik, all of which are located under quay.io/k0sproject/ .

The airgapped installation of k0s works like this. According to the official documentation, if you place the OCI bundle file in the worker's /var/lib/k0s/images/ , k0s loads it into containerd when starting the worker. Also, if you specify spec.images.default_pull_policy: Never in the cluster configuration, the components managed by k0s's imagePullPolicy becomes Never , so it does not even attempt to pull from the internet (measured: coredns policy is Never). You must note that this setting only applies to system pods launched by k0s. Apps you deploy directly follow their own policies, so you either include the app images in the bundle together (this time's apps-bundle.tar) or provide an internal registry as in part 3. The app bundle was created on the import server as k0s airgap bundle-artifacts and took 7 seconds. Because this command produces a different result file hash every time it is run, only for the app bundle we record the newly generated value instead of the official hash and compare it on the nodes.

The time on the import server side was as follows (1- and 2-node scenarios, measured):

4. Single Node: k0s install controller --single

4.1 When to Use

This is a configuration where the controller and worker are placed together on a single node. It is suitable when you do not need fault tolerance, such as for edge devices, demos, or CI disposable clusters, and want to keep imports lightweight. You just move one binary and one bundle and run two commands.

scp k0s-v1.37.1+k0s.0-amd64 node01:/usr/local/bin/k0s
scp k0s-airgap-bundle-*.tar apps-bundle.tar node01:/var/lib/k0s/images/
ssh node01 'cat > /etc/k0s/k0s.yaml' <<'YAML'
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata: {name: k0s}
spec:
  api: {address: 192.0.2.3, sans: [192.0.2.3, node01]}
  network: {provider: kuberouter, podCIDR: 172.20.0.0/16, serviceCIDR: 172.21.0.0/16}
  images: {default_pull_policy: Never}
  telemetry: {enabled: false}
YAML
ssh node01 'k0s install controller --single -c /etc/k0s/k0s.yaml --kubelet-extra-args=--node-ip=192.0.2.3 && k0s start'
ssh node01 k0s kubeconfig admin > ~/.kube/config

We explicitly defined three things in the configuration:

• api.address and --node-ip were pinned to internal network addresses. If not specified, k0s uses the first non-loopback address found on the node, which on this dual-NIC node happens to be the address of the default route NIC. It was not noticeable with a single node, but as nodes increased in part 2, it immediately manifested as join failures.
• We changed the service CIDR. The default k0s service CIDR (serviceCIDR, configuration documentation) is a /12 range starting with 10, and the DNS server address given to this test's VM happened to fall within it. When they overlap, the node mistakes packets destined for its own DNS server as Kubernetes Services. You must always check whether your corporate DNS and registry addresses in the field fall within the default service CIDR or default pod CIDR (podCIDR) by comparing them with the configuration documentation defaults before installation.
• telemetry.enabled: false was provided. It would be rejected in an airgapped environment anyway, but this is to prevent rejected attempts from cluttering logs and counters.

4.2 Measurement

4.3 --single limitation

--single is convenient but an irreversible choice.

• The installation documentationstates --single that this "turns off features required for multi-node." Therefore, a cluster built this way cannot have nodes added later. If you have even the slightest plan to scale up, it is better to separate the controllers and workers from the start or controller --enable-worker start with (for this test, --enable-worker a single-node configuration was not measured).
• CPLBand NLLB documentation both state that --single cannot be used with . The high availability devices from Part 3 cannot fit into this configuration.
• Naturally, if the node dies, both the API and workloads stop together. Failure testing was not performed.

5. Proof of Being an Air-Gapped Network

Simply succeeding at installation does not prove it was an air-gapped network. This time, three things were measured.

1. The outside is blocked. When attempting HTTPS using github.com·quay.io·registry-1.docker.io on all nodes and the import server, the HTTP code is 000, curl exit code 7 (connection refused), taking 0.014 seconds. Every time a lab step was graded, we re-measured to ensure it was still blocked.
2. What is not imported truly cannot be used. If you create a pod with an docker.io/library/redis:8.2-alpine that has not been imported, dial tcp <docker.io 주소>:443: connect: connection refused results in ErrImagePull . At the same time, pods started with imported images were all in the Running state. The seal counter for the single-node run recorded 262 refusals of attempts to reach the outside.
3. After importing, nothing attempted to go outside. Items 1 and 2 only prove that "it was blocked." To demonstrate that "it was unnecessary," we need to look at who tried to go out. Inserting a logging rule right before the sealing rule revealed that the only thing the node tried to send outside was the intentionally triggered redis pull from step 2 (8 addresses on docker.io, TCP 443). While installing and backing up with k0sctl on the import server, there were 0 outgoing TCP connections, and the only blocked traffic was UDP 123 for time synchronization (chrony).

The reason why point 3 is important is that firewalls in field environments are typically configured to drop packets. External attempts (such as telemetry and update checks) that fail immediately with a reject in a lab environment can cause installation to stall in the field while waiting for timeouts. Demonstrating in advance that there are zero external attempts eliminates that concern.

Additionally, k0sctl v0.33.1 does not have the --disable-telemetry·--disable-upgrade-check flag. If you copy it based on an older post, flag provided but not defined causes apply to finish in 0 seconds. Even without the flag, external connection attempts were zero.

6. Conclusion

The actual measurements in this post can be reproduced in the first module, "1 Node," of TT Lab's hands-on course "Air-gapped k0s: 1, 2, and 3 Nodes with Ceph and PostgreSQL" (site path /course/k0s-airgap-multinode), following the exact same board and import manifest. The import server sealing and --single installation are performed on actual VMs, with the grader verifying the results at each step.

TT's Word. Locking the door is not the end. You have to count even the footprints (logs) to see if there was a friend trying to sneak out before you can say "truly no one went out." And if you hang a nameplate that says "fwd," the whole house won't open!