Closed-network k0s multi-node measurement 2: While 2 nodes and the controller stopped for 90 seconds, workers kept responding

Status: Measured (2026-10-31). These are values measured by setting up k0s v1.37.1+k0s.0 with k0sctl v0.33.1 inside a TT Lab practice VM. For the 2-node scenario, we created new VMs and ran it twice from scratch, recording both times in the table. The method for creating the closed network is the same as in Part 1. Node addresses have been rewritten with example ranges (192.0.2.0/24). Writing rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. Starting with the Conclusion
• 2. Why 2 Nodes Mean "1 Controller + 1 Worker"
• 3. Setting Up with k0sctl
  • 3.1 Pitfall: Join Token Pointed to the Wrong Address
• 4. Stopping the Controller for 90 Seconds
  • 4.1 Measurement Method
  • 4.2 Results
• 5. Adding and Removing Workers
• 6. Backup
• 7. Summary

1. Starting with the Conclusion

• Two nodes do not provide control plane high availability. With 2 etcd members, the majority is 2, so writes stop even if just one dies. Therefore, the 2 nodes were set up as 1 controller + 1 worker.
• Even if the controller dies, worker services continue. While the controller was stopped for 90 seconds and measured 18 times at 5-second intervals, the API failed all 18 times, but the workers' nginx returned 200 all 18 times. When the controller was turned back on, the API recovered in 3 to 4 seconds.
• What stops is "making changes". Deployments, scaling, and rescheduling when a worker dies all go through the controller. The danger of a 2-node configuration is not service downtime, but that nothing can be done in the meantime.
• When a node has two NICs, k0sctl's privateAddress alone is not enough. The API address was determined by the default route NIC, causing worker joins to fail after 172 seconds, which we solved with spec.api.onlyBindToAddress: true .
• Other measurements: 2-node installation 1 minute 58 seconds to 2 minutes 44 seconds, adding a worker 1 minute 42 seconds to 2 minutes 31 seconds, removal via drain and reset: true took 24 to 39 seconds, k0sctl backup took 16 to 32 seconds (416 to 440 KB). Restoring from a backup was not measured (Unmeasured).

2. Why 2 Nodes Mean "1 Controller + 1 Worker"

Multi-controller configurations in k0s use etcd. etcd requires a majority of members to agree to accept writes. Translating the table from the etcd FAQ, it looks like this.

Two etcd members tolerate the same number of failures as 1, while only increasing the number of servers that can die. Therefore, "both as controllers" is actually worse than a 1-controller setup. If you take down one node for inspection, the API on the remaining node also stops accepting writes. The actual choice for 2 nodes is 1 controller + 1 worker , and the remaining question is whether the controller should also double as a worker.

• If it does, you gain one more node to place workloads on.
• However, memory pressure from workloads will destabilize the API and etcd, and taking down that node for inspection causes you to lose both the control plane and workloads.

This time, we did not combine them. Combining roles is used when there are three nodes (Part 3). There is a series of articles measuring the same problem with databases and search engines titled Why HA Doesn't Work with 2 Servers .

3. Setting Up with k0sctl

We installed a single node manually, but for 2 or more nodes, we use k0sctl. From the transfer server, we declare everything in a single file: uploading binaries and bundles to each node via SSH, writing configurations, generating join tokens, and attaching workers. The configuration used in the closed network is as follows.

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata: {name: k0s-two}
spec:
  hosts:
  - role: controller
    ssh: {address: 192.0.2.3, user: root, keyPath: /root/.ssh/id_ed25519}
    hostname: ctrl01
    privateInterface: eth1
    privateAddress: 192.0.2.3
    uploadBinary: true
    k0sBinaryPath: /opt/airgap/bin/k0s-v1.37.1+k0s.0-amd64
  - role: worker
    ssh: {address: 192.0.2.4, user: root, keyPath: /root/.ssh/id_ed25519}
    hostname: worker01
    privateInterface: eth1
    privateAddress: 192.0.2.4
    uploadBinary: true
    k0sBinaryPath: /opt/airgap/bin/k0s-v1.37.1+k0s.0-amd64
    installFlags: [--kubelet-extra-args=--node-ip=192.0.2.4]
    files:
    - {name: k0s-airgap-bundle, src: /opt/airgap/images/k0s-airgap-bundle-v1.37.1+k0s.0-linux-amd64.tar, dstDir: /var/lib/k0s/images/, perm: "0644"}
    - {name: apps-bundle, src: /opt/airgap/images/apps-bundle.tar, dstDir: /var/lib/k0s/images/, perm: "0644"}
  k0s:
    version: v1.37.1+k0s.0
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata: {name: k0s}
      spec:
        api: {onlyBindToAddress: true}     # NIC 가 둘이면 필요하다(3.1)
        network: {provider: kuberouter, podCIDR: 172.20.0.0/16, serviceCIDR: 172.21.0.0/16}
        images: {default_pull_policy: Never}
        telemetry: {enabled: false}

We will only point out lines that are meaningful in a closed network.

• uploadBinary: true + k0sBinaryPath. If you do not provide a path, k0sctl will try to download GitHub releases from the transfer server. You must point to the pre-downloaded binaries so it doesn't reach outside. When you provide a path, k0sctl leaves a warning that "this host does not check versions", which is harmless.
• files is placed only on workers src . Pure controllers do not run pods, so they do not need image bundles. If is a URL, it becomes another path reaching outside, so we use a local path.
• installFlags 's --node-ip. causes kubelet to announce itself with the corporate network address. k0sctl adds this flag automatically when privateAddress is present (host.go), but we wrote it out so it is immediately visible in the file what is being used.

When we ran k0sctl apply on the transfer server, it finished in 1 minute 58 seconds and 2 minutes 44 seconds(reported by k0sctl) for each of the two runs. Even with the same configuration, it varied this much between runs, and we did not look into the cause separately. For pure workers, k0sctl waits until Ready with "waiting for node to become ready", so when apply finished, the worker was already Ready. The fact that this waiting does not exist for controller-cum-workers will come up again in Part 3.

3.1 Pitfall: Join Token Pointed to the Wrong Address

At first, we wrote only onlyBindToAddress without privateInterface·privateAddress . We thought it would work because we explicitly specified to use the corporate network address, but the worker join hung for 172 seconds and ended like this.

failed to connect to kubernetes api using the join token ...
The connection to the server <NAT 주소>:6443 was refused

<NAT 주소> is the default route NIC address of these test VMs. Since KubeVirt's masquerade network gives the same NAT address to all VMs, when a worker goes to 6443 at that address, it goes to itself. There is no API there, so it is rejected.

We verified why that address went into the token from the k0sctl source code. configure_k0s.gouses privateAddress for the etcd peer address (spec.storage.etcd.peerAddress), but populates spec.api.address only when onlyBindToAddress is true . If left empty, k0s determines it; according to the configuration documentation, the default is "the first non-loopback address found on the node", which in this VM was the NAT address of the default route NIC. The join token contains this address.

spec.api.onlyBindToAddress: true Providing address changes two things together. k0sctl populates privateAddress for each host with that host's

, and the API server binds only to that address, not all interfaces. The k0s configuration documentation also notes that this option may be necessary for "multi-homed control plane nodes". After fixing this, the 2-node installation finished in 1 minute 58 seconds with rc=0. config In the 3-controller setup of Part 3, this method is practically the only way. k0sctl's address is unique to the cluster, so there is no place to write different #1026for each controller. Three k0sctl issues addressing the same problem are open: #1035(Cannot give different addresses to each HA controller), #760(Workers do not connect to the API with private addresses),

(privateInterface does not pick the right address).

4. Stopping the Controller for 90 Seconds

4.1 Measurement Method webAfter running two replicas of nginx on the worker (Deployment k0scontroller ) and a NodePort Service (30080), we stopped the controller's

• service. From the transfer server, we recorded two things every ~5 seconds:: kubectl get nodes Whether the API succeeds.
• The status code of the: curl http://<워커>:30080/ service.

Around 20 seconds in, we killed the oldest nginx process on the worker, i.e., the master process (pkill -o -x nginx). This was to see who restarts the container when it dies in the absence of an API. After 90 seconds passed, we turned the controller back on and measured the seconds until the API recovered.

4.2 Results

The worker's k0sworker service remained active throughout.

Why the service continued. All paths through which requests travel are entirely within the worker. kube-proxy rules routing NodePort to pods, containerd running containers, and kubelet protecting pods all run on the worker, and the controller is only needed when deciding "what to run where." Therefore, already running pods respond even without a controller.

Restarting dead containers. In the first run, after killing the nginx master, the restart count increased from 0 to 1. Since kubelet has already received the pod definition and restart policy, it restarts the container even without an API. However, because it could not report that fact to the API, the number was only visible after the API came back. In the final run, we did the same thing, but the restart count remained 0. During that time, nginx responses were 200 all 18 times. We could not verify why it was not counted as a restart this time (cause unverified). Therefore, we record "kubelet restarts without an API" only as an observed fact in one of out of two attempts.

What stopped. While the controller is missing, all of the following stop working:

• New deployments, scaling, configuration changes. They all go through the API.
• Rescheduling pods elsewhere when a worker dies. In a 2-node configuration, there are no other workers to migrate to in the first place.
• Status reporting. Things that happened in the meantime, like restart counts, only become visible after the API recovers.

The danger of a 2-node configuration is not that "the service stops," but "that nothing can be changed while it is stopped, and if a worker dies during that time, there is no entity to revive it." Therefore, in a 2-node setup, rapid controller recovery—namely backup (Section 6)—takes precedence over high availability.

5. Adding and Removing Workers

Adding. We added worker02 to hosts in k0sctl.yaml and ran apply. Since k0sctl is declarative, it did not touch existing nodes and only ran the "Install workers" step, taking 1 minute 42 seconds and 2 minutes 31 seconds later, worker02 became Ready. This is the "1 controller + 2 workers" configuration compared in Part 3.

removal. Order matters.

1. kubectl drain worker01 --ignore-daemonsets --delete-emptydir-data to move pods to worker02. Since there needed to be a place to move them to, addition was done first.
2. Write reset: true to the worker01 host in k0sctl.yaml and apply it. k0sctl deleted k0s on that node during the "Reset workers" stage, and worker01 disappeared from the node list as well. It took 24 seconds · 39 seconds .

Both nginx replicas remained available on worker02. A failed to remove existing configuration /etc/k0s/k0s.yaml: file does not exist warning appears during reset, but it is harmless since workers do not have k0s.yaml in the first place.

6. Backup

k0sctl backup --config k0sctl.yaml

k0sctl runs k0s backup on the controller and brings the result back to the import server. k0s backup documentationstates that the backup contains etcd snapshots, certificates, and keys. Since it also includes the cluster CA private key, backup files must be stored like secrets. Even in a dark site (air-gapped environment), this file should not be placed in just any shared folder.

Restoration (k0sctl apply --restore-from) was not measured this time (unmeasured). The same documentation notes that restoration must be performed on the controller, and there is a separate procedure to match addresses when restoring to a node with a different IP. To claim that backups substitute for high availability in a 2-node configuration, measuring the restoration time is the next task.

7. Summary

The measurements in this post can be followed along using the same version and import list in TT Lab's hands-on course, "Air-Gapped k0s: 1, 2, and 3 Nodes with Ceph and PostgreSQL" (site path /course/k0s-airgap-multinode), under the second module, "2 Nodes". Controller stoppage, worker addition and removal, and backups are performed on actual VMs, and the grader checks the results at each step.

TT's Word. Even if the class president (controller) steps away briefly, the class (service) keeps going. However, during that time, you cannot swap seats or bring in a new friend. And while electing two class presidents might sound more secure, if the rule requires both to agree, when even one falls ill, nothing can be decided!