Closed-network k0s multi-node actual measurement 3: Rook-Ceph and CloudNativePG on 3 HA nodes

Status: Measured (2026-10-31). These values were measured by setting up k0s v1.37.1+k0s.0 inside TT Lab practice VMs (three 8 vCPU / 8GiB nodes and one import server). We built the cluster from scratch four times modifying the same answer sheet; the first run was marked unmeasured because the answer sheet was cut off in the middle, mixing up the steps. The dispersion of the numbers reflects the actual variance in this environment. The VM disks were on an HDD. Node addresses have been changed to an example block (192.0.2.0/24). Writing convention: Measured data is the default; unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. Starting with the Conclusion
• 2. How to Split Three Nodes
  • 2.1 Three Ways to Bind the API Address
  • 2.2 k0sctl Configuration
• 3. Mirroring Images to the In-house Registry
• 4. Installation: k0sctl Does Not Wait for Controller-cum-Worker Ready State
• 5. Stopping One or Two Controllers
  • 5.1 Measurement Method
  • 5.2 Results
• 6. Rook-Ceph
  • 6.1 What Changes Because of k0s: Kubelet Path
  • 6.2 Actual Measurement
• 7. CloudNativePG
  • 7.1 Installation Order and Backup Method
  • 7.2 Pitfall: Creating ObjectStore Before the Bucket Name Exists
  • 7.3 Deleting the Primary
  • 7.4 Backup, Restoration, and How to Wait for WAL
• 8. How to Run Time-Consuming Steps
• 9. Resources
• 10. Limitations and Unmeasured Items
• 11. Summary

1. Starting with the Conclusion

• All three nodes were configured as controllers and workers. The 3-member etcd tolerates 1 node failure. The API address was bound without an external load balancer using CPLB (keepalived virtual IP) and NLLB (envoy per node) included in k0s.
• When stopping the node with the virtual IP (VIP), the API recovered within 0 to 3 seconds in all four runs. The two runs measuring writes showed 1 and 3 seconds. When stopping two nodes, all writes failed for 30 seconds, and they recovered in 1 to 2 seconds once turned back on.
• We mirrored 18 images (1.8 GB) to the in-house registry and ran them with containerd's hosts.toml . Not a single image name in the charts and manifests was modified.
• Rook-Ceph took 437 to 605 seconds (7 to 10 minutes) from chart installation to HEALTH_OK.Only a single kubelet path in k0s needed to be adjusted.
• For CloudNativePG, a new primary accepted writes within 35.4 to 62.6 seconds after deleting the primary pod, and both row-count runs remained identical. However, the deleted old primary remained in Terminating state for over 5 minutes in three out of four runs. After backing up to Ceph's object storage (4 to 53 seconds) and restoring to a new cluster (1 minute 54 seconds to 4 minutes 6 seconds), even rows inserted after the backup were recovered via WAL.
• Four Pitfalls: k0sctl does not wait for the Ready state of controller-cum-worker nodes. Starting from PostgreSQL 17, the boundary rules of pg_walfile_name changed, causing scripts waiting for WAL archiving to hang indefinitely. Creating backup settings before the bucket name exists breaks WAL archiving. systemctl is-active answers "no" for running oneshot units.

2. How to Split Three Nodes

There are two ways to divide three nodes.

Approach (b) does not achieve high availability for storage, which is the goal of this article. Ceph's default block pool places 3 replicas across different hosts (failureDomain: host), and mons are placed on distinct nodes to secure a majority. With only two storage nodes, there is nowhere to place 3 replicas, leaving data as undersized+degraded . Lowering it to 2 replicas causes writes to halt during a single node failure (min_size 2) or forces survival with a single data replica (min_size 1). For the same reason, Ceph and PostgreSQL sets were not deployed on 2 or fewer nodes. Ceph's placement principles are detailed in the Rook-Ceph Guide series.

2.1 Three Ways to Bind the API Address

When there are three controllers, you must decide which address kubectl and nodes will use to call the API.

1. CPLB(control plane load balancing). k0s runs keepalived to assign a virtual IP to a node via VRRP, and moves it to another node if that node dies. CPLB documentationexplains this is for "access originating from outside the cluster". Instead of increasing things to import, nodes must be on the same L2 network (documentation: virtual router ID must be unique within the broadcast domain). Controller-cum-worker setups only support k0s's userspace reverse proxy method.
2. NLLB(node-local load balancing). Runs envoy on the loopback (127.0.0.1:7443) of each worker so that node components like kubelet evenly connect to the set of running apiservers. NLLB documentationnotes that NLLB is for fault tolerance inside the cluster and does not assist with external access.
3. External Load Balancer(HAProxy, etc.). The example in the HA documentationcombines 6443 (API), 8132 (konnectivity), and 9443 (join API). That load balancer must be redundantly configured in turn, increasing the packages to import. We did not measure this time (unmeasured).

The CPLB documentation recommends using CPLB and NLLB together if there is no external load balancer. This configuration adopts that approach.

2.2 k0sctl Configuration

Only one host is listed. The other two differ only in address.

  - role: controller+worker
    noTaints: true                       # 컨트롤러에도 워크로드를 놓는다
    ssh: {address: 192.0.2.3, user: root, keyPath: /root/.ssh/id_ed25519}
    hostname: ctrl01
    privateInterface: eth1
    privateAddress: 192.0.2.3
    uploadBinary: true
    k0sBinaryPath: /opt/airgap/bin/k0s-v1.37.1+k0s.0-amd64
    installFlags: [--kubelet-extra-args=--node-ip=192.0.2.3]
    files:
    - {name: k0s-airgap-bundle, src: /opt/airgap/images/k0s-airgap-bundle-v1.37.1+k0s.0-linux-amd64.tar, dstDir: /var/lib/k0s/images/, perm: "0644"}
    - {name: apps-bundle, src: /opt/airgap/images/apps-bundle.tar, dstDir: /var/lib/k0s/images/, perm: "0644"}
    - name: cri-registry
      dst: /etc/k0s/containerd.d/cri-registry.toml
      data: |
        version = 3
        [plugins."io.containerd.cri.v1.images".registry]
          config_path = "/etc/k0s/containerd.d/certs.d"
    - name: mirror-quay.io               # docker.io·ghcr.io·registry.k8s.io·gcr.io 도 같은 모양
      dst: /etc/k0s/containerd.d/certs.d/quay.io/hosts.toml
      data: |
        server = "https://quay.io"
        [host."http://192.0.2.2:5000/v2/quay.io"]
          capabilities = ["pull", "resolve"]
          override_path = true
  k0s:
    version: v1.37.1+k0s.0
    config:
      spec:
        api: {onlyBindToAddress: true, sans: [192.0.2.100]}
        network:
          provider: kuberouter
          podCIDR: 172.20.0.0/16
          serviceCIDR: 172.21.0.0/16
          controlPlaneLoadBalancing:
            enabled: true
            type: Keepalived
            keepalived:
              vrrpInstances:
              - {virtualIPs: ["192.0.2.100/24"], authPass: <클러스터마다 다른 값>, interface: eth1}
          nodeLocalLoadBalancing: {enabled: true, type: EnvoyProxy}
        images: {default_pull_policy: Never}
        telemetry: {enabled: false}

Here too, the fact that there are two NICs appears in three places.

• onlyBindToAddress: true was included for the reason in Section 3.1 of Part 2. When there are three controllers, there is no place to write address for each host, making this the only way.
• You must also write VRRP's interface: eth1 . According to the CPLB documentation, if left blank, k0s picks the NIC with the default route . In this VM, that is the NAT NIC, and raising the VIP there prevents other nodes from reaching it. Put the VIP in
• sans . If omitted, connecting via VIP will cause TLS to fail because that address is absent from the API server certificate.

authPass As the documentation states, k0sctl kubeconfig is a value to prevent accidental mixing between clusters, not for encryption. Set it to avoid overlapping with other clusters on the same network. After installation, we changed the server address in the kubeconfig provided by

to the VIP. Pointing to a single controller causes kubectl to die along with it when that node dies.

3. Mirroring Images to the In-house Registry

The import server placed them with skopeo copy by 192.0.2.2:5000/quay.io/ceph/ceph:v20.2.4 , prepending the original registry name to the path . containerd on the node side points to this path using a single hosts.toml per registry. According to containerd's hosts.md, override_path = true means "the root of the API is specified in the URL path", so containerd requests it like /v2/ without re-appending http://192.0.2.2:5000/v2/quay.io/ceph/ceph/manifests/... . Therefore, we did not modify a single image name in the charts and manifests. Changing image names one by one in values files in a closed network invariably leads to missing one, which reveals itself minutes later as ImagePullBackOff.

The drop-in file layout followed the k0s runtime documentation. The documentation notes that this version of k0s must use the version = 3 and io.containerd.cri.v1.images plugin paths, and using version = 2 or old plugin paths will cause pre-checks to halt.

The list of images to mirror was based on deploy/examples/images.txtin the Rook repository, excluding unused COSI, csi-addons sidecars, and snapshot-metadata. Any missing items immediately manifest as ImagePullBackOff, so there is no need to fear trimming the list.

I will also note the limitations. The test registry was HTTP within the internal network. In the field, TLS is used and hosts.toml is given the internal CA (ca = ...) (unmeasured). The method of pulling even the k0s system images from the registry (spec.images.repository) was also not measured (unmeasured). The reason for keeping system images as a bundle is to ensure that CNI, kube-proxy, and coredns come up when nodes restart even if the import server's registry dies.

4. Installation: k0sctl does not wait for Controller-cum-Worker Ready

Just because apply finished did not mean the cluster was fully up. In run 2, apply finished successfully at 93 seconds, but two nodes left kubelet's 「Successfully registered node」 104 and 125 seconds later, respectively.

Looking at the k0sctl source code, the reason was obvious. install_workers.go's 「waiting for node to become ready」 runs only on Workers() , and this function selects only hosts whose role is strictly worker . controller+worker finishes during the controller installation phase, so it does not wait for the kubelet to become Ready. Therefore, automation following apply must independently verify whether the node set is Ready. If you proceed directly to the next step without doing so, Ceph will be installed with one node missing.

5. Stopping one or two controllers

5.1 Measurement method

Measured two things every second against the VIP from the import server.

• API: kubectl get --raw /readyz(4-second timeout).
• Write: Create a ConfigMap (6-second timeout). A new name is generated every time. Creating the same name again fails with AlreadyExists , which cannot be distinguished from 「failed due to loss of quorum」. The first attempt actually left such a false failure.

First, we found the node holding the VIP (the node with the VIP attached to its internal network NIC), stopped k0scontroller , and measured the time from the stop until API and writes recovered. Next, we also stopped the second controller, observed whether writes succeeded even once for 30 seconds, and measured the time until writes recovered after turning both back on.

5.2 Results

When one node was stopped, the live node's k0s etcd member-list still contained the member set. Stopping one does not remove it from the members, so the quorum is still 2, and the two surviving nodes met that quorum to accept writes. When two are stopped, the remaining one cannot meet the quorum, so writes stop. This matches the etcd table in Part 2.

One thing to watch out for: even if you stop kubelet, the Node object's Ready remains at its old value for a while. This is because there is a timeout (--node-monitor-grace-period, documentationdefault is 50 seconds) before the controller manager changes the node to 「Not Ready」. Saying 「the node was Ready even during the outage」 is not proof of high availability. Therefore, the table above was measured using actual requests rather than Node status.

6. Rook-Ceph

6.1 What changes because of k0s: kubelet path

k0s's kubelet uses /var/lib/kubelet as its working directory instead of /var/lib/k0s/kubelet (k0s storage documentation). CSI node plugins mount volumes by attaching the kubelet directory via hostPath, so if this path is incorrect, PVCs will bind but will not attach to pods.

Rook v1.21 separated the CSI driver into ceph-csi-operator and a separate chart (ceph-csi-drivers). The kubelet path was not found in the rook-ceph chart values, so it was placed in the driver chart values.

# ceph-csi-drivers 1.1.0 의 values(Rook 이 권하는 값에 이 줄만 더함)
operatorConfig:
  driverSpecDefaults:
    nodePlugin:
      kubeletDirPath: /var/lib/k0s/kubelet

Another way is to symlink the k0sctl host's kubeletRootDir to the kubelet side's /var/lib/kubelet , but this was not measured (unmeasured). Because the driver DaemonSet is created by the operator inspecting the Driver object, it did not yet exist immediately after chart installation (23 seconds). Automation checking for the DaemonSet immediately after installation must wait until it appears.

6.2 Measurements

The chart set (rook-ceph, ceph-csi-drivers, rook-ceph-cluster v1.21.1, Ceph v20.2.4) was installed from tgz imported via helm. One empty 17GiB disk per node was used as an OSD, and since nodes also bear workloads, the OSD's osd_memory_target was lowered to 1GiB.

HEALTH_OK is the value after investigating and silencing the cephx key format warnings raised by Ceph v20 (AUTH_INSECURE_*). After running all steps to the end, it became HEALTH_WARN due to BLUESTORE_SLOW_OP_ALERT(slow disk warning). This is the result of placing a PostgreSQL workload on a VM running on HDDs. On field hardware, this is a value that must be re-evaluated from the disks up.

7. CloudNativePG

7.1 Installation order and backup method

Imported manifests were applied in the order of cert-manager → CNPG operator (1.30.1) → barman-cloud plugin (v0.15.1). The CNPG manifest was applied using installation documentationas kubectl apply --server-side .

The reason for performing backups via the plugin is due to version lifecycles. CNPG 1.30's original backup documentationstates that the built-in barmanObjectStore method has been deprecated since 1.26, 1.31 is the last version to support it, and it will be removed in 1.32. There is no reason to start a newly built cluster with a method that will soon disappear. The plugin requires cert-manager (plugin installation documentation). Because the backup tool is included in the plugin sidecar, a small PostgreSQL image ( 18.6-minimal-trixie ) was sufficient.

The backup destination is the Ceph object storage (RGW) in the same cluster. When you create a bucket with an ObjectBucketClaim, Rook provides a ConfigMap (bucket name and address) and a Secret (keys) with the same name. However, RGW in the same cluster is not a separate failure domain. If you lose the cluster entirely, you lose the backups as well, so in production, you must send them to an external storage once more.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: {name: pg, namespace: db}
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:18.6-minimal-trixie
  storage: {size: 2Gi, storageClass: ceph-block}
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters: {barmanObjectName: ceph-rgw}

In run 2, it took about 13 minutes and 40 secondsfor all instance sets to become Ready (initdb 70 seconds, the rest for replica image pulling and joining). While this value is heavily influenced by the condition of being a VM on HDDs and should not be generalized as-is, it is clear that when setting upper bounds for automation waits, you should aim for 「tens of minutes」 rather than 「a few minutes」.

7.2 Pitfall: Creating ObjectStore before the bucket name is generated

In the first attempt, the ObjectBucketClaim did not become Bound within 2 minutes, but the script proceeded directly to the next step once the wait ended. Consequently, ObjectStore was created with an empty bucket name, causing destinationPath to become s3:/// . WAL archiving continuously failed with Invalid s3 URL address .

Deleting the primary pod in this state made things much worse. The new primary came up in 39.8 seconds, but holding onto WALs that the old primary failed to preserve, it remained Terminating for over 18 minutes . The CNPG pod's terminationGracePeriodSeconds is 1800 seconds, so Kubernetes waited that long. Now, we made it wait until the ConfigMap's BUCKET_NAME is generated and the OBC is Bound, and if it ultimately remains empty, it stops.

7.3 When the primary is deleted

When all instance sets were Ready, we deleted the primary pod and queried select pg_is_in_recovery() on the new primary every 0.5 seconds to measure until f(writable) appeared. It is crucial to start only when all three are Ready. There was a run where we deleted the primary when no replicas had come up yet behind a broken answer key, resulting in the loss of the cluster.

There were no issues with the promotion of the new primary and writing. The problem is the old primary. In two out of the three times archiving was normal, it was also Terminating for over 5 minutes, and that pod's logs ended like this.

"msg":"unable to run controller-runtime manager",
"error":"failed waiting for all runnables to end within grace period of 30s: context deadline exceeded"

Although PostgreSQL already shut down normally with server stopped , the process did not end even after leaving an error that CNPG's instance manager failed to finish all internal tasks within 30 seconds. The child postgres process remained as a zombie (<defunct>), and the pod headed toward consuming the entire grace period of 1800 seconds. The cause could not be determined. Although it seems related to the issue where shutting down the primary pod takes about 3 minutes when the barman-cloud sidecar is enabled ( plugin-barman-cloud#1039(open)), this observation is vastly longer (estimated).

Why should we care? While the old primary is Terminating, the cluster survives with 2 Ready instances. If another node dies during that time, it enters a state with only one replica. Even if you try to drain the node where that pod sits, it does not finish. After a failover, you must verify even whether the old primary actually returned as a replica. Settings to mitigate this phenomenon (such as adjusting the grace period) were not tested (unmeasured).

7.4 Backup, recovery, and how to wait for WAL

apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata: {name: pg-b1, namespace: db}
spec:
  cluster: {name: pg}
  method: plugin
  pluginConfiguration: {name: barman-cloud.cloudnative-pg.io}

After the backup finished, I inserted one more row (1002), rolled over the WAL file, confirmed that the WAL was uploaded to the bucket, and bootstrap.recovery created a new cluster pg-restore using it. If we recover up to row 1002, which was inserted after the backup point (row 1001), it means the WAL recovery was successful.

In the first attempt, I passed the WAL and waited for only 15 seconds, and the restoration source stopped at the backup point (1001). This is because restoration only reads WALs uploaded to the bucket up to that point. So, I added "Wait until the passed WAL is archived", but this time the wait did not end for 300 seconds.

The code I wrote initially used pg_walfile_name(pg_switch_wal()) to get the name of the file to wait for. According to the PostgreSQL documentation, pg_switch_wal() returns "end position of the WAL file just finished + 1", which is the boundary where the next file starts. Depending on the version, what name comes out when you pass the boundary position to pg_walfile_name varies.

• Up to the PostgreSQL 16 documentation, it stated: "If the position is exactly on a boundary, return the name of the previous file; when dealing with archiving, this is usually the desired behavior." That is why this idiom in older posts was correct.
• The PostgreSQL 17 release notesstate in the section "Migrating to 17" that these two functions were changed to return the current filerather than the previous file at a boundary. The PostgreSQL version used in this test was 18.6.

Therefore, in version 17 and later, pg_walfile_name(pg_switch_wal()) is the name of the next file that has not even started writing yet, and that file is not archived until the next switch occurs. This is why the wait did not end. The fix was to change the order.

-- 1002 를 넣은 뒤, 넘기기 「전에」 지금 파일의 이름을 적는다
select pg_walfile_name(pg_current_wal_lsn());   -- 예: ...0B
select pg_switch_wal();
-- 그다음 last_archived_wal 이 적어 둔 이름 이상이 될 때까지 기다린다
select last_archived_wal from pg_stat_archiver;

After making this fix, all three attempts successfully recovered up to row 1002. It is also worth noting that the instance restarts once after recovery finishes ("Primary instance is being restarted without a switchover"). If you finish just by seeing Ready once, you immediately encounter a moment where Ready is empty. I finished after seeing "healthy" twice at 10-second intervals.

8. How to run long-running steps

apply, Ceph, CNPG, chaos testing, backups, and restorations take anywhere from a few minutes to over ten minutes. To avoid sitting and waiting tied to the terminal, I ran them in the background using systemd-run --property Type=oneshot and queried separately whether they finished. Here I encountered one trap and two small surprises.

• systemctl is-active returns "no" for a running oneshot unit. According to the systemd.service documentation, Type=oneshot units remain in the "activating" state until the program finishes. Since is-active is only true when "active", it failed to detect that it was "still running", causing failures where waiting was required. I checked whether systemctl show -p ActiveState --value <유닛> is activating .
• helm v4.3.0 에서는 helm list -a of -a lacked unknown shorthand flag: 'a' and ended with
• . helm status -o json of the same version took 19 seconds per release. For status checking, helm list -o json is better.

9. Resources

This is a set of three nodes (8 vCPU and 8GiB) after everything is up.

/var/lib/k0s and pod logs were moved to the working disk, but journald and k0s logs accumulated on the root. If the root disk is small, you must set an upper limit for journald first. We have not run it for a long time with a limit set (unmeasured).

10. Limitations and Unmeasured Items

• Environment: Since it is k0s inside a KubeVirt VM, absolute time may be slower than on a physical server, and the disk was an HDD. In particular, the performance of Ceph and CNPG heavily depends on the disk.
• Iteration count: Four times for the three scenarios, and four times for failover. Please read this as case studies rather than a statistical distribution.
• Unmeasured items: Configurations wrapping the API with an external HAProxy, spec.images.repository pulling even system images from the internal registry, k0sctl apply --restore-from restoration, k0s upgrades, behavior after node reboots, TLS registry, kubeletRootDir changing the kubelet path, settings to reduce the Terminating state duration of the old primary, and long-term operation with a journald limit.

11. Conclusion

Measurements in this post can be reproduced in the TT Lab hands-on course 「Air-gapped k0s: 1, 2, and 3-node clusters with Ceph and PostgreSQL」 (site path /course/k0s-airgap-multinode), module 3 「3-Node HA」, using the same version and manifest list. VIP migration, Rook-Ceph, and CNPG failover, backup, and restore are performed on real VMs, with a grader checking the results at each step.

TT's word. If you elect three class presidents, even if one gets sick, the other two can discuss and decide. But if two get sick, a single person can't decide anything. Also, the moving president (the old primary) might stand at the door for 30 minutes without finishing packing, so don't just finish when the new president takes office—make sure to check whether the old president has returned to their seat!