Where the truth of k0s configuration lives

One-line summary

k0s cluster configuration lives in a file on each controller under static configuration, and in the ClusterConfig object inside the API under dynamic configuration. If you edit the file without knowing which mode you are in, you get "I edited it and nothing happened."

Why this was needed

k0s is introduced as setting up a cluster with one binary and one k0s.yaml. So when an operator wants to change settings, they naturally open /etc/k0s/k0s.yaml. Two kinds of incidents happen here.

First, under the default static configuration, k0s does not watch the file. The configuration documentation says you can change the file while running, but you must restart k0s to have it take effect. If there are several controllers, you have to make the files on all the controllers identical and restart them all. If you fix only one and forget, the settings change only when that node is next restarted, and the controllers end up holding different settings from one another.

Second, if you turn on dynamic configuration (--enable-dynamic-config) to get rid of this inconvenience, a trap in the opposite direction appears. According to the dynamic configuration documentation, when the cluster is first created, the first controller reads the file once as a bootstrap value and stores it in the API, and after that the object in the API is the source of truth for all controllers. From then on, cluster-level settings in the file are ignored even if you edit it and restart.

How it works

The configuration file may be partial

k0s config create prints a configuration with all defaults filled in. But k0s accepts a partial configuration and fills in missing values with defaults, so it reads better to keep only what you want to change in the actual file. The reason to look at the defaults is to know "what gets decided for the values I did not write." On this lab's VM, the default output was Pod CIDR 10.244.0.0/16, Service CIDR 10.96.0.0/12, datastore etcd, and telemetry on. This VM runs on top of Kubernetes, so the default ranges overlap with the host cluster, and they were changed to 172.20/172.21 in the file.

k0s sysinfo is a pre-check that the same binary runs before installation. It shows memory, the /var/lib/k0s filesystem and its free space, the cgroup version and controllers, and kernel settings as pass or warning per item, and with -o json it can also be read by machines.

Cluster settings and node settings are different

Even under dynamic configuration, the file is not entirely ignored. The documentation classifies spec.api (that node's API server), spec.storage (that node's etcd or SQLite), and spec.network.controlPlaneLoadBalancing as per-node settings that continue to be read from the file. This is a natural design, since the API server and storage have to be up before any object can be read. Also, items such as network.podCIDR, serviceCIDR, and provider cannot be changed after the cluster is created, so the documentation says you must write them in the file at manual installation.

The other cluster-level settings (worker profiles, kube-router and kube-proxy settings, Helm extensions, and so on) are decided by the clusterconfig/k0s object in the kube-system namespace. The controller watches this object in the Operator style, recreates the related resources when it changes, and leaves the result as events. k0s config status shows those events (SuccessfulReconcile, FailedReconciling).

정적 구성   파일 수정 ──(재시작해야)──▶ 반영
동적 구성   파일 수정 ──(재시작해도)──▶ 클러스터 수준 설정은 무시
            ClusterConfig 수정 ──(즉시)──▶ 반영, 이벤트 기록

Worker profiles appear as ConfigMaps

The worker profile (spec.workerProfiles) in the worker node configuration documentation is a bundle of kubelet configuration overrides. k0s creates a ConfigMap for each profile, and the worker reads at startup the ConfigMap it picked with --profile. In measurement, the name carried the Kubernetes minor version, as in worker-config-<프로필>-1.36 (the placeholder is the profile name). So you can visibly confirm whether the setting took effect from the existence of this ConfigMap and the values inside kubeletConfiguration.

Helm extensions create Chart objects

The Helm charts documentation describes two methods. One is creating the Chart object directly (recommended), and the other is declaring a repository and chart in spec.extensions.helm so that k0s converts it into a Chart object. The repository must be a proper Helm repository that has an index.yaml, and when a Chart is deleted, k0s removes the release. In measurement, the name of a Chart created by declaration was k0s-addon-chart-<차트 이름> (the placeholder is the chart name), and it had a finalizer attached for removal.

What it looks like in the field

Here are the scenes seen in measurement (k0s v1.36.4+k0s.0).

• Under static configuration, even after putting a worker profile in the file and waiting 45 seconds, no ConfigMap appeared, and it appeared 17 seconds after k0s stop and k0s start.
• Running k0s install controller --enable-dynamic-config again on an already installed controller was rejected with "Init already exists", and --force was needed.
• The ClusterConfig created after switching to dynamic configuration contained the file's worker profile, but spec.network.serviceCIDR appeared as 10.96.0.0/12. The actual API server's --service-cluster-ip-range and the kube-dns address were 172.21, as in the file. For items that cannot be changed, do not trust the value shown in the object; check it on the actual components.
• Under dynamic configuration, I added a profile to the file and even restarted, but the object's generation stayed the same and there was no ConfigMap. When I patched the object, it appeared within 1 second.
• When I sent a merge patch of the object's workerProfiles containing only the new list, the list was replaced wholesale, and the ConfigMap of the profile that was left out was deleted right away. When adding to the list, use the add operation of a JSON patch.
• When I kubectl deleted only the Chart object while leaving the declaration as it was, the release was deleted right away, but it did not come back even after waiting 99 seconds or editing ClusterConfig once more, and it was recreated only after k0s was restarted. This means that the declaration and reality can silently diverge.

What really matters in practice

• Check the mode first. The fastest way is to see whether the execution line of the service unit has --enable-dynamic-config. All controllers must be in the same mode, and the documentation warns that mixing them causes conflicts.
• If it is static configuration, making the files on all controllers identical and restarting is one job.
• If it is dynamic configuration, make cluster-level changes through the object and check the result with k0s config status. In the file, only per-node settings and the unchangeable network values are meaningful.
• Confirm that a change took effect by its products. Rather than that the command succeeded, look at whether the ConfigMap, Chart, and release were actually created or disappeared.
• When removing a Helm extension, delete the declaration. If you delete only the Chart, it comes back at the next restart. Conversely, if you delete the declaration, the release is removed, so for a chart with data you must think about that consequence first. The namespace remained.

What you will do in the next lab

You start from a single k0s that came up with static configuration. After comparing the defaults with the file and reading the pre-check, you put a worker profile in the file and compare before and after a restart. You switch to dynamic configuration and see the ClusterConfig created, then edit the file in the same way and confirm that this time it is ignored even after a restart. Finally, you declare a worker profile and the podinfo chart in the object, and record what remains when you delete the declaration.