Kubernetes Distributions — Build Them Yourself
Defaults Decide What You Can Do
One-line summary
Lightweight distributions differ not in features but in defaults, and those defaults decide what you can do in production.
Why you should know the defaults before choosing
k3s and k0s are both lightweight Kubernetes distributions shipped as a single binary. They look alike, but their defaults differ, and those defaults decide what you can do in production.
Storage
The default storage for k3s is sqlite. A layer called kine imitates the etcd API and keeps sqlite behind it. It is light and enough for a single machine, but you can use neither etcdctl nor member management.
If you set up k0s with --enable-worker, you get real etcd. You can actually work with snapshots, member management, and recovery. This is where the CKA etcd questions and real-world control plane operations come in.
Watch out for k0s --single mode. It looks convenient, but it uses kine (sqlite), so k0s etcd member-list rejects it like this.
Error: wrong storage type: kine
Default components
k3s ships with the traefik Ingress and the local-path provisioner by default. That is convenient in a development environment. k0s includes nothing and makes you pick what you need. In production this is the better approach — removing something that came by default later is more trouble.
CIDR ranges
The default k0s ranges (Pod 10.244.0.0/16 · Service 10.96.0.0/12) are the same as standard Kubernetes. So they overlap when you nest k0s inside another cluster. The k3s ranges 10.42/10.43 are nonstandard, so they happen to avoid the collision.
Neither is right or wrong; what matters is knowing the ranges of your own environment and choosing accordingly. And if you rely on a state that only happens to avoid the collision, the trap comes back the moment you switch distributions.
What to base the choice on
Choosing a distribution is decided not by taste but by what you will run on top of what.
| Situation | Best fit | Reason |
|---|---|---|
| Edge, IoT, single node | k3s | One binary, around 512MB of memory |
| Local development | kind, minikube | Fast to create and throw away |
| You want to learn the standard as is | kubeadm | Certification exams and documentation assume it |
| Many clusters, automatically | k0s, Cluster API | Stamped out declaratively |
| You want to reduce management burden | Managed (EKS, GKE) | You don't look after the control plane |
If you are preparing for an exam, use kubeadm. k3s uses SQLite instead of etcd and its control plane is a single process, so what the exam asks about — "etcd backup", "editing a static Pod", "certificate renewal" — does not carry over as is. What is convenient and what teaches you are different things.
The differences the defaults create
Even with the same Kubernetes, what ships by default differs, and that separates practice from production.
| kubeadm | k3s | k0s | |
|---|---|---|---|
| Data store | etcd | SQLite (default), etcd possible | etcd (default), kine possible |
| CNI | None — install it yourself | Flannel | Kube-router |
| Ingress | None | Traefik | None |
| LoadBalancer | None | ServiceLB (klipper) | None |
| Storage class | None | local-path | None |
When you first set up a cluster with kubeadm, the nodes stay stuck in NotReady. It is not a fault; there is simply no CNI. If you don't know this, you lose an hour on your first install.
k3s, on the other hand, comes with everything and is convenient, but you need to know how to remove those defaults.
# Traefik 과 ServiceLB 를 빼고 직접 고른 것을 쓴다
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik --disable=servicelb" sh -
What gets in the way when you move
Switching distributions happens more often than you might think. Three things usually get in the way.
- Storage class names — If names differ, such as
local-pathandstandard, the PVC stays stuck in Pending. Don't hard-code the name in the manifest; use the default class or inject it as a value. - LoadBalancer implementation — The k3s ServiceLB uses the node IP as is. If you move to a place that has MetalLB, you need to configure the IP range.
- Ingress controller differences — Traefik and NGINX use different annotations. If you use the Gateway API, this migration cost drops significantly.
What really matters in practice
If you have to work with etcd, check the storage first. On top of kine (sqlite), you can use neither etcdctl nor member management nor snapshot restore. Even with k0s, --single mode is kine, so the command rejects it with wrong storage type: kine.
When nesting, first count whether the ranges overlap. The k0s default ranges are the same as standard Kubernetes, so they collide when you set it up inside another cluster. The reason k3s worked was not design but the nonstandard ranges happening to avoid the collision, so the trap comes back the moment you switch distributions.
Components that came by default are more trouble to remove later. In a development environment it is handy to have traefik and local-path included, but in production it is better to pick only what you need yourself.
In the next lab, you will set up k0s and run through backup and restore once.