What Kubespray fixes for you, and what it doesn't know about

One-line summary

kubespray sets swap, modules, and sysctl by itself, but it does not know about ports that someone else holds or about configuration files that later revert the values. A pre-install check is knowing where that boundary lies.

Why this was needed

A kubespray installation takes several minutes at a time. If it fails after those minutes, the cause is usually something that was already there before the installation. And for such failures, the error message does not point to the cause. If another process is holding 10250, the first kubeadm init is blocked by [ERROR Port-10250] in preflight. But kubespray assumes the first attempt may have already started the kubelet, and on the retry it puts Port-10250 in the ignore list and runs again (roles/kubernetes/control-plane/tasks/kubeadm-setup.yml). Then the line that states the cause remains only in the first-attempt output higher up the log, and the conspicuous last error says something entirely different. After the first reboot, Pods lose communication with each other, yet the kubespray log is all green. Rather than tracing such things backward after the installation, it is much cheaper to look before the installation at what the host can accept.

How it works

What Kubernetes requires of a node is scattered across the official documentation. Gathered together, it falls into four branches.

스왑        kubelet 은 기본값(failSwapOn: true)으로 스왑이 켜져 있으면 시작을 거부한다
커널 모듈    overlay(containerd 의 기본 스냅샷터) · br_netfilter(브리지 트래픽이 iptables 를 거치게)
sysctl      net.ipv4.ip_forward = 1 · net.bridge.bridge-nf-call-iptables = 1
포트        6443(API) · 2379-2380(etcd) · 10250(kubelet) · 10257(controller-manager) · 10259(scheduler)

kubespray handles the first three of these with roles. roles/kubernetes/preinstall/tasks/0010-swapoff.yml removes the swap line from fstab, masks swap.target so it is not turned on again, and calls swapoff -a. This task runs only when kubelet_fail_swap_on is true, and the default is true. If you want to use swap, there is a path too: set this value to false and have the kubelet come up with swapBehavior: LimitedSwap. br_netfilter is loaded by the node role and recorded in /etc/modules-load.d/kubespray-br_netfilter.conf. Values such as ip_forward are written to sysctl_file_path (default /etc/sysctl.d/99-sysctl.conf). On Ubuntu this file is a symlink pointing to ../sysctl.conf, so kubespray follows the link and writes to /etc/sysctl.conf (0080-system-configurations.yml), but the place read at boot is still the order of the name 99-sysctl.conf.

Ports are different. kubespray has no task that stops someone else's process, and it should not have one. What is holding a port is that server's own business. So a person looks at ports first. ss -ltnp shows the PID, and you must find the systemd unit that started that PID, stop it, and disable it. If you only kill the process, a Restart=always unit revives it in 2 seconds.

sysctl has an ordering trap. At boot systemd-sysctl, and when a person runs sysctl --system procps, reads the *.conf files in /etc/sysctl.d, /run/sysctl.d, and /usr/lib/sysctl.d in file name order, and for the same key the value read later wins. If the names are the same, the /etc side wins. So even if you write ip_forward = 1 in k8s.conf, if zz-hardening.conf writes 0, it is 1 now but becomes 0 after a reboot. The 99-sysctl.conf that kubespray writes is under the same rule too — digits sort before letters, so every file whose name starts with a letter is read after it.

Finally, there is kubespray's own pre-check. 0040-verify-settings.yml judges by the facts Ansible gathered. It stops if a control plane node's memory is smaller than minimal_master_memory_mb (1500MB), and it stops on an unsupported distribution. And if you do not give the ip variable, it attaches the API server and etcd to the facts' default route address (default_ipv4). This is where the incident comes from of a control plane attaching to the wrong network on a server with several interfaces.

What it looks like in the field

This module's lab VM has three traces left deliberately. All of them are really common shapes. First, back in a time of low memory, someone created a swap file and wrote it in fstab. Second, an old monitoring agent is receiving HTTP on 10250 — the same port as the kubelet, so after installation the kubelet restarts repeatedly with bind: address already in use. Third, a security audit left a sysctl file that hard-set ip_forward to 0 as "no routing", and its name starts with zz-, so it is read later than any Kubernetes setting.

Of these three, the only one kubespray resolves by itself is swap. It does not touch the port, and sysctl is turned on at the moment of installation, but at reboot the zz- file puts it back to 0. So the checklist in the field asks not "what did you set?" but "is it still so after a reboot?"

What you will do in the next lab

You record the state before fixing, then turn swap off both now and at boot. After loading the modules and applying sysctl, you confirm that the reboot value reverts because of the zz- file, and fix it. You find and stop the unit that holds 10250, and gather the facts kubespray looks at and compare them with the minimum memory. Finally, you read the role code and sort out who fixes each of the four problems.