Building clusters with Kubespray and Terraform

One inventory file shapes the whole cluster

Goal

Using the sample inventory of kubespray v2.32.0, you design the inventory of a single-node cluster, pin the version in group_vars, and confirm for yourself which layer a variable wins at and what breaks when you get a group name wrong. You do not set up the cluster yet.

Why it matters

When you set up a cluster with kubespray, what a person writes directly is effectively only the inventory and group_vars. You leave the playbooks alone and change only these two to decide the shape of the cluster (who is the control plane and who is etcd) and its content (which version, which CNI, which runtime). So the cause when an installation goes wrong is usually here — get one group name wrong and the installation stops in a minute with an unrelated error, and if you do not write the version, the same inventory installs a different Kubernetes every time you upgrade kubespray. The goal of this module is the habit of checking the resolved values with inventory tools before running a long installation.

Steps

Copy /opt/ks/kubespray/inventory/sample wholesale to /root/ks/inventory/lab. group_vars/all/all.yml and group_vars/k8s_cluster/k8s-cluster.yml must be under it.
In /root/ks/inventory/lab/inventory.ini, put the node node1 in the kube_control_plane and kube_node groups, and write etcd so that it takes kube_control_plane as a child group ([etcd:children]). Also write k8s_cluster so that it takes kube_control_plane and kube_node as children ([k8s_cluster:children]). Put the connection method not on the inventory line but in /root/ks/inventory/lab/host_vars/node1.yml as ansible_connection: local. There must be no ansible_connection in inventory.ini.
In /opt/ks/kubespray, run ansible-inventory --list twice, count the parsed hosts, and write to /root/ks/parse.json dir_hosts (when you gave the directory as -i /root/ks/inventory/lab), file_hosts (-i /root/ks/inventory/lab/inventory.ini), and ignored_ext (among the extensions that kubespray's ansible.cfg skips in an inventory directory, the one that hides inventory.ini, a string including the dot).
Add kube_version: 1.35.8 to /root/ks/inventory/lab/group_vars/k8s_cluster/k8s-cluster.yml (it is not in the sample). Then read the actually resolved values with ansible-inventory --host node1 and write to /root/ks/vars.json the six — kube_version, container_manager, kube_network_plugin, kube_proxy_mode, kube_service_addresses, and kube_pods_subnet — as strings.
Temporarily add kube_version: 1.34.11 at the end of /root/ks/inventory/lab/group_vars/all/all.yml, look at the actual value with ansible -i /root/ks/inventory/lab/inventory.ini node1 -m debug -a var=kube_version, and look once more with -e kube_version=1.36.4 added to the same command. Write to /root/ks/precedence.json all_yml (the value you put in all.yml), effective (the value that won without -e), extra_vars (the value when -e was given), and winner_file (the path, relative to the inventory, of the file in which the value that won without -e is written), and then delete the line you put in all.yml.
/root/ks/broken/inventory.ini is an inventory in which someone wrote [masters]. Without fixing it, run ansible-playbook -i /root/ks/broken/inventory.ini playbooks/boilerplate.yml in /opt/ks/kubespray. Write to /root/ks/broken.json failed_task (the name of the failed task, without the role prefix), node1_groups (a sorted array of the names of the groups node1 belongs to in this inventory, excluding all and ungrouped), and missing_group (the name of the kubespray group that is empty even though it should not be).
In /opt/ks/kubespray, run ansible-playbook -i /root/ks/inventory/lab/inventory.ini playbooks/boilerplate.yml and save the entire output to /root/ks/logs/boilerplate.log. PLAY RECAP must show failed=0.
Write to /root/ks/report.json kubespray_tag (git -C /opt/ks/kubespray describe --tags), ansible_core (the version number on the first line of ansible --version, for example 2.19.0), kube_version_default (the default kube_version of this version), and kube_version_min (the lowest version this version accepts).

Notes

kubespray v2.32.0 is in the VM at /opt/ks/kubespray, and Ansible is in the /opt/ks/venv virtual environment. ansible, ansible-playbook, and ansible-inventory are linked into PATH.
Run the playbooks and inventory tools in /opt/ks/kubespray. The ansible.cfg in that directory tells them the roles and library paths.
Common mistake: passing a directory, as in -i /root/ks/inventory/lab. With this repository's configuration, inventory.ini is ignored.
Common mistake: when the cp -r target already exists, it copies one level deeper.
Documentation: Kubespray — Inventory · Kubespray — Ansible variable layers · Ansible — variable precedence

Copy the sample inventory to your own place

Copy /opt/ks/kubespray/inventory/sample wholesale to /root/ks/inventory/lab. group_vars/all/all.yml and group_vars/k8s_cluster/k8s-cluster.yml must be under it.

kubespray reads the group_vars next to the inventory directory. The reason to copy the sample rather than edit it is that the sample changes when you move to the next version. If the target directory already exists, cp -r creates one more subdirectory named sample inside it.

Put one node in three groups

In /root/ks/inventory/lab/inventory.ini, put the node node1 in the kube_control_plane and kube_node groups, and write etcd so that it takes kube_control_plane as a child group ([etcd:children]). Also write k8s_cluster so that it takes kube_control_plane and kube_node as children ([k8s_cluster:children]). Put the connection method not on the inventory line but in /root/ks/inventory/lab/host_vars/node1.yml as ansible_connection: local. There must be no ansible_connection in inventory.ini.

This VM is both the control node and the target node, so it uses a local connection instead of ssh. If you put that fact in host_vars, then when nodes are added you only add the name to the inventory and change the connection method in a per-node file. The group names must be exactly the spelling kubespray defined. The sample inventory.ini has no k8s_cluster — because kubespray creates it inside the playbook, which means tools outside the playbook (ansible-inventory, Ansible ad hoc) cannot read group_vars/k8s_cluster.

If you give a directory, there are 0 hosts

In /opt/ks/kubespray, run ansible-inventory --list twice, count the parsed hosts, and write to /root/ks/parse.json dir_hosts (when you gave the directory as -i /root/ks/inventory/lab), file_hosts (-i /root/ks/inventory/lab/inventory.ini), and ignored_ext (among the extensions that kubespray's ansible.cfg skips in an inventory directory, the one that hides inventory.ini, a string including the dot).

You can count the hosts by the number of keys in _meta.hostvars of the --list result. You can see which configuration file ansible-inventory reads with ansible-config dump --only-changed, and that file is looked up in the current directory.

Pin the version in group_vars

Add kube_version: 1.35.8 to /root/ks/inventory/lab/group_vars/k8s_cluster/k8s-cluster.yml (it is not in the sample). Then read the actually resolved values with ansible-inventory --host node1 and write to /root/ks/vars.json the six — kube_version, container_manager, kube_network_plugin, kube_proxy_mode, kube_service_addresses, and kube_pods_subnet — as strings.

If you do not write kube_version, the default changes with each kubespray version, so the same inventory installs a different Kubernetes on a different day. You confirm with the inventory tool whether what you wrote in the file and what Ansible actually resolved are the same. In YAML a number with one dot, like 1.35, is read as a float, but a three-part version number is a string.

If the same variable is in three places

Temporarily add kube_version: 1.34.11 at the end of /root/ks/inventory/lab/group_vars/all/all.yml, look at the actual value with ansible -i /root/ks/inventory/lab/inventory.ini node1 -m debug -a var=kube_version, and look once more with -e kube_version=1.36.4 added to the same command. Write to /root/ks/precedence.json all_yml (the value you put in all.yml), effective (the value that won without -e), extra_vars (the value when -e was given), and winner_file (the path, relative to the inventory, of the file in which the value that won without -e is written), and then delete the line you put in all.yml.

In Ansible, the group_vars of a more specific group win over all. k8s_cluster is a child of all. Extra vars given with -e win over everything — which is why the kubespray documentation tells you to narrow the use of -e to 'when overriding internal variables.' Run the commands in the kubespray directory.

What breaks if you get a group name wrong

/root/ks/broken/inventory.ini is an inventory in which someone wrote [masters]. Without fixing it, run ansible-playbook -i /root/ks/broken/inventory.ini playbooks/boilerplate.yml in /opt/ks/kubespray. Write to /root/ks/broken.json failed_task (the name of the failed task, without the role prefix), node1_groups (a sorted array of the names of the groups node1 belongs to in this inventory, excluding all and ungrouped), and missing_group (the name of the kubespray group that is empty even though it should not be).

The error message does not tell you the group name is wrong. Read which group the failed conditional expression was looking for when it failed. kubespray moves the old name kube-master over for you, but not masters. You can see the groups a node belongs to (including those reached through child groups) with ansible ... -m debug -a var=group_names.

Your inventory passes the check

In /opt/ks/kubespray, run ansible-playbook -i /root/ks/inventory/lab/inventory.ini playbooks/boilerplate.yml and save the entire output to /root/ks/logs/boilerplate.log. PLAY RECAP must show failed=0.

boilerplate.yml is the bundle of checks that cluster.yml, upgrade-cluster.yml, and reset.yml all call at the very start. What gets blocked here is blocked before the installation even begins, so if you run it separately before installing, you avoid the situation of stopping after a minute while running a long playbook.

The range this version of kubespray accepts

Write to /root/ks/report.json kubespray_tag (git -C /opt/ks/kubespray describe --tags), ansible_core (the version number on the first line of ansible --version, for example 2.19.0), kube_version_default (the default kube_version of this version), and kube_version_min (the lowest version this version accepts).

kubespray does not write the default and minimum versions separately; it computes them as the first and last keys of the kubelet checksum list in roles/kubespray_defaults/vars/main/checksums.yml (roles/kubespray_defaults/defaults/main/main.yml). A version with no checksum cannot be installed.