Building clusters with Kubespray and Terraform

Renew one-year certificates, then wipe and rebuild

Goal

You renew the kubeadm certificates by hand and make the control plane use the new certificates, then turn on kubespray's automatic renewal timer and check when it actually renews. Finally, you erase the cluster with reset.yml, see what remains, and build it again from scratch with the same inventory.

Why it matters

The leaf certificates kubeadm creates are valid for one year. If you upgrade even once within a year, kubeadm renews them along with it, but in a cluster that did not, one day authentication between components is rejected all at once. Renewal does not end with changing the files — the processes that read those files must be using the new certificates. And there are days when erasing and rebuilding is faster than reviving a half-broken cluster. You need to know what reset.yml deletes and what it leaves to trust the phrase "clean reinstall." You wait about 15 minutes in total for two installations and one reset, so extend the session if needed.

Steps

In /opt/ks/kubespray, run ansible-playbook -i /root/ks/inventory/lab/inventory.ini cluster.yml and leave the entire output in /root/ks/logs/cluster-1.log (about 7 minutes). node1 in the PLAY RECAP must have failed=0.
Write to /root/ks/certs/before.json apiserver_serial (the serial of /etc/kubernetes/ssl/apiserver.crt, as the hexadecimal openssl prints it), apiserver_not_after (the notAfter string from openssl, as is), admin_client_serial (the serial of client-certificate-data in /etc/kubernetes/admin.conf), ca_serial (the serial of /etc/kubernetes/ssl/ca.crt), and node_uid.
Renew the kubeadm certificates with kubeadm certs renew all, then start the kube-apiserver, kube-controller-manager, and kube-scheduler static Pods again so that the control plane uses the new certificates, and copy /etc/kubernetes/admin.conf to /root/.kube/config. When you are done, the serial of apiserver.crt must have changed, the certificate served on 6443 must be the same as that file, and the controller-manager and scheduler containers must have started after their respective kubeconfigs changed.
Change auto_renew_certificates in /root/ks/inventory/lab/group_vars/k8s_cluster/k8s-cluster.yml to true, rerun only that part with cluster.yml --tags control-plane, and leave the entire output in /root/ks/logs/cp-tags.log. When you are done, k8s-certs-renew.timer must be enabled and active.
Run the renewal service once now with systemctl start k8s-certs-renew.service and look at the result with journalctl -u k8s-certs-renew.service. Write to /root/ks/certs/timer.json oncalendar (the timer's OnCalendar value), renewed (whether this run renewed the certificates, a boolean), and decision_line (the line that starts with ## and states whether it renewed, as is).
In /opt/ks/kubespray, run ansible-playbook -i /root/ks/inventory/lab/inventory.ini reset.yml -e reset_confirmation=yes and leave the entire output in /root/ks/logs/reset.log (about a minute and a half). Then write to /root/ks/certs/remnants.json hostname (the current hostname), releases_kept (whether /tmp/releases remains, a boolean), and left_bins (a sorted array of the names of the regular files remaining in /usr/local/bin, excluding labhub-agent — symbolic links are excluded).
Run cluster.yml again with the same inventory and leave the entire output in /root/ks/logs/cluster-2.log. When you are done, node1 must be Ready, and as evidence that it is a new cluster, the CA serial and the node UID must differ from the records in step 2.

Notes

kubespray v2.32.0 is ready at /opt/ks/kubespray and the inventory at /root/ks/inventory/lab/inventory.ini (kube_version 1.35.8).
kubespray puts the kubeadm certificates in /etc/kubernetes/ssl (/etc/kubernetes/pki is a link pointing there).
Common mistake: stopping after only kubeadm certs renew all. The files are new but the processes may keep running with the old kubeconfig.
Common mistake: leaving ~/.kube/config as it is after renewal. On the day the old certificate expires, only kubectl is suddenly rejected.
Common mistake: running reset.yml without the confirmation variable and not realizing it stopped at the prompt.
Documentation: Kubernetes — Certificate Management with kubeadm · Kubespray — Getting started (reset)

Build the cluster to renew and erase

In /opt/ks/kubespray, run ansible-playbook -i /root/ks/inventory/lab/inventory.ini cluster.yml and leave the entire output in /root/ks/logs/cluster-1.log (about 7 minutes). node1 in the PLAY RECAP must have failed=0.

It is the same installation as in the earlier modules. Start it with systemd-run or tmux so that it keeps running even if the console is cut off, and pass HOME=/root.

The serials before renewal

Write to /root/ks/certs/before.json apiserver_serial (the serial of /etc/kubernetes/ssl/apiserver.crt, as the hexadecimal openssl prints it), apiserver_not_after (the notAfter string from openssl, as is), admin_client_serial (the serial of client-certificate-data in /etc/kubernetes/admin.conf), ca_serial (the serial of /etc/kubernetes/ssl/ca.crt), and node_uid.

openssl x509 -noout -serial prints the serial as serial=.... The certificate inside a kubeconfig is base64, so decode it and pass it to openssl. These values become the basis later for telling 'is it really a new certificate' and 'is it really a new cluster.'

Renew, and make it use the new certificates

Renew the kubeadm certificates with kubeadm certs renew all, then start the kube-apiserver, kube-controller-manager, and kube-scheduler static Pods again so that the control plane uses the new certificates, and copy /etc/kubernetes/admin.conf to /root/.kube/config. When you are done, the serial of apiserver.crt must have changed, the certificate served on 6443 must be the same as that file, and the controller-manager and scheduler containers must have started after their respective kubeconfigs changed.

When renewal is done, kubeadm prints 'restart.' Static Pods are started by the kubelet, so if you delete the Pod sandbox (crictl pods --name ... -q | xargs crictl rmp -f), the kubelet sees the manifest and starts it again. kubespray's /usr/local/bin/k8s-certs-renew.sh uses the same order — read it. It is normal for the API not to respond for a few seconds after you delete.

Leave renewal to the calendar

Change auto_renew_certificates in /root/ks/inventory/lab/group_vars/k8s_cluster/k8s-cluster.yml to true, rerun only that part with cluster.yml --tags control-plane, and leave the entire output in /root/ks/logs/cp-tags.log. When you are done, k8s-certs-renew.timer must be enabled and active.

This switch makes the control plane role install a systemd timer and service. When it runs is decided by auto_renew_certificates_systemd_calendar, and systemctl list-timers shows the next run time. The default is the first Monday of every month.

When does the timer actually renew

Run the renewal service once now with systemctl start k8s-certs-renew.service and look at the result with journalctl -u k8s-certs-renew.service. Write to /root/ks/certs/timer.json oncalendar (the timer's OnCalendar value), renewed (whether this run renewed the certificates, a boolean), and decision_line (the line that starts with ## and states whether it renewed, as is).

The script renews and restarts the control plane only when there is a certificate that expires before the point of the next timer time plus a 7-day margin. The certificates you just renewed have a year left, so predict first what will happen and then check. The script body is in /usr/local/bin/k8s-certs-renew.sh.

Erase with reset.yml — what remains

In /opt/ks/kubespray, run ansible-playbook -i /root/ks/inventory/lab/inventory.ini reset.yml -e reset_confirmation=yes and leave the entire output in /root/ks/logs/reset.log (about a minute and a half). Then write to /root/ks/certs/remnants.json hostname (the current hostname), releases_kept (whether /tmp/releases remains, a boolean), and left_bins (a sorted array of the names of the regular files remaining in /usr/local/bin, excluding labhub-agent — symbolic links are excluded).

reset.yml is an irreversible job, so it requires a confirmation variable. What it deletes is in the list in roles/reset/tasks/main.yml — what is not on that list remains. If you know what remains, you can decide how far to trust the phrase 'erased cleanly.'

Build it again from scratch

Run cluster.yml again with the same inventory and leave the entire output in /root/ks/logs/cluster-2.log. When you are done, node1 must be Ready, and as evidence that it is a new cluster, the CA serial and the node UID must differ from the records in step 2.

reset deletes /etc/kubernetes wholesale, so the CA disappears too. When you build again, a new CA is created, and an old kubeconfig signed by the old CA can no longer be used. The downloaded file cache (/tmp/releases) remains, so it is a little faster than the first installation.