An OS with nowhere to log in — why Talos removed the shell

One-line summary

Talos Linux is an OS that runs only Kubernetes, with no shell, SSH, or packages, and instead of logging in to a node, you read and change everything through the gRPC API open on each node.

Why this was needed

Once you have about thirty nodes, something like this is bound to happen. During an outage, someone SSHes into one node, changes sysctl, edits the kubelet arguments, and puts out the fire. No record is left, and three months later only that node behaves strangely. Even if you attach a configuration management tool, "things touched outside the management tool" remain possible.

Talos blocks this problem not with rules but with structure. The philosophy document says there is no shell, no SSH, no GNU utilities, not even busybox. The first process the kernel starts is not systemd but machined, written in Go, and the root filesystem is SquashFS, so it cannot be written to. With nowhere to log in, there is nothing to tamper with. Instead, all the state of a node is determined by a single declarative machine configuration, and the only way to change it is one API.

How it works

There are few components. apid receives gRPC requests on port 50000 and passes them to machined, and if it is a control plane, it also relays to the apid of other nodes. machined runs only a fixed set of services (containerd, etcd, kubelet, trustd, and so on) and accepts no arbitrary user services. talosctl is the client of this API; -e decides which node's apid to go through, and -n decides which node answers.

The internal state is expressed as resources and controllers, as in Kubernetes. Services, cluster members, and the machine configuration are all resources, so they can be read with the same command.

talosctl -n 10.5.0.2 services                    # machined 가 돌리는 서비스
talosctl -n 10.5.0.2 get members                 # 클러스터 멤버(discovery)
talosctl -n 10.5.0.3 get mc v1alpha1 -o yaml     # 머신 설정도 버전이 붙은 리소스
talosctl -n 10.5.0.3 logs kubelet                # 셸 대신 로그도 API 로

The commands that change the configuration are three — apply-config, edit machineconfig, and patch machineconfig — and the way to apply is chosen with --mode. According to the editing machine configuration documentation (v1.13), auto reboots if needed, no-reboot applies immediately but rejects the change if the field requires a reboot, staged applies at the next reboot, and try applies immediately and then reverts if there is no other change within a set time. Items such as nodeLabels, kubelet, and network are on the list of those that take effect without a reboot. A patch is a strategic merge YAML that writes only part of the configuration, and lists are normally appended to, but cluster.network.podSubnets and serviceSubnets are overwritten. A specific key is removed with $patch: delete.

Locally, as in the Docker guide, you can bring up nodes as containers with talosctl cluster create docker. In container mode, APIs such as upgrade and reset do not apply.

What it looks like in the field

This was measured on this course's VM. If you run docker exec ... sh on a node container, you get exit 127 with executable file not found, port 22 of the node IP is refused, and only port 50000 is open. The habitual "let's just get in first" is blocked on the very first line.

The CIDR trap showed up as well. The default ranges talosctl creates are Pod 10.244.0.0/16 and Service 10.96.0.0/12, but this VM's own address is in the 10.244 range of the host cluster. So I brought it up giving 10.200/10.201 with --config-patch.

The choice of version was also decided by measurement. When I brought it up with the default image of talosctl v1.14.0, the node log showed FSCONFIG_SET_FD failed ... lowerdir+ while creating the /etc overlay, and the API never became ready (host kernel 6.8). On the same VM, v1.13.10 had both nodes Ready in about 2 minutes. Talos inside a container uses the host kernel as is, so when you raise the version, the host kernel comes first as a condition.

The most important measurement is the limit of validation. A patch that put a space in a label name was rejected with InvalidArgument and the configuration version stayed the same. On the other hand, when a flag that does not exist was put into the kubelet's extraArgs, it was accepted with "Applied configuration without a reboot", and the kubelet restarted every 5 seconds with unknown flag. Validation looks only at the document format and does not know whether the kubelet recognizes that flag. With no shell, I looked for the cause in LAST EVENT of talosctl services and in talosctl logs kubelet, and when I removed just that key with $patch: delete, the kubelet became healthy within 3 seconds.

What really matters in practice

The machine configuration file is the node. There is nothing edited on the node, and the version of the configuration resource is the backbone of the change history. It becomes natural to keep patch files in a repository and apply the same file to several nodes.

Use try mode as a safety line for remote changes. Put in with try any setting that, if changed wrongly, might leave you unable to reach the API again, such as networking, and once confirmed, apply the same change again to make it permanent. Also remember that if you put in another change while you are confirming, the revert is canceled.

Do not read passing validation as success. After applying, always check the service state and node Ready again. And --dry-run only shows the merged result and does not validate (measured).

Accept that the only means of emergency treatment is the API. If you lose the network that can reach apid and the talosconfig certificate, there is practically no way to repair the node. Safekeeping of the certificate and the access path become more important than SSH key management.

What you will do in the next lab

On two Talos nodes brought up with Docker inside the VM, you leave evidence that there is no shell, read the services, members, and machine configuration, and get the kubeconfig through the API to check the CIDR ranges. Then you attach a label to the worker with no-reboot, and see the label attached with try revert after 30 seconds. Finally, you put in, one after the other, a patch that validation rejects and a patch that is accepted but kills the kubelet, find the cause from the logs, revert it, and summarize in a report.