Kubernetes Distributions — Build Them Yourself

I deleted traefik and it came back after a restart

Goal

On a real k3s, you confirm the structure in which the k3s server applies the files in the manifests directory as AddOns and installs traefik as a HelmChart. You leave a record of what each of these changes and what it leaves behind: deleting with kubectl, restarting, .skip, --disable, HelmChartConfig, and the precedence between the configuration file and CLI flags.

Why it matters

From a one-line install, k3s gives you a cluster equipped with Ingress, DNS, and even metrics. That convenience comes from the design in which "k3s rewrites and applies the manifests of the default components at every startup." So if you delete a component with kubectl as you would on a managed cluster, it quietly comes back at the next restart, and conversely, even if you delete the manifest file, the cluster object remains. Turning a component off is --disable, ignoring only the file is .skip, and changing only the values is HelmChartConfig. If you don't know the difference among the three, you end up tracing "who installed this again?" after an upgrade or a reboot. The same goes for the configuration file: if you don't know that the flags the install script put into the service unit take precedence over config.yaml, you cannot find out why the value you edited was not applied.

Steps

Write to /root/k3s-pkg/inventory.json the fields version (the kubeletVersion of the node), addons (a sorted array of the AddOn names in kube-system), helmcharts (a sorted array of the HelmChart names in kube-system), and manifest_files (a sorted array of the relative paths of the files under the manifests directory).
In /var/lib/rancher/k3s/server/manifests/lab-banner.yaml, write a Namespace k3s-pkg and a ConfigMap banner (namespace k3s-pkg, msg: v1) and deploy them as an AddOn. Delete the resulting ConfigMap with kubectl, wait 20 seconds and confirm that it does not come back, then change the value in the file to msg: v2 so that it is created again. Write to /root/k3s-pkg/banner.json the fields deleted_uid (the UID of the deleted one), restored_uid (the UID of the recreated one), and returned_before_edit (whether it came back before you edited the file, a boolean).
Delete the HelmChart traefik in kube-system with kubectl and wait until the traefik Deployment disappears (do not restart yet). Write to /root/k3s-pkg/gone.json the fields deleted_at (the time of deletion, UTC RFC3339, for example 2026-01-01T00:00:00Z), manifest_still_there (whether the traefik.yaml file still exists, a boolean), and release_left (the number of remaining traefik release Secrets, an integer).
Restart k3s with systemctl restart k3s and wait until the traefik Deployment becomes Available again. Write to /root/k3s-pkg/revive.json the fields helmchart_uid (the UID of the recreated HelmChart traefik), helmchart_created (its creationTimestamp), and revived (whether the Deployment came back, a boolean).
Deploy a ConfigMap pinned (namespace k3s-pkg, v: "1") with /var/lib/rancher/k3s/server/manifests/lab-pinned.yaml. After confirming that it was created, create lab-pinned.yaml.skip in the same directory, change the value in the file to v: "2", and wait 20 seconds. Write to /root/k3s-pkg/skip.json the fields file_value (the value written in the file), live_value (the value of the ConfigMap in the cluster), and addon_still_exists (whether the AddOn lab-pinned remains, a boolean).
Leave /etc/rancher/k3s/config.yaml as it is, and create /etc/rancher/k3s/config.yaml.d/50-disable.yaml to turn metrics-server off. local-storage must stay off. Right after writing the file, confirm for 20 seconds that metrics-server is unchanged, and then restart k3s. Write to /root/k3s-pkg/disable.json the fields changed_before_restart (whether it disappeared before the restart, a boolean), removed_addons (a sorted array of the AddOn names that disappeared because of the restart), and files_left (the number of files left under manifests/metrics-server, an integer).
In /etc/rancher/k3s/config.yaml, add write-kubeconfig-mode: "0600" while keeping the existing disable list, and restart k3s. Look at what happened to the permissions of /etc/rancher/k3s/k3s.yaml and write to /root/k3s-pkg/precedence.json the fields config_value (the value written in config.yaml, a string), effective_mode (the octal permission of k3s.yaml after the restart, a string, for example "640"), winner (cli or config), and flag_file (the absolute path of the file in which the winning value is written).
Without restarting, place a HelmChartConfig traefik (kube-system) in /var/lib/rancher/k3s/server/manifests/traefik-config.yaml to change the traefik log level to DEBUG (logs.general.level). Write to /root/k3s-pkg/hcc.json the fields revision_before (the highest revision of the traefik release before applying, an integer), revision_after (after applying, an integer), and log_arg (the whole log level argument among the traefik container arguments).
Write to /root/k3s-pkg/report.json the fields needs_restart (an object: for each of manifest_file, skip_file, helmchartconfig, and config_dropin, a boolean for whether a restart was needed for it to take effect), revived_by_restart (the name of the HelmChart that came back to life through a restart in steps 3 and 4), disable_deletes_files (a boolean), skip_removes_resources (a boolean), current_addons (a sorted array of the AddOn names currently in kube-system), and traefik_revision (the highest revision of the current traefik release, an integer).

Notes

There is one k3s v1.35.8+k3s1 in the VM. The recipe turns off only local-storage through config.yaml, and --write-kubeconfig-mode 644 was passed to the install script.
A restart is systemctl restart k3s. The API comes back in about 5 seconds and traefik in about 20 seconds (measured).
To view AddOns: kubectl -n kube-system get addon -o custom-columns=NAME:.metadata.name,SOURCE:.spec.source
Common mistake: writing disable: in a drop-in. The last value wins, so local-storage from config.yaml is installed again.
Common mistake: editing the manifest files of the default components (traefik.yaml and so on) directly. They are rewritten at startup and your edit disappears.
Managing Packaged Components · Helm · Configuration Options

What was installed as an AddOn

Write to /root/k3s-pkg/inventory.json the fields version (the kubeletVersion of the node), addons (a sorted array of the AddOn names in kube-system), helmcharts (a sorted array of the HelmChart names in kube-system), and manifest_files (a sorted array of the relative paths of the files under the manifests directory).

AddOn is a CRD in the k3s.cattle.io group, and one file is one AddOn. Files in subdirectories each become an AddOn as well. local-storage has already been turned off in the recipe's config.yaml, so it should not be in the list.

When does a ConfigMap deleted with kubectl come back

In /var/lib/rancher/k3s/server/manifests/lab-banner.yaml, write a Namespace k3s-pkg and a ConfigMap banner (namespace k3s-pkg, msg: v1) and deploy them as an AddOn. Delete the resulting ConfigMap with kubectl, wait 20 seconds and confirm that it does not come back, then change the value in the file to msg: v2 so that it is created again. Write to /root/k3s-pkg/banner.json the fields deleted_uid (the UID of the deleted one), restored_uid (the UID of the recreated one), and returned_before_edit (whether it came back before you edited the file, a boolean).

The deploy controller applies when a file changes and when k3s starts. The fact that an object in the cluster was deleted is not a trigger for applying. The AddOn remembers the checksum of the file, so a change with identical content, such as a touch, is not a trigger either.

You deleted traefik with kubectl

Delete the HelmChart traefik in kube-system with kubectl and wait until the traefik Deployment disappears (do not restart yet). Write to /root/k3s-pkg/gone.json the fields deleted_at (the time of deletion, UTC RFC3339, for example 2026-01-01T00:00:00Z), manifest_still_there (whether the traefik.yaml file still exists, a boolean), and release_left (the number of remaining traefik release Secrets, an integer).

The HelmChart has the helm-controller's finalizer attached, so when you delete it, the helm-delete Job removes the release first. A Helm release remains as a Secret in kube-system with the label owner=helm,name=traefik. The manifests directory is not touched even if you delete the cluster object.

After the restart, traefik came back to life

Restart k3s with systemctl restart k3s and wait until the traefik Deployment becomes Available again. Write to /root/k3s-pkg/revive.json the fields helmchart_uid (the UID of the recreated HelmChart traefik), helmchart_created (its creationTimestamp), and revived (whether the Deployment came back, a boolean).

At every startup, k3s rewrites the manifests of the default components to disk and applies them. If the HelmChart created by the file is missing, it is created anew, and then the helm-install Job runs again. There is a gap of about 20 seconds between the API becoming ready and traefik coming up.

.skip pretends not to see it instead of deleting it

Deploy a ConfigMap pinned (namespace k3s-pkg, v: "1") with /var/lib/rancher/k3s/server/manifests/lab-pinned.yaml. After confirming that it was created, create lab-pinned.yaml.skip in the same directory, change the value in the file to v: "2", and wait 20 seconds. Write to /root/k3s-pkg/skip.json the fields file_value (the value written in the file), live_value (the value of the ConfigMap in the cluster), and addon_still_exists (whether the AddOn lab-pinned remains, a boolean).

A .skip file counts only by its existence, not its content. It leaves the AddOn that has already been applied and the objects it created as they are, and ignores later changes to that file.

Turn off metrics-server with a single drop-in

Leave /etc/rancher/k3s/config.yaml as it is, and create /etc/rancher/k3s/config.yaml.d/50-disable.yaml to turn metrics-server off. local-storage must stay off. Right after writing the file, confirm for 20 seconds that metrics-server is unchanged, and then restart k3s. Write to /root/k3s-pkg/disable.json the fields changed_before_restart (whether it disappeared before the restart, a boolean), removed_addons (a sorted array of the AddOn names that disappeared because of the restart), and files_left (the number of files left under manifests/metrics-server, an integer).

Drop-ins are read in name order, and if the same key is in several files, the last value wins. To append to a list, add + to the end of the key name. --disable not only removes the AddOn but also deletes the original file. The configuration file is read when k3s starts.

You edited config.yaml but the permissions stay the same

In /etc/rancher/k3s/config.yaml, add write-kubeconfig-mode: "0600" while keeping the existing disable list, and restart k3s. Look at what happened to the permissions of /etc/rancher/k3s/k3s.yaml and write to /root/k3s-pkg/precedence.json the fields config_value (the value written in config.yaml, a string), effective_mode (the octal permission of k3s.yaml after the restart, a string, for example "640"), winner (cli or config), and flag_file (the absolute path of the file in which the winning value is written).

If the configuration file and the CLI arguments have the same key, the CLI takes precedence. INSTALL_K3S_EXEC passed to the install script is stored as an argument in the ExecStart of the systemd unit. Look at it with systemctl cat k3s.

Change only the traefik values without a restart

Without restarting, place a HelmChartConfig traefik (kube-system) in /var/lib/rancher/k3s/server/manifests/traefik-config.yaml to change the traefik log level to DEBUG (logs.general.level). Write to /root/k3s-pkg/hcc.json the fields revision_before (the highest revision of the traefik release before applying, an integer), revision_after (after applying, an integer), and log_arg (the whole log level argument among the traefik container arguments).

A HelmChartConfig must have the same name and namespace as its target HelmChart. Its valuesContent takes precedence over the HelmChart's valuesContent and is weaker than spec.set. The helm-controller runs a helm upgrade, so the release Secret count increases by one (sh.helm.release.v1.traefik.vN).

Deleting, ignoring, and overriding in one table

Write to /root/k3s-pkg/report.json the fields needs_restart (an object: for each of manifest_file, skip_file, helmchartconfig, and config_dropin, a boolean for whether a restart was needed for it to take effect), revived_by_restart (the name of the HelmChart that came back to life through a restart in steps 3 and 4), disable_deletes_files (a boolean), skip_removes_resources (a boolean), current_addons (a sorted array of the AddOn names currently in kube-system), and traefik_revision (the highest revision of the current traefik release, an integer).

Write them based on the json files you left in earlier steps and the current cluster. The grader recalculates the same values from the cluster and from the record files.