TT Lab
Get started
Learn Learning paths Courses

CKAD — Kubernetes Application Developer

What Changes When Configuration Leaves the Image

Continue in TT Lab

In one line

ConfigMaps and Secrets are not a "configuration store" but injection points that let you reuse the same image across many environments, and securityContext and resources are not "options" but a contract for what that container may do on the node and how much it may use.

Why this was needed

If you bake configuration into the image, you have to build a different image for development, staging, and production. Then "the image that passed in staging" and "the image that went to production" become different things, and the meaning of the test disappears. So you split it into an immutable artifact (the image) + mutable configuration (ConfigMap/Secret).

There are two ways to inject it. Environment variables are read once when the process starts and never change after that. Volume mounts come in as files, and when you modify the ConfigMap, the kubelet updates the file contents (except that items mounted with subPath are not updated). If the app can reread its configuration, a volume is convenient; if it reads only once at startup, environment variables are convenient.

Resources has two faces. requests is the value the scheduler looks at. The node must have that much room before the Pod is placed on it. limits is the value the kubelet/runtime enforces. CPU is throttled, and memory gets OOMKilled when exceeded. The relationship between the two determines the QoS class, and decides who gets evicted first when the node is under pressure.

How it works

The options for injecting configuration, in a table:

Method Field Characteristics
Only one key as an environment variable env[].valueFrom.configMapKeyRef You can inject it under a different name
Everything as environment variables envFrom[].configMapRef The key name becomes the variable name as is
As files volumes[].configMap + volumeMounts Updates are reflected; set permissions with defaultMode
Only one file into an existing directory volumeMounts[].subPath Does not cover the directory, not updated

With optional: true, the Pod starts even when the referenced target does not exist. Use it for apps that run on defaults when it is missing. A ConfigMap/Secret with immutable: true cannot be modified, but in exchange the kubelet does not watch it for changes, which reduces API server load on large clusters.

A Secret is base64 encoding, not encryption. It goes into etcd close to plaintext. Real protection comes from three things: narrowing with RBAC who can read it, setting --encryption-provider-config on the apiserver to encrypt at rest, and mounting into a Pod only what it needs.

securityContext exists separately at the Pod level and the container level. At the Pod level (spec.securityContext), runAsUser, runAsNonRoot, and fsGroup apply by default to all containers; at the container level (spec.containers[].securityContext), capabilities, readOnlyRootFilesystem, and allowPrivilegeEscalation apply only to that container and override the Pod values. fsGroup exists only at the Pod level because a volume is a per-Pod resource.

QoS is a read-only value that is determined automatically.

LimitRange and ResourceQuota are a pair. LimitRange sets the default, minimum, and maximum for an individual container, and ResourceQuota limits the total across the whole namespace. In a namespace with a quota, a Pod that does not specify requests/limits is rejected outright, but if a LimitRange fills in the defaults, it passes. That is why you set both together.

What it looks like in the field

This happened when I attached my homelab NAS as a Kubernetes volume. To check the NFS mount, I started a privileged: true Pod, and it failed like this.

mount.nfs: Operation not permitted for 10.0.0.109:/volume1/k8s on /mnt/t
mount: permission denied (are you root?)

It was blocked even though I had given it the maximum privileges. The cause was a missing hostNetwork: true. An NFS mount talks to rpcbind and uses a reserved port below 1024 as the source port, and that behavior is constrained inside a Pod network namespace. privileged grants capabilities, but it does not solve a network namespace problem. It was a costly lesson that problems solved by securityContext and problems solved by other fields of the Pod spec are different.

On the GPU side of the same cluster there was a lesson in the opposite direction. Writing just one line, resources.limits: {nvidia.com/gpu: 1}, in the Pod was enough for the scheduler to pick a node with a GPU and for the runtime to insert the device. You can see directly that requests are the language of scheduling. But a new problem arose here. The GPUs on the four workers are 24GB (3090), 32GB (5090), and 8GB (4070 Laptop) ×2, yet from Kubernetes' point of view they are all "1 GPU." A training job that needed 32GB actually ended up on an 8GB laptop GPU. In the end I attached semantic labels such as gpu.homelab/tier=xlarge|large|small myself and made Pods choose with a nodeSelector. I filled in the limitation that a resource request expresses only quantity and cannot express quality with labels.

What you will do in the next lab

In the ckad-config namespace, you create ConfigMaps from literals and files, and specify envFrom, configMapKeyRef, volume mounts, defaultMode, subPath, optional, and immutable all by hand. Then, in the ckad-secure namespace, you create a ServiceAccount assignment, Pod- and container-level securityContext, resources, a LimitRange, and a ResourceQuota, and check what QoS class is actually assigned.