TT Lab
Get started
Learn Learning paths Courses

Container Internals

Apply Limits and Compare Against What You Observe

Continue in TT Lab

This lab runs on a real VM

This box is not a Pod but a virtual machine started by KubeVirt. A Linux kernel of its own runs, systemd actually manages services, and docker is a real Docker engine, not an imitation. A container started with docker run becomes an actual process, and docker exec and docker logs work as usual.

This lab used to run inside a Pod. Since it was a box with all kernel capabilities dropped, the step of starting a container was blocked, so you learned through a workaround of unpacking the image archive yourself. The workaround is no longer needed.

There are two things to know.

Goal

You apply memory, CPU, and PID limits yourself and compare the requested values (HostConfig in docker inspect) side by side with the values the container actually sees (/sys/fs/cgroup/*). You create exit code 137 yourself and sort out how to tell it apart from an OOM kill.

Why it matters

The questions that come up in operations are almost always "we set a limit, so why isn't it being respected?" or "the CPU is idle, so why is it slow?" The first comes from not knowing that the requested value and the enforced value can differ (in a rootless environment, without systemd delegation the request is only recorded), and the second comes from not knowing that the CPU cgroup does not kill the process but puts it to sleep. And both can be confirmed only by reading the cgroup files directly, not from docker stats or a dashboard. So the purpose of this lab is to build the habit of reading a value in two places and comparing them. If you read only one, you will surely reach the wrong conclusion.

Steps

  1. Create the /root/int2 directory and save the output of stat -fc %T /sys/fs/cgroup as it is to /root/int2/version.txt (cgroup2fs means v2, tmpfs means v1).
  2. Save the contents of the /sys/fs/cgroup/cgroup.controllers file as they are to /root/int2/controllers.txt.
  3. From the alpine:3.20 image, run a dk-mem container in the background with a memory limit of 64 MiB (67108864 bytes). It must still be running at grading time.
  4. Inside dk-mem, read /sys/fs/cgroup/memory.max and save that value as it is to /root/int2/memmax.txt.
  5. From the alpine:3.20 image, run a dk-cpu container in the background with a CPU limit of 0.5 cores.
  6. Start a dk-killed container and then send SIGKILL yourself so that it ends with exit code 137 (it must not be an OOM). Then write in /root/int2/exit137.txt the breakdown that 137 is made up of 128 + 9, and the point that an OOM kill also gives the same 137.
  7. From the alpine:3.20 image, run a dk-pids container with a PID limit of 64.
  8. Write three rows in /root/int2/limits.md. The dk-mem row must include the actual byte value 67108864, the dk-cpu row must include 0.5 (or 500m), and the dk-pids row must include 64.

Notes

Determine the cgroup version

Create the /root/int2 directory and save the output of stat -fc %T /sys/fs/cgroup as it is to /root/int2/version.txt (cgroup2fs means v2, tmpfs means v1).

There is a stat option that asks for the type of a mounted filesystem. Do not interpret the resulting string and write v1/v2; save the string that comes out as it is.

The list of available controllers

Save the contents of the /sys/fs/cgroup/cgroup.controllers file as they are to /root/int2/controllers.txt.

cgroup v2 keeps a file in the root hierarchy that lists the available controller names on a single line. Do not copy them by hand; copy the contents of that file as they are. Differences in whitespace are allowed, but no item may be missing.

Request a 64 MiB memory limit

From the alpine:3.20 image, run a dk-mem container in the background with a memory limit of 64 MiB (67108864 bytes). It must still be running at grading time.

The memory limit flag accepts suffixes such as m and g. 64 MiB is exactly 67108864 bytes, and grading looks at this number. The container must be up at grading time, so do not start it with a command that finishes soon.

The memory.max the container actually sees

Inside dk-mem, read /sys/fs/cgroup/memory.max and save that value as it is to /root/int2/memmax.txt.

Read /sys/fs/cgroup/memory.max inside the container. The value may come out as max rather than 67108864, and that is not a wrong answer but the heart of this step — the requested value and the enforced value can differ. Save it as observed.

A CPU limit of 0.5 cores

From the alpine:3.20 image, run a dk-cpu container in the background with a CPU limit of 0.5 cores.

There is a flag that takes the number of cores as a decimal. Internally the quota becomes half of the period, which means not 'cut the core in half' but 'run for only half of every period'.

Create exit code 137 and interpret it

Start a dk-killed container and then send SIGKILL yourself so that it ends with exit code 137 (it must not be an OOM). Then write in /root/int2/exit137.txt the breakdown that 137 is made up of 128 + 9, and the point that an OOM kill also gives the same 137.

You must not kill it by blowing up its memory but send SIGKILL directly (if OOMKilled is true, this step fails). And in the memo file, write the explanation that breaks 137 down into 128 and 9, together with the fact that an OOM kill also gives exactly 137.

Stop a fork bomb with a PID limit

From the alpine:3.20 image, run a dk-pids container with a PID limit of 64.

There is a separate flag that sets an upper limit on the number of processes. Memory or CPU limits cannot stop a fork bomb — each process is small, but by sheer count they exhaust kernel resources.

Write the limits summary table

Write three rows in /root/int2/limits.md. The dk-mem row must include the actual byte value 67108864, the dk-cpu row must include 0.5 (or 500m), and the dk-pids row must include 64.

All three rows must contain values you confirmed with docker inspect, not guesses. Memory must go in as the byte number as is, CPU must be recognizable as 0.5, and PIDs must go in as the number as is.