Reading containerd and CRI: How a gRPC request from kubelet goes down to runc

Status: Draft (2026-10-17), partially measured. Reference versions: containerd/containerd v2.3.5 (commit 1294c24a7da8), kubernetes/cri-api v0.34.0 (commit 9b3ce5ec4798). Comparisons with 1.7 are limited to what was verified by opening containerd v1.7.27 (commit 05044ec0a9a7) together in the source code. Measurements were taken on 2026-10-17 in a small cluster by reading only (reading the node list, sshing into two worker nodes as a regular user, ps·mountand reading configuration files without sudo). Writing convention: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items as "Unverified".

Table of Contents

• 1. One-line Summary: The Order Form and the Kitchen
• 2. Architecture Diagram
• 3. Code Map
  • Differences between 1.7 and 2.x (verified via source code only)
• 4. Following the Core Path
  • 4.1 Daemons Are Assembled via a Plugin Graph
  • 4.2 RunPodSandbox: Network First, Pause Handled by the Controller
  • 4.3 CreateContainer·StartContainer → tasks → shim → runc
  • 4.4 Image Pull and Snapshot: From Layers to Rootfs
• 5. Connection to User Contributions
• 6. Measurement: Unprivileged Reading of 1.7 and 2.3 Workers
• 7. Where to Read First
• 8. Good Places for a First Contribution
• 9. Limitations
• References

1. One-line Summary: The Order Form and the Kitchen

Kubelet does not create containers directly. It simply sends orders like "Set up a sandbox for this pod," "Create a container with this image," and "Start it" in a standardized format called CRI (Container Runtime Interface) . Because the order format is standardized, the kitchen (runtime) can be swapped between containerd, CRI-O, etc. containerd receives the order, unpacks and prepares the ingredients (image layers) (snapshots), assigns a staff member (shim) to each table, and delegates the actual ignition to the chef (runc). The chef sets up the dish and leaves immediately, while the assigned staff member attends to the rest.

2. Architecture Diagram

• A single pod creation flows from top to bottom. Only the communication between kubelet and containerd uses gRPC (Unix socket), while the communication between the CRI plugin and core services is in-process function calls.
• The three cylinders represent state persisted on disk: meta.db(bolt metadata), blobs/sha256(content store), and snapshots/N(overlayfs snapshotter).
• Communication between containerd and the shim uses ttrpc (a lightweight gRPC variant). There is one shim per pod, and runc disappears once its work is done.
• CNI is drawn in orange because it is a binary outside containerd.

• The stack on the left represents the snapshots/<번호>/fs directory of the overlayfs snapshotter. The bottom three are committed read-only layers, and the top one is a newly created writable layer for each container.
• Numbers ① to ④ on the right correspond to the function execution order tracked in Section 4.4.

3. Code Map

The containerd 2.x repository is a single Go module (github.com/containerd/containerd/v2). Only the essential parts for an initial look are summarized below.

kubernetes/cri-api consists essentially of a single file. pkg/apis/runtime/v1/api.proto contains all 35 RPCs and messages, and the rest is generated Go code.

Differences between 1.7 and 2.x (verified via source code only)

• CRI's location. In 1.7.27, there were two implementations under pkg/cri/ : the default server/(108 Go files) and the experimental sbserver/(98 files), chosen via the environment variable ENABLE_CRI_SANDBOXES (pkg/cri/cri.go:102). Version 2.x adopted the sandbox-side implementation as default and moved it to internal/cri/ , also splitting the plugin into three: the gRPC-receiving io.containerd.grpc.v1.cri, and what it depends on, io.containerd.cri.v1.runtime·io.containerd.cri.v1.images (cmd/containerd/builtins/cri.go).
• Who spawns pause. In 1.7, RunPodSandbox created the pause container in a single function (sandbox_run.go:178), called CNI (line 338), and started the task (line 481). In 2.x, the CRI server handles only up to the network, and pause is delegated to the podsandbox called sandbox controller plugin(Section 4.2).
• Configuration version. Version 2.x uses configuration version 4 (version/version.go:41). Version 2 configs are also read but migrated and applied to versions 3 and 4 upon startup (config.go:48 of migrations). Version 4 moves server properties (such as[grpc]·[ttrpc] ) into the io.containerd.server.v1.* plugin configuration.

4. Following the Core Path

4.1 Daemons Are Assembled via a Plugin Graph

Almost all features in containerd are plugins. Each package registers itself in init() via registry.Register , and builtins.go simply pulls those packages in via blank imports (_). Whether they are included in the build is determined by a single import line.

1. Plugins are identified by typeand ID . The type is a string like io.containerd.grpc.v1, io.containerd.snapshotter.v1, io.containerd.cri.v1 (aroundplugins/types.go:35 ). In configuration files, [plugins."io.containerd.cri.v1.images"] literally uses "type.ID".
2. Registration involves Requires(a list of required types). The CRI gRPC plugin requires eleven types, including CRI service, sandbox controller, NRI, events, lease, transfer, and shim (plugins/cri/cri.go:46 init).
3. The daemon sorts them in dependency order in LoadPlugins via registry.Graph(...) (line 386), and initializes them one by one in a loop in New (line 161) via p.Init (line 189). Graph sorting itself resides in a separate module, github.com/containerd/plugin v1.1.0, so it is not in this repository.
4. Services are usually double-layered: a "local" implementation of the io.containerd.service.v1 type that performs the actual work (plugins/services/tasks/local.go:92), and a thin io.containerd.grpc.v1 wrapper that exposes it via gRPC (containers/service.go:34).

Why two layers? The CRI plugin is the answer. initCRIService creates a containerd client but passes containerd.WithInMemoryServices(ic)(line 121). External ctr and nerdctl call the gRPC wrapper over the socket, whereas CRI within the same process calls the local implementation directly. This design avoids an extra serialization step while keeping client code unified . The namespace used by CRI requests is also fixed to k8s.io at this time. Therefore, Kubernetes containers are visible only when viewed with ctr -n k8s.io .

4.2 RunPodSandbox: Network First, Pause Handled by the Controller

The first RPC in the CRI protocol is RunPodSandbox . The single line "The sandbox must be ready upon success" is the entirety of the contract, and the runtime decides what a sandbox actually is. containerd's answer is "Network namespace + pause container holding it."

1. criService.RunPodSandbox first creates a lease with the sandbox ID (line 97). While this lease is active, content and snapshots used by the sandbox are excluded from GC.
2. hostNetwork If not netns.NewNetNS , it creates a named network namespace under /var/run/netns/ using
3. Next, setupPodNetwork calls CNI's Setup (line 496). Network attachment happens before any container is spawned . The acquired IP is cached in the sandbox state so that the namespace does not need to be inspected on every PodSandboxStatus .
4. It then calls the sandbox service's CreateSandbox(line 281) and StartSandbox(line 299). Here, the actual work is performed by the sandbox controller.
5. The default controller podsandbox 's Start locates the pause image (line 86), prepares a snapshot over that image to create a container (line 209), creates a task without standard I/O (line 253), sets up exit monitoring (Wait, line 270), and starts it (line 295).

Why was it split this way? To avoid locking down a sandbox strictly as a "pause container." Since the controller is a plugin, implementations where the sandbox is a virtual machine (like VM-based runtimes) can be plugged in under the same CRI server. The change of the pause image name from a single line in settings to the sandbox_image entry in the pinned_images table (in configuration struct sandbox ) follows the same rationale. Comments define these as "images that CRI clients must not delete," allowing other plugins to look them up by key. Pause is merely one of them. config.go:3074.3 CreateContainer·StartContainer → tasks → shim → runc

converts CRI's

1. CreateContainer into an OCI runtime spec, prepares the writable layer on top of the image via ContainerConfig (line 378), and calls WithNewSnapshot(line 462). This phase creates client.NewContainermetadata only . No process exists yet. If the sandbox holds a task API address,
2. StartContainer attaches it as an option (line 197) and calls container.NewTask(line 206) → task.Start(line 247).
3. The client's NewTask calls the snapshotter's handleMounts(line 328). Mounts to include the rootfs mount list in the request. The tasks service's local.Create receives this and passes it to runtime v2's TaskManager.Create .
4. ShimManager.Start and binary.Start execute the shim binary as a start subcommand. The shim returns its ttrpc socket address and detaches from the daemon.
5. On the shim side, manager.Start contains a few lines that are the most important in this post.

grouping := id
spec, err := readSpec()
...
for _, group := range groupLabels {
	if groupID, ok := spec.Annotations[group]; ok {
		grouping = groupID
		break
	}
}
...
s, err := newShimSocket(ctx, socketDir, opts.GetContainerdGrpcAddress(), grouping, false)
if err != nil {
	if errdefs.IsAlreadyExists(err) {
		params.Address = s.addr
		return &params, nil

Source: containerd v2.3.5 cmd/containerd-shim-runc-v2/manager/manager_linux.go, Apache-2.0

groupLabels has io.kubernetes.cri.sandbox-id (line 67). Containers in the same pod get the same socket name with the same sandbox ID, and if the socket already exists, it returns the existing address instead of launching a new shim. Therefore, there is one shim per pod, not per container. The measurements in Section 6 demonstrate this exact behavior. 6. The shim's service.Create is runc.NewContainer in rootfs under the bundle, where the shim directly mounts overlay onto the shim directly mounts(line 120), and Init.Create executes runc create (line 149). Start is start in runc start .

Why do we have a separate shim? If containerd were the direct parent of the container process, restarting or upgrading containerd would disrupt all running containers. By having the shim act as the parent (collecting exit codes and maintaining standard I/O), the daemon can go down and come back up at any time to reattach via sockets. Keeping runc non-resident is for the same reason. The fewer long-lived processes there are, the easier upgrades become.

4.4 Image Pull and Snapshots: How Layers Become rootfs

1. CRI's PullImage is received as CRIImageService.PullImage . Since version 2.1, the default is the transfer service, and if use_local_image_pull is enabled, it uses the legacy client.Pull path (line 184). The transfer path is pullImageWithTransferService → Transfer(Line 326) → localTransferService.pull .
2. The downloaded manifests, configurations, and layers all go into the content-addressable storage. The path is based directly on the digest blobs/sha256/<hex> is(blobPath). Even if multiple images use the same layer, it exists only once on disk.
3. Unpacking is Unpacker.unpack done per layer by the snapshotter Prepare(line 415) → diff service Apply(extracting tar, line 542) → Commit(Line 494). The name of the committed snapshot is the chainID up to that layer, so identical layer chains are pulled only once even if the images differ.
4. When creating a container, WithNewSnapshot uses this topmost chainID as the parent Prepare to create an empty active snapshot (line 276). If the image has not been pulled yet, the CRI-side wrapper unpackImage pulls it and retries.
5. Finally, the overlayfs snapshotter's mounts generates the mount options.

if s.Kind == snapshots.KindActive {
	options = append(options,
		fmt.Sprintf("workdir=%s", o.workPath(s.ID)),
		fmt.Sprintf("upperdir=%s", o.upperPath(s.ID)),
	)
}
...
parentPaths := make([]string, len(s.ParentIDs))
for i := range s.ParentIDs {
	parentPaths[i] = o.upperPath(s.ParentIDs[i])
}
options = append(options, fmt.Sprintf("lowerdir=%s", strings.Join(parentPaths, ":")))

Source: containerd v2.3.5 plugins/snapshots/overlay/overlay.go, Apache-2.0

The fs directories of the parent snapshots become the lowerdir as they are, and its own fs·work becomes upperdir·workdir . Since all files used by the container are accumulated only in the upper layer, a hundred containers using the same image share the lower layers. The story of applying the same idea to VM disks is found in Until Student VMs Spin Up 4.

The ledger that ties all of this together is bolt DB. Under the namespace, there are image, container, snapshot, content, and lease buckets (buckets.go:60 comment diagram), and GarbageCollect deletes contents and snapshots that are not reached by any image, container, or lease. The reason layers are not deleted during a pull is that the operation holds a lease. It is designed around the principle of "delete if unreferenced," with ongoing operations being the only exceptions handled via leases.

5. Connection to User Contributions

As of October 17, 2026, the author has no PRs submitted to containerd or kubernetes/cri-api. The contribution ledger lists "Container runtime low-level (containerd, CRI-O, runc, etc.)" as an area under investigation, and this post is a preparatory exercise for that. The candidates in Section 8 were found directly from the source while writing this post.

One thing to note in advance: kubernetes/cri-api is a repository where bots mirror staging/src/k8s.io/cri-api from kubernetes/kubernetes, so PRs sent directly to this repository are ignored (repository CONTRIBUTING). CRI-related modifications must be sent to kubernetes/kubernetes, in which case Kubernetes' AI usage rules (notification in the PR body, prohibition of AI-written commit messages and co-author attribution) apply.

6. Measurement: Read-Only Inspection of 1.7 and 2.3 Workers without Privileges

On October 17, 2026, read-only checks were performed on a small cluster (8 nodes). The node runtime distribution is kubectl get nodes -o custom-columns=...containerRuntimeVersion , and inside the nodes, only ps·mount·/etc/containerd/config.toml was read using regular user ssh. /run/containerd·/var/lib/containerd could not read the list due to lack of privileges (sudo was not used), and crictl was not installed.

A shim line looks like this (ID is masked).

containerd-shim /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 0c1e…(64자) -address /run/containerd/containerd.sock
overlay on /run/containerd/io.containerd.runtime.v2.task/k8s.io/0c1e…/rootfs type overlay (rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs,upperdir=…/snapshots/3774/fs,workdir=…)

Read:

• Number of shims = Number of pods. Both editions were exactly the same. This means that the grouping in section 4.3 already existed in 1.7. sandbox-id This means that the grouping in section 4.3 was already present in 1.7. -id The value recorded in is the sandbox ID, not the container ID. ps The comm of containerd-shim is truncated to because Linux cuts process names at 15 characters.
• hostNetwork pods also have a pause container. Out of 22 pause containers, 10 belong to hostNetwork pods, and the number of network namespaces (12) matches the exact number of remaining pods. The branch "create a netns unless hostNetwork" from Section 4.2 is visible as numbers.
• Task directory 50 = 22 sandboxes + 28 containers . Pause is also counted as one container. Worker B leaves one unaccounted for with 31 + 42 = 73, but which container it belongs to could not be verified due to lack of list permissions (unverified).
• 22 overlay 1st layers = 22 pause containers. The pause image has only one layer, so it has only one lowerdir. The 33 on Worker B appear to be 31 pause containers and two single-layer images (estimated).
• Worker B's lowerdir snapshots/1 is the first snapshot extracted on this node, and the upper number 3774 means that many snapshots were created and deleted in between. Since the number is storage.CreateSnapshot assigned by this bolt bucket's NextSequence (line 252), deleted numbers are not reused.
• If you apply the 1.7 node configuration as-is to the 2.3.5 daemon, it performs the edition 2 → 4 migration on every startup and leaves a warning (Section 3). I haven't actually tried running it (unmeasured).

On the kubelet side, both editions were containerRuntimeEndpoint: unix:///run/containerd/containerd.sock(2.x nodes are denoted as /var/run/... ), cgroupDriver: systemd . This matches the runtime's SystemdCgroup = true .

7. Where to start reading

1. api.proto of RuntimeService·ImageService Just glance at the comments. You can see what the contract requires and what it delegates to the runtime.
2. plugins/cri/cri.go and server.go to see how plugins are tied together.
3. sandbox_run.go → podsandbox/sandbox_run.go to spin up a single pod.
4. container_start.go → shim_manager.go → manager_linux.go down to the process level.
5. unpacker.go and overlay.go to look at the disk side. docs/snapshotters/·docs/content-flow.md is a great complement.

You can continue reading about how distributions bundling containerd, such as k3s and RKE2, wrap this architecture in Comparing k3s, RKE2, and k0s architectures.

8. Good First Contributions

• The version number in the CRI configuration documentation is outdated. docs/cri/config.md The "Config versions" section states that version 3 is the latest and version 2 is "automatically converted to version 3," and the "Full configuration" example is also version = 3 . The RELEASES.md with the same tag states that version 4 (since v2.3.0) is the latest, and the ConfigVersion in the code is also 4. This can be aligned with a single documentation PR.
• Two TODOs. plugins/cri/cri.go In initCRIService , TODOs remain at the end: "Where is Closer called?" and "How to stop containerd." Simply tracking whether Close is actually called in the shutdown path and organizing it into an issue will be helpful.
• Temporary handling of sandbox platform identification. sandbox_run.go Around line 180 of internal/..., there is a comment stating, "The controller cannot communicate the platform early, so we infer it from the presence of the Linux section; this is a hack." While this requires controller interface discussions and is large for a first PR, it is a great entry point to read the issue context.
• The changelog per version in the cri-api READMEstops at v1.32 (based on the v0.34.0 tag; master is unverified). Contributions should be sent to the staging directory in kubernetes/kubernetes.
• Labels: containerd uses exp/beginner labels (2 opened on 2026-10-17). AI rules are in the "Automated and AI-generated contributions" section of CONTRIBUTING. PRs must be opened by a human, and all code, text, and comments generated by AI must be fully reviewed by the author.

9. Limitations

• The kubelet source code was not opened. The order in which kubelet calls the CRI (e.g., retries between PullImage and CreateContainer ) is out of scope for this post.
• NRI, CDI (device injection), user namespaces, checkpoints, streaming (exec, attach, port-forward) servers, and Windows paths were not covered.
• The difference between the transfer service and local pull, as well as the GC scheduler cycle and thresholds, were only examined up to function names and details were not read.
• Measurements were unprivileged reads, so content store size, bolt DB contents, and snapshot counts were not measured. There is also no record of actually performing a 1.7 to 2.3 upgrade (unmeasured).
• Comparison with 1.7 is merely a comparison of file organization and a few functions in the v1.7.27 source.

A word from Titi. kubelet only places the order, and a dedicated staff member (shim) is assigned to each table. So even if the chef (containerd) takes a short break, the customer's plate stays right there!

References

All materials were viewed on 2026-10-17.

• kubernetes/cri-api v0.34.0 pkg/apis/runtime/v1/api.proto, CONTRIBUTING.md
• containerd v2.3.5: plugins/cri/cri.go, cmd/containerd/server/server.go, internal/cri/server/sandbox_run.go, internal/cri/server/podsandbox/sandbox_run.go, core/runtime/v2/shim_manager.go, cmd/containerd-shim-runc-v2/manager/manager_linux.go, plugins/snapshots/overlay/overlay.go, core/unpack/unpacker.go, docs/cri/architecture.md, docs/containerd-2.0.md, RELEASES.md, CONTRIBUTING.md
• containerd v1.7.27: pkg/cri/cri.go, pkg/cri/server/sandbox_run.go
• Cited code: containerd v2.3.5, Apache License 2.0.