Status: Draft (2026-10-16), our code + official docs + partial measurements. Operational descriptions strictly follow the TT Lab backend code (backend/app/vms.py, backend/app/main.py) and design documents. Versions are values read and verified on the day of writing (Kubernetes v1.34.10, KubeVirt v1.9.0, CDI v1.66.0). Time values are records measured while building the labs, with sample sizes and dates noted together. Writing rule: Measured data is the default; unmeasured items are marked as "Unmeasured", and items not found in documentation are marked as "Unconfirmed".

Until a Student VM Starts 1: The Big Picture Across Four Layers

Table of Contents

• 1. Why VMs Instead of Pods
• 2. The Four Layers
• 3. Ten Steps
  • ① "Start Lab"
  • ② Checking Capacity
  • ③ cloud-init Secret
  • ④ Choosing a Root Disk
  • ⑤ Creating a VirtualMachine
  • ⑥ KubeVirt: VM → VMI → Pod → QEMU
  • ⑦ Attaching Volumes
  • ⑧ Verifying Readiness
  • ⑨ Console Proxy and ⑩ Web Terminal
• 4. How Long It Takes — Current Measurements
• 5. What This Picture Deliberately Omits
• References

1. Why VMs Instead of Pods

Most TT Lab labs run on a single Kubernetes pod. Pods start in seconds, changing environments is as simple as swapping images, and they leave no traces when deleted. However, there were some labs that pods could never replicate.

• There is no systemd. Labs for restarting services or modifying unit files had to be reduced to "syntax checking".
• Setting up nodes with kubeadm and backing up and restoring etcdare impossible. Inside a pod, Kubernetes was simulated using kwok, but kwok does not actually run containers and instead fakes pods as Ready.
• Kernel features cannot be touched. Lab pods have 0 capabilities, so even rootless podman does not run. Moreover, those 0 capabilities are a principle maintained by TT Lab, with no plans to increase them.

Therefore, only these kinds of labs are opened as virtual machines. Students have true root access inside the VM and can freely handle systemd, the kernel, and disks. Instead, preventing that root access from escaping the VM and ensuring that used VMs are reliably reclaimed are the main themes of this series. VMs are launched on Kubernetes using KubeVirt because they use the same cluster, scheduler, and storage as lab pods.

2. The Four Layers

This series deliberately focuses on only four layers: Student (Browser), Backend (Web Pod), KubeVirt, and Kubernetes Storage. Intermediaries sitting between them, such as gateways, image registries, and monitoring, are omitted from the diagram because these four form the backbone of the process "a VM is allocated".

The role of each layer can be summarized in one line:

3. Ten Steps

① "Start Lab"

When a student clicks the button, the browser sends POST /api/sessions . If the lesson is runtime: vm , this session creates a VM instead of a pod. The session record includes runtime from the very beginning. This is because the reaper determines "what to delete" later using this single value. If the type is misidentified here, the pod is recorded as deleted while the VM continues to hold onto 4GiB of memory.

② Checking Capacity

Two things are checked. One is one session per person. If another lab is already open, it notifies the user of what is open and does not create a new one. Since checks and records are performed within per-user locks, clicking the button twice still results in only a single session.

The other is **the number of concurrently running VMs (default: 8)**. This number is counted by querying the cluster , not the database. The database cannot count ghost VMs missed by the reaper, whereas it is the cluster side that is actually holding the node's memory. When the limit is hit, it returns "There are currently 8 VM labs open, so no capacity is available." Why the limit is 8 will be covered in Part 5.

③ cloud-init Secret

Rather than pre-baking VM images, we use a public cloud image and inject what we need via cloud-init on first boot. We inject three things:

• Grading agent. A lightweight HTTP server running inside the guest on port 8899. It takes received commands and runs them in a shell for grading. Why this channel is needed is covered in ⑧.
• A unique token per VM. The agent only accepts requests with the correct token. Students have root access to their own VMs and can see the token. What we are trying to block is not the student, but another pod that might accidentally reach it if the network policy is misconfigured (Part 6).
• Lab files and setup scripts. The grader and lab materials. If the setup script fails, the "ready" status marker is not written.

There is a practical reason for putting these in a Secret instead of writing them directly into the VM manifest. KubeVirt limits inline cloud-init to 2048 bytes, but including the grading agent pushes it past 7,000 bytes. Therefore, all practice VMs go through the same path (Secret). If the path diverged based on size, it would introduce a failure path that only triggers when adding the agent.

④ Choosing the Root Disk

This is the core of the series. When a lesson specifies vm_disk: "golden" , the backend reads the operator-prebuilt golden PVC (get ) to determine if it can be used right now. If it is usable, that PVC is used as a read-only base, and a thin writable layeris stacked on top for each VM. If it is missing or unusable (not found, unauthorized, in the process of being fetched, single-node volume, different image, or being deleted), it opens the default disk (container disk) without throwing an error. This is because the golden disk is an optimization, not a prerequisite. The three disk branches are covered in detail in Part 3, and the golden and writable layers in Part 4.

⑤ Creating the VirtualMachine

The backend creates a VirtualMachine object in the Kubernetes API. This object is labeled with a practice session indicator, the session ID, and which environment's backend created it (dev vs. prod). The last label stems from an incident. The development and production backends were watching the same VM namespace, and one side's garbage collector treated VMs not found in its own DB as orphans and deleted another student's VM (Part 5).

The cloud-init Secret lists this VM as its owner so that when the VM is deleted, the Secret is deleted along with it. However, since the owner reference link can sometimes fail, the Secret is also explicitly deleted one more time during cleanup.

⑥ KubeVirt: VM → VMI → Pod → QEMU

From here, KubeVirt takes over. It looks at the VM, creates a running VirtualMachineInstance (VMI), and creates one virt-launcher pod per VMI. When the Kubernetes scheduler places that pod onto a node, the virt-handler on that node instructs libvirt inside the pod to "turn on this VM." The guest runs as a QEMU process inside that pod. Why VMs run inside pods and the trade-offs involved are the topics of Part 2.

⑦ Attaching Volumes

When the virt-launcher pod is created, the volumes that will serve as the guest's disks are attached as well. If it is a golden practice session, the golden PVC is read-only, and KubeVirt creates a write-layer file on top of it in the pod's temporary directory (emptyDir). The guest only sees a single combined disk (/dev/vda). If there is no PVC to attach, KubeVirt cannot create the pod at all and the VM will never start. This is why step 4 performs a pre-check.

⑧ Readiness check

The backend waits in two stages. First, it waits for the VMI to become Running (with a limit defined per lab, defaulting to 150 seconds), and then it waits for the agent inside the guest to respond as "Ready".

KubeVirt does not have a feature like pod exec for "running commands inside the guest." The subresources are limited to things like console, vnc, and portforward, which is by design. Therefore, both the readiness check and grading are handled by the agent embedded in step ③.

Running We also learned from experience that we must not let students in based solely on . The agent starts near the beginning of cloud-init, and the preparation script that sets up the lab environment runs after it. Letting students in during that gap caused failures starting from stage 1 with "file not found," and grading only passed 4 out of 8 stages. Now, the last line of cloud-initwrites a readiness marker, and if the preparation script fails, it is not written. As a result, the backend times out and fails honestly .

⑨ Console relay and ⑩ Web terminal

The backend connects via WebSocket to the VMI's console subresource (⑨) and relays data between it and the student's browser web terminal (⑩). There were two traps here as well.

• The Kubernetes client stream function used for pod terminals reads the first byte as a channel number. The KubeVirt console sends raw bytes without such a wrapper (plain.kubevirt.io) so if you connect with the same function, the connection opens, but nothing appears on the screen. No errors occurred, which made it even more confusing. So, we connect directly via WebSocket.
• What is at the end of the console is not a shell, but a login prompt (getty). Since the practice VM purposely does not have a password set, cloud-init configures automatic root login on the serial console.

When opening a WebSocket, the backend first checks if the connected user is the owner of that session and whether the session is running. The only Kubernetes identity that can reach the console subresource is the backend service account (Part 6).

4. How long does it take — Measurements so far

The table tells us two things. The VM itself powers on in around 20 seconds, and the rest is the time spent setting up the lab inside the guest. Also, if you fetch a new disk for each VM, that alone takes several minutes. The golden disk in Part 4 targets this very last line.

The limitations are also clear. Most values were measured just once, and the distribution of preparation times across all student sessions (median and variance) is unmeasured. Since session logs retain both the start time and preparation time, they can be calculated simply by gathering the logs.

5. What this illustration intentionally omitted

• Network policies. Incoming connections to student VMs are blocked by default, and only the backend is reachable via 8899. Outbound traffic is restricted to the internet only. Covered in Part 6.
• Reclamation. When the session time expires, the reclaimer deletes the VM, and the write layer disappears along with the pod. Covered in Part 5.
• Multi-VM sessions. Sessions requiring a set of nodes, such as the kubeadm lab, repeat the same steps (② to ⑧) for each VM and provision an additional session-dedicated network. This series is written based on single-node setups.

A word from Titi. Ten steps are hidden behind a single button click. However, the VM booting finishes in 20 seconds, and almost everything else consists of 「preparing the lab inside the guest」 and 「provisioning the disk」. Therefore, if you want to speed things up, look at the disk first. See you in Part 4!

References

All documents were accessed on 2026-10-16.

• KubeVirt Components: https://github.com/kubevirt/kubevirt/blob/main/docs/components.md
• KubeVirt Disks and Volumes: https://kubevirt.io/user-guide/storage/disks_and_volumes/
• TT Lab Code: backend/app/vms.py(Manifests, cloud-init, golden judgment), backend/app/main.py(create_session·make_vm·vm_console·reap_once), backend/app/vm_agent.py(Grading Agent)
• Design and Measurement Records: docs/plans/2026-08-29-vm-labs-design.md, docs/plans/2026-09-12-vm-reaper-environment.md, docs/plans/2026-09-24-multi-vm-sessions.md, docs/plans/2026-10-14-vm-golden-overlay.md