Status: Draft (2026-10-16), official documentation + partial measurements. Component descriptions were verified against the KubeVirt repository's component documentation and user guide on 2026-10-16. The version is KubeVirt v1.9.0. Values marked with 「Measured」 were obtained from TT Lab's practice VMs. Convention: Measured data is the default, unmeasured items are marked as 「Unmeasured」, and items not found in the documentation are marked as 「Unconfirmed」.

Until Student VMs Start 2: Inside KubeVirt

Table of Contents

• 1. VirtualMachine and VirtualMachineInstance
• 8. Four Components
• 3. Why VMs Are Placed Inside Pods
• 4. KVM: Hardware Acceleration
• 5. Commands Cannot Be Executed Inside the Guest
• 6. Real-World Incident: A Feature Gate Prevented VMs from Starting Entirely
• 7. Limitations
• References

Part 1opens item ⑥ 「VM → VMI → Pod → QEMU」.

1. VirtualMachine and VirtualMachineInstance

When you first encounter KubeVirt, the two similarly named objects can be confusing.

The TT Lab backend creates only VMs. By writing running: true in the manifest, we turn them on as soon as they are created. KubeVirt provides runStrategy(Always, RerunOnFailure, Once, Manual, Halted) to define this behavior more granularly, but the two cannot be used together. The code comments marked running as deprecated, but we could not find that sentence in the run strategy guide (unconfirmed). We haven't changed it yet to match the behavior of already running VMs. When checking the status, we read the VMI. If there is no VMI, we determine it is 「starting」, and if even the VM is missing, we determine it is 「deleted」.

2. Four Components

The roles explained in the KubeVirt repository's component documentation are summarized as follows:

• virt-api. The entry point for virtualization-related requests. It populates and validates default values for VMIs and serves subresources like console and vnc. This is where students' web terminals connect.
• virt-controller. A cluster-wide controller. It watches VMIs, creates the Pods to house those VMIs, and manages their lifecycles. Each VMI is paired with a single Pod for its lifetime, though migration may change the paired Pod.
• virt-handler. Runs once per node as a DaemonSet. It observes VMIs assigned to its node, aligns the libvirt domains with the VMI states, and reports domain states back to the cluster.
• virt-launcher. The main container of the Pod created for each VMI. It provides the cgroups and namespaces where the VM process runs, starts the VM via libvirtd inside the Pod, monitors the VM process, and terminates itself when the VM finishes. If Kubernetes attempts to stop the Pod first, it forwards the signal to the VM and delays Pod termination until the VM shuts down.

The creation sequence is outlined in the documentation in seven steps. Once a VMI is created, the virt-controller creates a Pod, the scheduler assigns it to a node, and once the Pod starts, the virt-controller writes the node name into the VMI. From that point on, the virt-handler on that node takes over and instructs libvirtd inside the Pod to create the domain. The measurements from Part 1 (Pending 0s → Scheduled 10s → Running 20s) captured this interval.

3. Why VMs Are Placed Inside Pods

At first, this seems strange. VM isolation is heavier than container isolation, so why wrap it inside a container again? The answer is to avoid reinventing things Kubernetes already does well. The benefits actually experienced in TT Lab include:

1. Scheduling and Resource Calculation. A VM's CPU and memory translate into Pod resource requests, allowing the scheduler to calculate them directly. Measuring the Pod request for a single 2 vCPU / 4Gi VM showed 200m CPU and 4372Mi memory (measured, 2026-08-29). Requesting only 0.1 CPU for a vCPU is due to KubeVirt's cpuAllocationRatio default value of 10 (User Guide 「Resource requests and limits」), and the memory included about 280Mi of overhead on top of the 4Gi guest allocation.
2. Networking and Policies. Guests use the Pod network directly (masquerade). Therefore, network policies applied to practice Pods are applied to VMs in the exact same way. The target selected by the policy is not the VM, but the label of the virt-launcher Pod, because the actual network endpoint is that Pod (Part 6).
3. Volumes. Guest disks are Pod volumes. Kubernetes handles attaching PVCs, creating node-local ephemeral directories, and injecting Secrets (Parts 3 and 4).
4. Lifespan. When a Pod is deleted, the VM also terminates, and the ephemeral volumes attached to the Pod disappear together, making reclamation easier (Part 5).

There are also trade-offs:

• An extra Pod is always attached. The aforementioned 280Mi overhead and a small container collecting console logs (measured CPU 5m, memory 35M) are required per VM.
• virt-launcher requires privileges. Consequently, the namespace where the VM lives is set to the most relaxed Pod Security Standard: privileged . This privilege belongs to KubeVirt, not the student. However, since permission to create Pods in that namespace becomes a gateway to node root access, Pod creation permissions were removed from the backend (Part 6).

4. KVM: Hardware Acceleration

Guests are run by QEMU using hardware acceleration via KVM (/dev/kvm). KubeVirt exposes device plugin resources (devices.kubevirt.io/kvm) on nodes that support KVM, and virt-launcher Pods request those resources. Therefore, VMs will not be scheduled on nodes lacking KVM support.

Labs requiring nested virtualization inside the guest (running VMs inside a VM) switch the CPU model to host-passthrough . Since the default CPU model hides hardware virtualization flags, leaving it as-is causes QEMU inside the guest to run via software emulation, resulting in multi-minute boot times. Enabling host-passthrough exposed /dev/kvm inside the guest and made lscpu visible in (measured, 2026-08-29). The trade-off ismigration

; VMs cannot be moved if node CPU models differ. Since practice VMs are not migrated anyway, we accept this, but do not enable it for labs that do not require it.

5. Commands Cannot Be Executed Inside the Guest exec While Pods have allowing the backend to run grading commands inside them, KubeVirt's VMI subresources include console, vnc, portforward, and guestosinfo, butlack exec

(verified via measured subresource lists). This is because the guest is a separate operating system, and running commands inside it requires something running within the guest itself.

To address this, TT Lab embedded a small grading agent inside the guest (Part 1 ③). Student shells are opened via the serial console (console subresource). Since there is no need to open SSH, one connection path is eliminated. In fact, prototyping showed that grading succeeded even with SSH completely disabled.

6. Real-World Incident: A Feature Gate Prevented VMs from Starting Entirely

exec: "/container-disk-binary/usr/bin/container-disk": no such file or directory

When KubeVirt v1.9.0 was first deployed, not a single VM would start. virt-launcher Pods failed with this error: ImageVolume Suspecting broken container runtime settings, we found the root cause elsewhere. As KubeVirt 1.9 enabled beta feature gates by default, turned on automatically. This feature attaches container disks as Pod image volumes, but Kubernetes 1.34 had that gate disabled, causing the API server to silently drop image volumes and their mounts. Because Pods are accepted without rejection, no warnings appear anywhere. We only confirmed this after noticing that the image volume was missing from the spec of the running Pod. The container runtime was a secondary hurdle; since image volumes are only supported starting from containerd 2.1, enabling the Kubernetes gate alone would have merely turned the silent deletion into a mount failure.

We reverted to the old method (empty directories and copy init containers) by disabling the KubeVirt-side gate. Because this setting goes into KubeVirt's custom resources, it must be re-verified every time KubeVirt is upgraded . Even on the day of writing, we verified by reading that ImageVolume remained present in the disabled gates list.

The lesson is this: 「VMs inside Pods」 stand at the intersection of three versions—Kubernetes, KubeVirt, and the container runtime. Upgrading just one can silently break promises with the other two.

7. Limitations

• The resource requests above were measured once for a single VM size (2 vCPU / 4Gi). Overhead tables by size are unmeasured}.
• A method to report VM ephemeral storage (node-local disk) usage to the scheduler has not yet been verified (Part 4, Limitations section).
• Live migration is not used. Practice VMs live for about an hour and are then deleted, so when nodes need to be cleared, sessions are ended rather than migrated.

Titi's Note. Putting a VM inside a Pod initially feels like 「a larger box inside a box」, but thanks to that, Kubernetes handles scheduling, network policies, and volumes for us. Just remember that the three versions must work together in sync. Upgrading just one might cause your VMs to silently fail to start!

References

All documentation was accessed on 2026-10-16.

• KubeVirt Component Documentation: https://github.com/kubevirt/kubevirt/blob/main/docs/components.md
• KubeVirt User Guide, Resource Requests and Limits(cpuAllocationRatio): https://kubevirt.io/user-guide/compute/resources_requests_and_limits/
• KubeVirt User Guide, Disks and Volumes: https://kubevirt.io/user-guide/storage/disks_and_volumes/
• KubeVirt User Guide, Run Strategy: https://kubevirt.io/user-guide/compute/run_strategies/
• TT Lab code and history: backend/app/vms.py(SIZES Resource sizing in comments, vm_manifest of nested), backend/app/main.py(vm_console·vm_info), deploy/vms/README.md(ImageVolume gate), docs/plans/2026-08-29-vm-labs-design.md