Status: Draft (2026-10-16), our code + official docs & source + toy model measurements. Golden VM measurements not yet available. The golden disk includes code and operator tools, but on the day this post was written, neither the golden PVC nor the permission to read it had been applied to the cluster yet (verified via read-only inspection). Therefore, the golden VM behavior in this post is verified through KubeVirt v1.9.0 and CDI v1.66.0 documentation and source code, and the numbers were measured from a toy model and container disks (same principle) run on a laptop. Writing rules: measured data is the default, unmeasured items are marked as "Unmeasured", and items not found in the documentation are marked as "Unverified".

Student VM Launch Part 4: Overlay from a PVC Perspective

Table of Contents

• 1. What We Want to Achieve
• 2. Principle: Base and Write Layer
  • Measuring with a Toy Model
• 3. KubeVirt's Ephemeral Volume: Doing This on Top of a PVC
• 4. From the Perspective of a PVC
  • Verdict and Alternatives
• 5. Why Not CSI Cloning or Snapshots
• 6. When the Session Ends: Only the Write Layer Disappears
• 7. Cost and Limitations
  • What to Measure and How to Measure It
• References

Part 3ended with the void between two approaches. Container disks boot in 20 seconds but have a 2.4GiB root, while DataVolumes have a large root but fetch a new image for every VM, pushing the first boot up to 195 seconds. This post explores the third path bridging that gap: Golden PVC + Write Layer.

1. What We Want to Achieve

• Give each student a large root(10Gi or 20Gi) so that root-heavy labs like kubeadm do not fail due to insufficient space.
• Fast startup. Do not fetch or copy several gigabytes every time a VM is created.
• Small footprint. Do not create thirty copies of 10Gi for thirty students.
• No traces left after the session ends. The next student receives a clean disk.

There is an old way to achieve all four at once: placing a thin layer containing only the changed parts per person on top of a single read-only base. Container image layers, virtualization linked clones, and qcow2 backing files all share the same idea.

2. Principle: Base and Write Layer

A qcow2 disk image can point to a backing file. When created with qemu-img create -f qcow2 -b <바탕> -F raw <쓰기 층> , the new image records only the differences from the base (qemu-img documentation). Here, the disk is divided into fixed-size blocks (clusters), and the rules are two lines:

• Reads. If the block exists in the write layer, read it from the write layer; otherwise, read it from the base (qcow2 format specification: unallocated clusters are read from the backing file).
• Writes. The first time a block is touched, it is pulled from the base, newly allocated in the write layer, and modified. The base is never modified.

Thus, the write layer starts empty and grows only by the blocks written by the guest. Because the base is not modified, multiple disks can point to the same base without interfering with each other.

Measuring with a Toy Model

Since qemu-img was not available on the laptop (we didn't install it), we simulated the two rules above in Python and measured them under the same conditions. The base is 512MiB (random bytes), and the guest performs 1,000 random 4KiB writes. These are the medians of five runs each for "Copy entirely then write" and "Write to an empty write layer" (run 0 was excluded as warmup). The total disk hashes for both methods matched across all five runs, confirming that the disk seen by the guest is identical.

Environment: Apple M1 Max laptop, internal SSD, Python 3.12, standard library only. Code and raw data are available in docs/experiments/2026-10-16-cow-overlay-toy/ .

There are three takeaways from the numbers.

1. Creation is practically free. An empty write layer takes 0.2 milliseconds regardless of disk size. Copying is proportional to disk size. The 0.2-second copy time here occurred because it was within the same SSD and the base was in the file cache, so the time to copy 10Gi from storage across the network cannot be represented by this figure (unmeasured). A simple calculation shows that 10GiB over a 1Gbps line theoretically takes about 86 seconds.
2. The write layer grows by the touched blocks
3. , not the written amount. Although only 3.9MiB was actually written, 945 blocks of 64KiB were touched, resulting in 59MiB (about 15x). Scattered small writes get bloated to the block size. If blocks are large and writes are scattered, the layer grows as large as the base. With 2MiB blocks, a 512MiB disk has only 256 blocks, so 1,000 writes touched almost every block. Maintaining a thin write layer relies on the assumption thatthe guest modifies only a small part of the disk

. This assumption breaks down during labs that involve upgrading the entire OS. What about in an actual VM? Inspecting a container disk VM with the same structure (for operators, running for 9 days), the write layer on top of the base (approx. 596MiB) was about 1.10GiB (measured, 2026-10-14). Since this VM had packages installed and logs accumulated over 9 days, it represents much heavier usage than a one-hour lab session. The write layer sizes immediately after boot and after the lab for a golden VM are unmeasured.

3. KubeVirt's Ephemeral Volume: Doing This on Top of a PVC

KubeVirt already has a volume source that provides this structure on top of a PVC. It is the ephemeral volume in the user guide. In summary:

• A network volume (PVC) is used as a read-only baseand the base image remains unchanged.
• All writes go to the local write layer image.
• The write layer is created when the VM starts and discarded when the VM stops or terminates.
• Currently, only PVCs are supported as the base.

The root volume of the golden VM created by the TT Lab backend is just this single line (backend/app/vms.py of vm_manifest).

volumes:
  - name: root
    ephemeral:
      persistentVolumeClaim:
        claimName: golden-ubuntu-24.04-10gi   # OS 와 크기로 정해진 이름
        readOnly: true                        # 빼지 않는다

Tracing the KubeVirt v1.9.0 source code, this single line performs the following:

1. When virt-controller creates the virt-launcher pod, it includes this PVC as a pod volume as is(readOnly , and attaches an additional empty directory (emptyDir) for the write layer (pkg/virt-controller/services/rendervolumes.go).
2. Inside the pod, the PVC is mounted at /var/run/kubevirt-private/vmi-disks/root/ , and the base within it is disk.img(raw).
3. At startup, qemu-img create -f qcow2 -b <바탕 disk.img> -F raw <쓰기 층 disk.qcow2> creates the write layer (pkg/ephemeral-disk/ephemeral-disk.go). The virtual size of the write layer is the same as the base. Therefore, the size of the base becomes the size of the guest root.

It is interesting that container disks follow the same principle. The only difference is whether the base is a node cache pulled from a registry or a PVC provisioned by a StorageClass (Figure in Part 3).

4. From the perspective of PVCs

Summarized as "who holds which PVC, and what is created and destroyed when," it looks like this:

There are reasons behind certain decisions.

• Created as RWX. Student VMs are scattered across multiple nodes and must be attached to the same golden image simultaneously . Because RWO attaches to only one node, the backend judges that the RWO golden image "cannot be written to." It is RWX rather than ROX because CDI needs to write to it while importing.
• The real protection is readOnly: true . As seen in Part 3, the access mode does not prevent writing. If the base changes even once, all write layers on top of it will point to the wrong disk. Therefore, the read-only mount is kept, and exams check that no golden images are attached as read-write in the manifest.
• It must be an original that has never been booted. If a disk that has been booted once is used as a golden image, SSH host keys, machine-id, and cloud-init logs are duplicated identically across all student VMs. Therefore, the golden image imports the same imageas the container disk as-is, and the original image address is left as an annotation on the PVC so the backend can verify it.
• Bind immediately. If we followed "postpone until the first user" as seen in Part 3, the import would start when the first student VM arrives. Since the golden image must be fully imported before that, it requests immediate binding.
• The backend only reads. The only thing the backend can do with the golden image is to get the PVC whose name it knows. Since it has no creation or deletion permissions, there is no way for a single mistake in the cleanup code to wipe out everyone's base (Part 6).

Evaluation and Fallback

Before creating a VM, the backend reads the golden image to evaluate whether it can be used right now. This is because a VM pointing to a non-existent PVC will never start as KubeVirt fails to create the Pod (v1.9.0 rendervolumes.go 's missing PVC error).

When fallback occurs, the student terminal displays "root is about 2.4GiB, smaller than usual, df -h / is displayed. The prompt is printed only in the interactive shell. Since the grading agent also runs commands via a login shell, otherwise the prompt would get mixed into the grading output. The reason it falls back to a container disk instead of a DataVolume is to avoid granting the backend permissions to create PVCs again, and because the first boot taking 195 seconds would exceed the preparation time limit of the golden lab, resulting in a failure instead. Therefore, golden is an optimization, not a prerequisite. Labs with golden enabled must also pass on a container disk.

5. Why Not CSI Cloning or Snapshots

Kubernetes already has two official ways to scale up volumes.

• CSI volume cloning. If you specify the source PVC in the dataSource of a new PVC, a clone of the source is created instead of an empty volume. The clone is an independent volumefrom the source. It requires a CSI driver, must be in the same namespace, must have the same volume mode as the source, and must be equal to or greater in size. Whether a driver implements this feature varies by driver (Kubernetes documentation).
• Volume snapshots. You take a snapshot of a volume at a certain point in time and create a new PVC from it. This requires the CSI driver's snapshot feature, a snapshot controller and CRDs, and a VolumeSnapshotClass (Kubernetes documentation).

CDI automatically chooses between these two when cloning a PVC via a DataVolume. If snapshots are available, it uses "snapshot → restore → delete snapshot" (smart clone); if CSI cloning is available, it uses that; if neither is available, it falls back to host copy, which streams data from the source to the target(CDI cloning and StorageProfile documentation). This choice is recorded in the StorageProfile of the storage class as cloneStrategy .

Checking TT Lab's shared storage class on the day of writing, I found that the snapshot CRD was installed, but there was not a single VolumeSnapshotClass, and the cloning strategy in the StorageProfile was copy(host copy) (checked on 2026-10-16). Therefore, if 10Gi per student were created using PVC cloning in this storage backend, each VM would copy a full 10Gi over the network. The snapshot route is completely unavailable.

Even if the storage changes, the writing layer retains its advantages.

The last row is the price paid by this approach. I accepted it because a lab session runs once and then ends. This approach is not suitable for assignments where students need to continue using the same disk over several days.

6. When the Session Ends: Only the Writing Layer Disappears

The golden VM manifest does not contain dataVolumeTemplates . Since the VM does not own the golden, deleting the VM leaves the golden intact. The garbage collector only deletes the VM and the cloud-init Secret (Part 5), and when the VM is deleted, the virt-launcher Pod terminates along with its emptyDir—that is, the writing layer. Nothing is passed on to the next student.

When the golden itself needs to be changed (updating the image), it is deleted and recreated with the same name. If there are VMs using it, Kubernetes' "Storage Object in Use Protection" keeps the PVCs until those VMs finish, during which time the backend sees "Terminating" and opens new VMs with container disks. There is also a method of putting a version in the name to create a new golden alongside the old one (the way CDI's DataImportCron works), but that requires the backend to resolve names with an extra permission, and automatically changes the student environment to an untested image. With a scale of six OSes, we decided that manual replacement was cheaper.

7. Costs and Limitations

• The writing layer has no size limit. Since there is no size restriction on the emptyDir where the writing layer resides, the upper limit is simply the virtual size of the base. If a student fills up the root, it consumes that much node-local disk space. When node disk pressure occurs, kubelet evicts pods that consume the most space, which happens to be that student's VM . Therefore, we bundled sizes into 10Gi and 20Gi menus. We have not yet figured out how to inform the scheduler of the VM's ephemeral storage requests and limits.
• Every read of an unwritten block goes to the shared storage. The speed when multiple VMs read the same base simultaneously varies by storage and is unmeasured.
• We haven't run tests yet to see if the golden VM actually boots. In the path where writes are made based on a read-only attached filesystem PVC, if KubeVirt tries to write to the base to adjust disk file ownership or fix sizes, it may fail. The first round of measurements will verify this.
• Guest root sizecalculations are inconsistent. As seen in Part 3, according to CDI documentation, a 10Gi golden guest disk should be 10Gi, while the design document expected about 9.4GiB. This is unmeasured.

What to Measure and How

The measurement script (scripts/measure_vm_disks.py) creates, measures, and deletes container disks, DataVolumes, and goldens one by one on the same node. It checks node status and free memory before starting and in each round, records round 0 separately as a warmup, and calculates the median, minimum, and maximum starting from round 1. It measures preparation time (from request to agent readiness), time until VMI Running, guest root size and free space, actual size of the writing layer (immediately after boot and after writing 128MiB), and deletion time. Since this involves creating and deleting VMs, it is run by someone with cluster write permissions.

python3 deploy/vms/golden.py manifest --os ubuntu-24.04 --storage-class <공유 스토리지 클래스>   # 골든 하나(적용은 사람)
python3 scripts/measure_vm_disks.py --preflight --node <노드>                                 # 아무것도 만들지 않고 본다
python3 scripts/measure_vm_disks.py --node <노드> --rounds 4                                  # 예열 1 + 3바퀴

Whether to enable golden for labs will be decided based on these results. If the median preparation time is within 1.5 times that of the container disk, the root is 9GiB or more, and the writing layer immediately after boot is within a few hundred MB, we will enable it for labs like kubeadm.

A Word from TT. We distribute 10Gi to thirty people, but what we actually create is a single 10Gi and thirty empty files. Instead, remember that "never touching the base" is vital, so make sure to attach readOnly: true , and also keep in mind that writes expand block by block!

References

All documents and sources were accessed on 2026-10-16.

• KubeVirt User Guide, Disks and Volumes (ephemeral): https://kubevirt.io/user-guide/storage/disks_and_volumes/
• KubeVirt v1.9.0 Source: pkg/ephemeral-disk/ephemeral-disk.go(Base path vmi-disks/<볼륨>/disk.img, qemu-img create -f qcow2 -b … -F raw) https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/ephemeral-disk/ephemeral-disk.go, pkg/virt-controller/services/rendervolumes.go https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/rendervolumes.go
• QEMU Documentation, qemu-img (backing file): https://www.qemu.org/docs/master/tools/qemu-img.html, qcow2 Format Specification (reading unallocated clusters): https://www.qemu.org/docs/master/interop/qcow2.html
• Kubernetes Documentation, CSI Volume Cloning: https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/, Volume Snapshots: https://kubernetes.io/docs/concepts/storage/volume-snapshots/, Persistent Volumes (in-use protection): https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• CDI Documentation, DataVolume Cloning: https://github.com/kubevirt/containerized-data-importer/blob/main/doc/clone-datavolume.md, smart clone: https://github.com/kubevirt/containerized-data-importer/blob/main/doc/smart-clone.md, StorageProfile(cloneStrategy): https://github.com/kubevirt/containerized-data-importer/blob/v1.66.0/doc/storageprofile.md
• TT Lab Code and History: backend/app/vms.py(golden_unusable_reason·choose_root_disk·vm_manifest), deploy/vms/golden.py, deploy/vms/README.md(Golden Disk Section), scripts/measure_vm_disks.py, docs/plans/2026-10-14-vm-golden-overlay.md
• Toy Measurement: docs/experiments/2026-10-16-cow-overlay-toy/(cow_toy.py, output.jsonl)