Status: Draft (2026-10-16), our code + incident log + partial measurements. The operation followed the TT Lab backend's reap_once·delete_vm_session and design/incident logs. Quotas and limits were queried on the same day as read-only to verify they match the stored values. Writing rules: Measured data is the default, unmeasured items are marked as [Unmeasured], and items not found in documentation are marked as [Unconfirmed].

Until Student VMs Spin Up 5: Lifespan and Garbage Collection

Table of Contents

• 1. Lifespan: 60 minutes, extensions up to three times
• 2. Garbage Collector: Three directions every 60 seconds
  • Incident 1: A garbage collector from another environment deleted student VMs
  • Incident 2: A single row failure halted the entire garbage collection
• 3. What to delete and in what order
  • Why is the Secret deleted separately one more time?
• 4. Upper limit: Bounding the magnitude of damage
• 5. Remaining gaps
• References

TT Lab's single-sentence principle is "Give students a real root shell while preventing them from escaping, and unfailingly reclaim used resources." The last phrase weighs particularly heavy for VMs. If a practice pod is left behind, a few hundred MBs leak, but if a single VM is left behind, 4 GiB is tied up entirely.

1. Lifespan: 60 minutes, extensions up to three times

• When a session is opened, the expiration time is set to start + 60 minutes(default value, can be changed via environment variables).
• Students can click extend. Extending pushes the expiration to "now + 60 minutes", but it cannot exceed start + 180 minutes. This is the upper limit preventing infinite extensions.
• The remaining time is calculated using the same functionby both the on-screen countdown and the garbage collector. If the two calculate it separately, situations might occur where the screen says 5 minutes remain but the VM suddenly disappears.
• Since it is one session per person, a student cannot stack up multiple VMs either (Part 1 ②).

2. Garbage Collector: Three directions every 60 seconds

The backend runs a garbage collection cycle every 60 seconds. The "Collect Now" button on the administrator screen also uses this same function. It looks in three directions.

There is also a reason for scanning VM orphans separately. The reclaimer initially only looked at the list of lab pods, but the VMs were in another namespace, so they would never show up on that list. A VM whose DB row had disappeared could end up holding 4 GiB of memory while remaining completely invisible to anyone.

Right before deleting orphans, the DB is checked one more time. This is to prevent the reclaimer from mistaking a VM from a session created just after it read the list as an orphan.

Incident 1: A reclaimer in another environment deleted student VMs

2026-09-12, a VM session opened via the student API in the production environment failed to become ready. Upon checking the logs, the VM was deleted 26 seconds after the VMI started. It was not deleted by the test tool.

The cause was sharing. The development and production environments have separate backends, but they share a single VM namespace. At the time, the garbage collector read the entire VM namespace and judged any VM "not in my DB to be an orphan." Therefore, to the garbage collector in the development environment, the production student's VM appeared to be an orphan. Under identical conditions in a test, the call to delete the production VM was reproduced (we did not extract the actual identity of the requester from the Kubernetes audit logs).

The fix was a single label. When creating VMs and Secrets, the backend attaches its own environment name as a label, and orphan collection targets only VMs whose label matches its own environment. It does not guess the owner of old VMs without a label. Since the list itself still views the entirety, it does not mistake old VMs registered in the DB as "non-existent resources." After the fix, the session reopened via the student API became ready in 90.991 seconds, and we confirmed that the VM, VMI, and Secret all disappeared after completion (1 occurrence).

Incident 2: A Single Row's Failure Halted the Entire Collection

This issue was discovered during a site security audit on 2026-10-14. Within the loop that deletes expired sessions, if a single deletion threw an exception, the entire iteration terminated. While the same row failed at the same point every minute, other expired sessions, orphans, and ghosts behind it accumulated without being collected by anyone.

Now, failures are contained per row.

for r in rows:
    try:
        delete_session_resource(r)
    except Exception as exc:
        failed += 1            # 실패를 세고
        print(f"[reaper] 만료 세션 회수 실패 {str(r['id'])[:8]}: {type(exc).__name__}", flush=True)
        continue               # 기록은 그대로 두어 다음 바퀴에 다시 본다
    con.execute("update sessions set status='expired', finished_at=now() where id=%s", (r["id"],))

Changing the record expired to occurs after a successful deletion. If the order is reversed, a VM that failed to be deleted would be recorded as a "completed session" and never become a collection target again. The failure count is also retained in the collection status displayed on the administrator screen.

In the same audit, a deadline ( pods= 60 mins × 3 + 15 mins) was also added toactiveDeadlineSeconds prevent expired root shells from living forever, even if the web and DB pods stop together and the reaper fails to run at all. Normal sessions end earlier than the extension limit (180 mins), so they never hit this deadline.

3. What Gets Deleted and in What Order

The backend directly deletes only two things.

1. VirtualMachine Objects
2. cloud-init Secret (containing the grading token)

Everything else is subsequently deleted by Kubernetes and KubeVirt. The order is as follows.

One of the reasons the backend does not need to manually coordinate this order is Kubernetes' Storage Object in Use Protection. According to Kubernetes documentation, if a PVC in use by a pod is requested for deletion, the actual deletion is deferred until the pod stops using it. Thus, the order of VM → pod → volume is naturally preserved, preventing disks still in use by the guest from disappearing prematurely. The same protection applies when deleting golden images, leaving golden PVCs as "Terminating" until all VMs using them have finished (Part 4).

For a three-node session, deleting a single representative VM causes the remaining VMs, Secrets, session-exclusive network policies, and services to be deleted together via labels. In actual measurements, 15 seconds after deleting one representative, all 3 VMs, 3 Secrets, services, and policies dropped to 0, and when a second worker creation was made to fail, even the already created ones were entirely reclaimed (2024-09-24).

Why Delete Secrets Separately Again?

Since the cloud-init Secret lists the VM as its owner, the garbage collector should delete it when the VM is deleted. However, that owner link can fail. Attaching the owner to the Secret after creating the VM is a separate modification request, and security audits indicated that because the backend service account lacks Secret modification permissions , this linking consistently fails. In fact, an orphaned cloud-init Secret was found lingering 32 days without a VM. Therefore, the backend explicitly deletes the Secret by name after deleting the VM. If left behind, the old cloud-init would revive if the same session name is reused. Cleaning up remaining Secrets and permissions is documented as a manual task.

4. Upper Limits: Bounding the Scale of Damage

No matter how robust reclamation is, failures can happen. Therefore, we bound "how much can leak at worst" using upper limits.

The reason for having two layers is that if the backend's upper limit calculation fails (if KubeVirt stutters and cannot read the list, the backend does not impose an upper limit—because blocking here would halt all VM labs together), quotas act as a final safety net.

The reason the number 8 was not raised is also documented. When creating a multi-VM session (2024-09-24), the combined available memory request headroom of nodes actually providing KVM was only about 17 GiB, and during measurements, one node even entered NotReady status for about a minute due to system OOM. Raising the quota only changes rejections from "no capacity" to scheduling failures and node pressure. Node memory runs out first. The fact that VMs receive no discounts on memory requests (Part 2) and that when a node falls under memory pressure, it is the lab pods rather than the VMs that die first are also reasons why the upper limit was set conservatively.

5. Remaining Gaps

• VMs lack final safety measures like pod deadlines. If the reaper stops, expired VMs remain as they are. At that point, the only things bounding the damage are the aforementioned upper limits (8 VMs, 48Gi). Setting deadline configurations directly on VM objects is unverified.
• VM orphan cleanup fails as a single block. While expiration loops isolate failures per row, the loop scanning for VM orphans is merely wrapped as a whole to prevent KubeVirt instability from crashing the entire reclamation process. Thus, if a single orphan fails to be deleted, the remaining VM orphans in that cycle are deferred to the next cycle. They do appear in the logs.
• The distribution of time taken from "expiration to VM disappearance" in student sessions is unmeasuredSince the reclamation period is 60 seconds, we expect it to take a little over a minute, though we haven't measured the exact value including the guest shutdown grace period (30 seconds).

TT's Note. The shorter the cleanup code, the better. The backend only cleans up two things, leaving the rest to "who owns whom" and "do not delete while in use." Also, skip failed rows and retry on the next iteration! Everything must not halt because of a single failure.

References

All documentation was accessed on October 16, 2026.

• Kubernetes Documentation, Persistent Volumes (Storage Object in Use Protection): https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• KubeVirt Component Documentation (virt-launcher Termination Handling): https://github.com/kubevirt/kubevirt/blob/main/docs/components.md
• TT Lab Code: backend/app/main.py(reap_once·reaper·delete_vm_session·delete_session_resource·extend_session·count_live_vm_sessions), backend/app/vm_session_scope.py, backend/app/vm_group.py, deploy/vms/rbac.yaml(ResourceQuota)
• Incident and Inspection Log: docs/plans/2026-09-12-vm-reaper-environment.md, docs/plans/2026-09-24-multi-vm-sessions.md, docs/SECURITY-REVIEW-2026-10-14.md