KubeVirt Live Migration Actual Measurement 1: The Guest Paused for 0.34 Seconds, But the Service Was Disconnected for 2.4 Seconds

Status: Measured (2026-10-31). These values were measured in a nested cluster with Kubernetes and KubeVirt v1.9.0 set up inside three TT Lab practice VMs. The baseline test was run five times, while the blocking condition and CPU model tests were run once per condition. All numbers are extracted from the raw data left behind for each migration run (VMIM phase records, libvirt job info, probe records). Writing rules: Measured data is the baseline; unmeasured items are marked as 「Unmeasured」; interpretations whose causes could not be verified are marked as 「Estimated」.

Table of Contents

• 1. Starting with the Conclusion
• 2. What Live Migration Does
• 3. Measurement Environment and Method
• 4. Baseline Measurement: 1Gi VM Without Load, Five Times
  • 4.1 Why a 0.34-Second Pause Becomes a 2.4-Second Outage
• 5. Conditions That Prevent Migration
  • 5.1 RWO Is Blocked Even If the Storage Is NFS
  • 5.2 The Trap of containerDisk and ctr
  • 5.3 Time Until Failure
• 6. CPU Model: Which Nodes Can Migrate to Each Other?
  • 6.1 What Happens with Mixed Hardware?
  • 6.2 How to Check on Your Node
• 7. Summary

1. Starting with the Conclusion

To fix a node or upgrade the kernel, you need to move the VMs on that node somewhere else. KubeVirt does this via live migration. Without shutting down the guest, memory is copied to another node, paused very briefly, and then resumed on the new node. Reading the documentation makes it sound like "almost no disruption," but when moving it ourselves, we found quite a few conditions.

• Guest pause and service outage are different numbers. Across five runs with an unloaded 1Gi VM, the pause measured by libvirt had a median of 343ms, but new connections going through the Service experienced a median outage of 2.4 seconds, and established TCP connections were dropped in all five runs.
• There are more unmigratable VMs than expected. If the disk is RWO, it is blocked even if the storage is NFS, and local-path, hostDisk, and pod network bridge are also blocked. containerDisk migrates, but if the image was pulled to the node via ctr , it fails after waiting 15 minutes.
• LiveMigratable: True does not look at the CPU. The default host-model requires all CPU features of the source node on the target, so even between nodes that look identically named, migration failed after 5 minutes due to one or two feature differences. In mixed hardware, setting up a named CPU modelwas the correct solution.

VMs with heavy memory usage, node draining, and concurrent migration limits will be covered in Part 2.

2. What Live Migration Does

In KubeVirt, a VM is a QEMU process running inside a pod (virt-launcher) on a node. Since a pod cannot be "moved" to another node, KubeVirt performs the migration by spinning up a new pod on the target node, copying the guest state over, and then terminating the original pod . The object that initiates this process is VirtualMachineInstanceMigration(VMIM for short). virtctl migrate simply creates one of these objects.

1. Target pod preparation. virt-controller creates a new virt-launcher pod on the target node. QEMU in this pod waits for an "incoming migration" without a guest. In our measurements, it took 2–4 seconds from creating the VMIM until the target pod was ready. If the target pod cannot be scheduled, it stops here(Sections 5 and 6).
2. Pre-copy. QEMU on the source side sends the guest memory. Because the guest continues to run during this time, pages that were already sent are modified again. Therefore, the entire memory is sent in round 1, and starting from the next round, only the pages that changed (dirtied) in the meantime are resent. Zero-filled pages are hardly sent, so out of a 1.016GiB guest, the actual amount sent was 456MiB. The iteration repeated 3–4 times at a speed of about 88MiB/s.
3. Stop-and-copy. Once the remaining amount can be sent within the target time, the guest is paused, and the remaining pages and CPU state are sent. This is when the guest actually pauses, which libvirt records as Total downtime . KubeVirt v1.9 starts the target at 300ms and increases it gradually if progress stalls (upper limit maxDowntimeMs default 900ms).
4. Resumption on target. The target QEMU resumes the guest, and the original pod is terminated. Disks are not migrated. This is because both pods attach to the same PVC simultaneously, which requires the PVC to be RWX (ReadWriteMany), allowing it to be attached across multiple nodes (Section 5).

Data transmitting the memory flows between the virt-handler pods of the two nodes—that is, through the pod network. Unless a dedicated migration network is provided, it shares the same line as service traffic.

3. Measurement Environment and Method

There are three measurement methods.

• VMIM Phase: Recorded the VMIM phase (Pending → Scheduling → Scheduled → PreparingTarget → TargetReady → Running → Succeeded) every 0.5 seconds.
• Values Measured by libvirt: After completion, elapsed time, total downtime, sent amount, and iteration count were read from virsh domjobinfo --completed on the target pod. The actually applied configuration was read from status.migrationState.migrationConfiguration of the VMI.
• Outage Observed from Outside: An echo server (ncat -lk 7777 --exec /bin/cat) was run inside the guest and probed every 20ms from the control plane via a ClusterIP Service targeting the VMI. There are two types: newly established connections per probe (new connection) and continuously kept connections (maintained connection). An attempt is counted as failed if there is no response within 200ms. "Outage" refers to the longest consecutive failure period.

The time taken to set up the environment is also noted: VM set preparation 276s, csi-driver-nfs 52s, KubeVirt Available 122s, CDI Available 91s, Fedora base disk retrieval 193s. We hit a snag once during CDI retrieval: if mountPermissions is not provided to csi-driver-nfs, the volume folder becomes root 0755, causing the importer pod (uid 107) to fail with permission denied in an endless restart loop. Since NFS does not change ownership via fsGroup, we resolved this by providing mountPermissions: "0777" to the StorageClass.

4. Baseline Measurement: 1Gi VM Without Load, Five Times

A Fedora VM (1Gi, RWX, masquerade, evictionStrategy: LiveMigrate) was round-tripped five times between the two workers. No migration policy was set (defaults used). The CPU was fixed to the named model Skylake-Client-v4 , which both workers could support. Section 6 explains why this was necessary.

It took about 10 seconds from creating the VMIM until completion, of which actual copying (libvirt elapsed time) took 6.4 seconds. The remainder consists of scheduling and spinning up the target pod (1–2s), target preparation (0–2s), and post-completion state cleanup.

4.1 Why a 0.34-Second Pause Becomes a 2.4-Second Outage

In masquerade , KubeVirt's default network binding, the guest is hidden behind a pod. The address visible from the outside is the IP of the virt-launcher pod, which passes traffic to the guest via NAT inside the pod. However, as seen in Section 2, migration is the process of creating a new pod and moving the workload there , so upon completion, the pod wrapping the guest changes and the pod IP also changes.

• Why established connections are dropped. The NAT state of a TCP connection originally established through Pod A resides within Pod A's network. When Pod A terminates, that state also disappears. All five persistent connections timed out, and the re-established connections succeeded.
• Why new connections also have nowhere to go for a while (estimated). Until the Service's endpoint transitions from Pod A to Pod B and is reflected in each node's rules, new connections will not reach their destination even after the pause ends. In the probe logs, the failure period started about 1 second before VMIM appeared as Succeeded and lasted for another 1-3 seconds. We assume that the time remaining after subtracting the 0.34-second pause was spent on the endpoint transition, but the exact time when the endpoint changed was not separately recorded (unmeasured).

Therefore, you should not design under the assumption that "it will not disconnect because it is a live migration." You must ensure that clients re-establish disconnected connections and endure failures lasting a few seconds through retries. We did not measure the same values in bindings that are not masquerade (such as a separate network attached via Multus) (unmeasured).

5. Conditions That Prevent Migration

The VMI status includes LiveMigratable conditions. If False, the reason is provided alongside, and if you attempt to create a VMIM, the admission webhook rejects creation itself. We actually tested this by creating a cirros VM for each condition.

5.1 RWO is blocked even if the storage is NFS

KubeVirt does not check whether the storage can actually be attached across multiple nodes, but only the PVC access mode We will see. Even if sharing is originally possible because the storage is NFS, if the PVC was created as RWO, it cannot be migrated. When inspecting the actual environment as read-only, we found that the existing VM was exactly in this case. The storage's StorageProfile can provide RWX, but the code creating the VM specified the PVC as RWO, resulting in all LiveMigratable: False being generated. To make migration possible, the root disk must be recreated as an RWX PVC (by shutting down the VM and cloning/changing it to RWX, or using volume migration in v1.9). Volume migration was not measured this time (unmeasured). The discussion on access modes and volume modes is covered in Part 3 until the student VM starts.

5.2 containerDisk and ctr Pitfalls

containerDisk (where the disk is received as a container image) can be migrated. It is handled via block migration (BlockMigration), which sends the write layer along with it. However, in environments where images are pre-loaded onto nodes, such as in air-gapped networks, there was a pitfall.

• For images pulled on nodes via ctr or imported via ctr images import , the storage digest (repoDigests) is empty in CRI, and the original pod's imageID leaves only sha256:<설정 digest> .
• To fix the target pod's image to "the same digest as the original", KubeVirt appends the sha256 value of imageID as-is to create <저장소>@sha256:<값> . Since this value is a reference that does not exist in the repository, the target pod falls into NotFound·ImagePullBackOff and the VMIM stayed in Scheduling for 15 minutes and 4 seconds before failing.
• A VM newly launched after pulling the same image again via crictl pull(CRI path) to populate repoDigests was successfully migrated.

This behavior has been reported as an issue upstream (kubevirt#19694). Until then, it is safe to pull pre-loaded images via crictl pull or verify that repoDigests has been populated via crictl inspecti after import.

5.3 Time Until Failure

The waiting time varies greatly depending on where it gets blocked. Conditions rejected by the webhook are known immediately, but if the target pod fails to schedule, it becomes Failed after 5 minutes, and if the target pod starts but fails to pull the image, it becomes Failed only after 15 minutes . To avoid waiting, you can cancel it by deleting the VMIM.

6. CPU Model: Which Nodes Can Be Migrated Between

Guests assume that they continue using the CPU features they had when first booting. If the destination node lacks those features, the guest may crash, so KubeVirt attaches a node selectorto the target pod to schedule it only on nodes with matching features. The CPU labels attached to each node (cpu-model.node.kubevirt.io/…, cpu-feature.node.kubevirt.io/…, cpu-model-migration.node.kubevirt.io/…) are applied by virt-handler's node labeller.

Since the two workers were on Intel hosts with different CPU generations (family 6, model 151 and 170), we were able to test the exact conditions directly. The node labeller assigned the same host-model nameto both workers, but the required features differed. One side had rfds-clear, and the other side had ddpd-u·rfds-no .

There are three takeaways here.

1. LiveMigratable does not calculate CPU compatibility. The host-model VM was True, but there was no place to move it, and the target pod stayed Pending for 5 minutes and unschedulable pod … timeout period exceeded ended with .
2. The target condition for host-model is "same host-model name as the source + all required features of the source node". Even if the names are the same, if it lacks even a single feature, it will not work. And since the host-model name is merely the "closest named model," it does not indicate the generation. Two hosts of different generations received the same name.
3. host-passthrough only checks the vendor. Therefore, it was moved even to a node of a different generation. Since this guest was checking features not present on the target (ddpd-u·rfds-no , etc.), it would have been dangerous if it were a program actually using those features. Passthrough must only be used between nodes with completely identical CPUs.

Since this relationship is not documented in the user guide, I submitted a documentation PR to add that section (user-guide#1052).

6.1 What Happens with Mixed Hardware

We applied the same rule (the rule verified empirically by the host-model selector above) to the labels of actual nodes consisting of four Intel nodes and one AMD node, and calculated the results. Excluding node names and grouping by pair type, it looks like this.

• With host-model, only pairs with completely identical CPUs are supported. Out of five nodes, there was only one such pair. The assumption that "it should work since they are in the same product family" did not hold, and the differences were one or two subtle feature bits like security mitigations or prefetcher controls.
• Specifying a named model broadens the options. There were 31 named models that four Intel nodes could provide together, among which models like Skylake-Client-v4·Skylake-Client-noTSX-IBRS· Broadwell-noTSX-IBRS·Haswell-noTSX-IBRS preserve AVX2. There are only 10 models that all five nodes including AMD can provide together ( IvyBridge·SandyBridge· Westmere·Nehalem·Penryn(including each -v1 )), losing AVX2. When vendors are mixed, you trade off "migratability" for "guest performance."
• In reality, there are factors other than hardware to consider. Nodes with low memory or nodes bound via device passthrough should be excluded on the receiving end, which further reduces the usable pairs.

This table is based on label calculations. The only thing verified in the nested testing is the single row for "two with only the same host-model name."

6.2 How to Check on Your Nodes

By comparing the CPU feature labels of two nodes, you can predict in advance whether a host-model VM can be migrated.

diff <(kubectl get node A -o json | jq -r '.metadata.labels|keys[]|select(startswith("cpu-feature."))') \
     <(kubectl get node B -o json | jq -r '.metadata.labels|keys[]|select(startswith("cpu-feature."))') \
  && echo "A·B CPU 기능 같음"

The named models that two nodes can provide together are the intersection of labels starting with cpu-model.node.kubevirt.io/ . For a VM to be migrated in a mixed node setup, you write one of them in spec.domain.cpu.model , or change the cluster default via the spec.configuration.cpuModel of the KubeVirt CR.

7. Conclusion

The empirical tests in this post can be followed step-by-step in TT Lab's hands-on course "KubeVirt Live Migration" (site path /course/kubevirt-live-migration). You follow the same VM set and setup to go through shared storage, KubeVirt, choosing a common CPU model, observing migration and downtime, observing RWO rejection, and draining. If KubeVirt's architecture is new to you, the series Until Student VMs Start Upand Reading KubeVirt Source Code will serve as a foundation.

TT's closing word. During the move, the guest waited at the door for 0.34 seconds, but the guest reported that the doorbell didn't work for 2.4 seconds. Because the house address (pod IP) changed! And the moving truck only goes to "neighborhoods with the same CPU features." Neighborhoods that only sound similar by name turn out to have no roads only upon arrival, so make sure to align your maps in advance.