KubeVirt Live Migration Measurement Part 2: Busy Memory, Node Drain, Concurrent Limits

Status: Measured (2026-10-31). Part 1These values were measured in the same nested cluster (three TT Lab practice VMs, KubeVirt v1.9.0) as . Policy tests were conducted 1–3 times per condition, drain once per scenario, and concurrent limits once per limit. Writing convention: Measured data is the default, unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. VM with Constant Memory Activity
  • 1.1 Why Default Settings Fail to Complete
  • 1.2 Summary by Policy
  • 1.3 Bandwidth Limits Were Not Respected
  • 1.4 How to Apply Policies
• 2. Node Drain
  • 2.1 It is a Webhook, Not a PDB
  • 2.2 Eviction Flags Persist Even After Reversion
• 3. Concurrent Migration Limit
• 4. Limitations of This Measurement
• 5. Conclusion: Before Trusting Live Migration

1. VM with Constant Memory Activity

The VM in Part 1 did nothing. Real VMs use memory. Pre-copy works by "resending pages modified during the transmission", so if the guest's modification rate is similar to the sending rate, it never finishes. We reproduced this scenario.

• Inside the guest (Fedora, 1Gi), we ran stress-ng --vm 1 --vm-bytes 512M --vm-method write64 --vm-keep -t 0 . The same 512MB was continuously rewritten.
• Migrations were performed with policies applied one by one using MigrationPolicy . The default without policies had unlimited bandwidth, no completionTimeoutPerGiB: 150, progressTimeout: 150, maxDowntimeMs: 900, and autoConverge and postCopy disabled (values actually applied to the VMI).

1.1 Why Default Settings Fail to Complete

Reading ongoing libvirt task information every 5 seconds revealed that the unconverged runs repeated pre-copy 184–244 times over 300 seconds. The remaining data volume fluctuated between 15–90MiB, and the expected downtime required to send this was 1,782–1,913ms. It ultimately failed to reach the target (starting at 300ms and maxing out at 900ms). The guest's dirtying rate was about 19,000–23,000 pages per second (74–92MiB/s), which was nearly identical to the sending rate. The exact same amount remained after every iteration.

Even continuously writing only 128MB resulted in failure. During this time, the guest continued to run normally on the source, so there was no service disruption. However, because the VM could not be migrated, that node could not be drained (Section 2).

1.2 Summary by Policy

autoConverge is, as the name implies, a mechanism to "force convergence". When progress stalls, QEMU throttles the guest vCPUs slightly to lower the dirtying rate. Under this load, this feature alone changed the outcome from failure to success within the 30-second range.

postCopy works in the opposite direction. It starts running the guest on the target immediately, and any pages not yet transferred are fetched from the source the moment the guest accesses them. Because Linux's userfaultfd catches those moments, the node kernel must permit it, and Ubuntu's default was 0 (nested workers were also 0). Enabling it led to success, but KubeVirt waited until the pre-copy timeout expired before switching to PostCopy. Furthermore, since the guest's memory is split across both nodes during post-copy, the guest is lost if either node dies. For environments with busy VMs, trying autoConverge first is the order recommended by these measurements.

1.3 Bandwidth Limits Were Not Respected

bandwidthPerMigration: 32Mi When setting , the configuration was present across all layers. Both the applied VMI setting of 32Mi and QEMU's max-bandwidth: 33554432, virsh migrate-getspeed 32 were recorded. However, QEMU's info migrate throughput was 630.76Mbps (approx. 75MiB/s), and the amount processed over 10 seconds went from 17.316GiB to 18.040GiB (approx. 73.5MiB/s). Transmission occurred over 8 multifd channels. This is nearly identical to when there is no limit (86–90MiB/s).

Whether this is due to the relationship between multifd and speed limits or a KubeVirt-side configuration requires a control test with multifd disabled to distinguish (Unmeasured). Therefore, we currently recommend not trusting that "setting a limit protects the network" and instead verifying actual transfer volumes using virsh domjobinfo . For reference, the default bandwidth of 64MiB/s stated in the user guide is an old value, and the actual default in v1.9.0 was unlimited (0). A PR to fix the documentation is already open.

1.4 How to Apply Policies

MigrationPolicy selects targets by namespace or VMI label. This is an example of enabling only autoConverge in a namespace packed with busy VMs.

apiVersion: migrations.kubevirt.io/v1alpha1
kind: MigrationPolicy
metadata:
  name: busy-vms-auto-converge
spec:
  allowAutoConverge: true
  selectors:
    namespaceSelector:
      kubernetes.io/metadata.name: busy-vms

You can verify whether it was applied by checking the VMI's status.migrationState.migrationConfiguration and status.migrationState.migrationPolicyName after migration.

2. Node Drain

Live migration is most frequently used when draining nodes. Placing evictionStrategy: LiveMigrate on a VM causes kubectl drain to move the VM without shutting it down. We tested this path across four scenarios.

2.1 It is a Webhook, Not a PDB

During drain, kubectl get pdb -A was No resources found . Although the user guide's node maintenance documentation states that it "creates a PodDisruptionBudget per VMI", v1.9.0 receives eviction requests for the virt-launcher pod via an admission webhook(virt-launcher-eviction-interceptor.kubevirt.io). The webhook rejects the eviction while leaving an eviction flag (status.evacuationNodeName) on the VMI, which virt-controller observes to create an eviction migration (kubevirt-evacuation-…). Drain re-requests eviction every 5 seconds, and completes successfully once migration finishes and the source pod disappears. A documentation PR has been submitted to fix this discrepancy (user-guide#1051).

Therefore, whether a drain finishes depends on whether all VMs on that node are successfully migrated.

• Setting LiveMigrate on a VM that cannot be migrated (such as with an RWO disk) will cause the drain to hang indefinitely (Scenario B). VMs that cannot be migrated should be set to None or LiveMigrateIfPossible (whereLiveMigrateIfPossible is based on the API documentation and was not measured this time).
• If there is even one VM that fails to converge, the drain will stall waiting for that VM (Scenario A). autoConverge from Section 1 is also required here.
• The receiving node must have sufficient free memory(Scenario D). When there are two workers, draining one requires all VMs on it to fit into the other. If they cannot fit, the target pod remains Pending for 5 minutes, fails, is recreated, and fails again.
• Eviction migrations are also subject to concurrent limits. In Scenario D, while two units consumed the limit (2 per node), the remaining unit spent 12 seconds in Pending.
• The service interruption for the VM moved during the drain was 3607ms for new connections and 3606ms (interruption) for maintained connections. It was slightly longer than the basic test in Part 1.

2.2 Eviction Label Remaining Even After Reverting

Scenario C was the most unexpected. The drain failed, so the node was uncordoned, but the VMI's status.evacuationNodeName label remained as is. Seeing this label, the virt-controller continued creating eviction migrations (created twice after uncordoning twice), and when the load was turned off, one of them succeeded and the VM left the node just reverted. Even if you think, "Let's just stop the drain and revert," the VM is essentially still in the middle of moving.

The command to clear this label is virtctl evacuate-cancel . However, when executed with v1.9.0 virtctl, it failed like this.

Internal error occurred: the server rejected our request due to an error in our request

The server (virt-api) enforces a condition that "it is cleared only when the node name currently being evicted matches the request," which is a bug caused by v1.9.0 virtctl failing to include the node name in the request. Sending the same request with the body {"evacuationNodeName": "<노드>"} cleared it with a 200 status code. Since this has already been fixed (kubevirt#17008, release-1.9 backport #18864), virtctl v1.9.1 or higher should be used. Even if the server is v1.9.0, only virtctl needs to be updated.

3. Concurrent Migration Limits

After gathering six cirros VMs onto a single node, six VMIMs were created all at once. Only the parallelOutboundMigrationsPerNode in the KubeVirt CR was changed while the cluster limit (parallelMigrationsPerCluster: 5) was left as is.

Raising the limit from 2 to 4 did not make it faster. Since a single node's available bandwidth (approx. 90MiB/s) is shared, each operation becomes slower accordingly. If there are many small VMs, the gain from increasing from 1 to 2 (34s → 22s) is significant, but beyond that, the network bandwidth is the bottleneck.

The migration-dedicated network (spec.configuration.migrations.network) was not measured (unmeasured). The nested node had only two NICs and lacked Multus, so a separate network could not be created. In this configuration, migration, NFS disks, and service traffic all shared the same approximately 800Mbps line. In production, if VMs are large or numerous, a dedicated network comes first, and limits come second.

# KubeVirt CR 의 일부. 비워 두면 위 기본값(클러스터 5, 노드당 2, 대역폭 무제한)
spec:
  configuration:
    migrations:
      parallelMigrationsPerCluster: 5
      parallelOutboundMigrationsPerNode: 2

4. Limitations of This Measurement

Please verify the conditions before using these numbers.

• It is a nested cluster. Since KubeVirt was run inside VMs again, absolute times are slower than on physical nodes. In particular, downtime and speed are likely to be better on physical nodes.
• The network bandwidth is about 800Mbps. If it is 10GbE or higher, pre-copy can converge even under the same load. "Failure by default" in Section 1 is the result of this condition where "dirtying speed ≈ network bandwidth".
• The VMs are small. The basic test uses a single 1Gi guest, and the conditional test uses a 128Mi cirros. Times for guests of tens of GiBs were not measured (unmeasured).
• The number of repetitions is small. The basic test was run 5 times, policy tests 1–3 times, and the drain scenario and concurrency limits once each. Please read them as case studies rather than statistical distributions.
• Probes had a 200ms timeout per attempt, so brief failures were occasionally recorded even when no migration was occurring. Only the longest continuous interval was observed.
• Unmeasured items: impact of autoConverge on guest throughput, guest downtime for postCopy (target workload info was not retained), volume migration, network bindings other than masquerade, migration-dedicated network, and bandwidth comparison with multifd disabled.

5. Conclusion: Before Trusting Live Migration

If you want to try it yourself, TT Lab's hands-on course "KubeVirt Live Migration" (site path /course/kubevirt-live-migration) is prepared with the same VM set and setup as these measurements. It consists of eight steps, ranging from choosing a common CPU model, moving and reading downtime, reading RWO rejections, to draining.

TT's Note: If a friend keeps making a mess while packing, the move will never end. In that case, gently slowing that friend down (autoConverge) finishes the move in 30 seconds. And canceling the move doesn't mean it's over. If you don't remove the "Moving" note (evacuationNodeName) attached to the door, your belongings might end up moving out on their own!