The operator that searched for an unused webhook every 6 seconds: How the KubeVirt upstream PR was merged in 2 days and 16 hours

Result: Merged (2026-10-20 16:05, kubevirt/kubevirt#19499). As of 2026-10-29, no release contains this change yet. In fact, this was taken from GitHub PRs, reviews, inline comments, timelines, check logs, the KubeVirt source (the PR's base commit 580886af), and raw measurement data from the contribution ledger (operator metric raw values, reconcile intervals) and drafts. Times are in Korea Standard Time.

Table of Contents

• One-line Summary
• 1. What was the problem: 843,995 404s over 59 days
• 2. Cause: Which line in the source
• 3. What changed: Diff summary
• 4. Reaction: One inline comment, two approvals
• 5. What I learned: Tips for open-source contributions
• Links

One-line Summary

1. What was the problem: 843,995 404s over 59 days

TT Lab runs student practice VMs with KubeVirt. On 2026-10-17, while browsing KubeVirt components in the TT Lab cluster using read-only commands only (from the contribution ledger's measure/kv), I collected the metrics exported by the leader virt-operator kubevirt_rest_client_requests_total as-is. The version was KubeVirt v1.9.0, Kubernetes v1.34.10, and the operator had been running for 59 days. The counter accumulates values since the process started.

The number of GETs receiving a 404 matched the number of LIST nodes executed per reconcile down to a single digit. This means a 404 occurred exactly once per reconcile. These 404s accounted for 5.4% (843,995 ÷ 15,509,446) of all operator requests and virtually all non-2xx responses except for four.

Counting 237 lines of "Handling KubeVirt resource" in the leader operator logs on the same day showed a reconcile interval with a minimum of 6.007 seconds, median of 6.055 seconds, and maximum of 6.088 seconds (236 intervals). Dividing 59 days by 6.055 seconds yields about 842,000 times, which aligns closely with the counter (calculation). The fact that reconciles run every 6 seconds is a separate issue caused by Lease renewals, for which an issue (#19342) and#19417a fix PR (, open as of 2026-10-29) already existed. What we observed was another issue layered on top of that. Regardless of how many seconds each reconcile took, this GET was an unnecessary request from the start.

There was no noticeable damage. However, as noted in the PR description, these 404s always form a baseline error rate of about 5% beneath what VirtOperatorRESTErrorsBurst alerts watch. While well below the alert threshold (80%) so no alert fires, it obscures real errors on the dashboard.

2. Cause: Which line in the source

Line numbers are based on the PR's base commit 580886af .

• Called on every reconcile, createOrDeleteContainerPathVolumesWebhook creates the webhook ContainerPathVolumes if the feature gate virt-launcher-pod-mutator is enabled, and deletes existing webhooks if it is disabled.
• The disabled case was the problem. Lines 273–285 first checks the informer cache, and if missing, sends a direct GET to the API server with the comment "cache might not be populated yet." If it's a 404, it returns assuming there is nothing to delete.
• This gate is disabled by default, and clusters that have never enabled the gate never had that webhook in the first place. Therefore, the cache is always empty, a GET goes out every time, and a 404 returns every time.

So is the concern that "the cache might not be populated" valid? The controller waits for multiple informers to sync before starting its work. kubevirt.go Lines 169–194 shows the hasSynced list. The validating webhook informer (line 180 ValidationWebhook) was there, but the mutating webhook informer (MutatingWebhook) was missing. Thus, there was a reason the cache wasn't trusted, and the correct place to fix it seemed to be the sync list, not the GET.

This direct GET was introduced along with the webhook in #16662(VEP 165, merged 2026-02-24).

3. What changed: Diff summary

 # pkg/virt-operator/kubevirt.go: 시작 전에 기다리는 인포머 목록
 			informers.ValidationWebhook.HasSynced() &&
+			informers.MutatingWebhook.HasSynced() &&

 # pkg/virt-operator/resource/apply/admissionregistration.go
-	// Feature gate is disabled, delete the webhook if it exists
+	// Feature gate is disabled, delete the webhook if it exists.
 	_, exists, _ := r.stores.MutatingWebhookCache.Get(webhook)
-	if !exists {
-		// Also check directly in case cache is not populated
-		_, err := r.k8sClient.AdmissionregistrationV1().MutatingWebhookConfigurations().Get(
-			context.Background(), webhook.Name, metav1.GetOptions{})
-		... (404 면 nil, 그 밖의 오류면 반환, 있으면 exists = true)
-	}

Once the cache becomes reliable, absence in the cache means there is nothing to delete, eliminating the need for an API request. If the gate was enabled and then disabled, the webhook created by the operator always carries the managed-by label picked by the informer, so it appears in the cache and gets deleted as before. The PR body also noted the trade-off for this choice. If someone manually removes that label, the operator won't recognize the webhook and won't delete it. Other objects selected by the operator via labels are handled similarly. We considered an option to "leave the GET only until the first sync," but it would require tracking an extra state, so we chose the one-line sync addition.

For testing, we expanded the existing unit test "should not create virt-launcher-pod-mutator webhook when disabled." We made it fail if get was recorded even once with this webhook name in the fake client. The results stated in the PR body are as follows (Go 1.27.2, linux/amd64, GOFLAGS=-mod=vendor, main 6e80bca).

One time, ./pkg/virt-operator/resource/apply/ 's TestInstallStrategy failed, but we noted in the body that it did not reproduce when rerun on both the fixed and unfixed versions. The raw output file for this was not kept separately in the ledger. We did not measure whether 404s disappeared by deploying the fixed operator to the TT Lab cluster (unmeasured). This is because the TT Lab cluster is still on v1.9.0, and no release includes this change yet.

AI usage was disclosed as follows. KubeVirt's AI contribution policyencourages disclosing the use of AI tools via a commit trailer (Assisted-by:·Co-authored-by:·Generated-by:) while still requiring DCO sign-off. We added Assisted-by: Claude and Signed-off-by to the commits, checked the "AI Contributions" box in the PR template, and added a paragraph at the end of the body stating: "Claude was used to trace the cause from metrics and draft the changes and tests; all lines were reviewed, and the numbers above were measured in my own cluster." One thing didn't go according to rule: commit 47a37b0a , recreated to incorporate review feedback, appears as an unsigned commit on GitHub. It seems rule 3's SSH signature was omitted in the rewritten commit.

4. Reaction: One inline comment, two approvals

The review consisted of a single inline comment. The initially submitted version had a five-line comment where the GET was deleted, explaining: "The controller waits for webhook informer sync, and webhooks always have labels, so if they aren't in the cache there is nothing to delete. Querying the API server here causes a 404 GET on every reconcile for all clusters not using the gate." The reviewer questioned whether this was really such a big issue. Since event handlers detect webhook creations or deletions and re-queue them, they opined that the comment could be removed.

That made sense. Even if the cache is empty during the brief gap before sync, the moment a webhook appears, the informer triggers a re-reconcile. The comment was essentially exaggerating the usefulness of this GET. After deleting the five lines of comments, updating and pushing the commit, I replied briefly: "The informer re-queues on add/delete anyway, so the comment overstated it, removed." As a result, the PR code was left with just a one-line addition and the deletion of the GET block.

The approver was someone listed in OWNERS for the virt-operator directory, and also the author of #16662 which introduced this direct GET. Their remark was /approve along with a brief note of thanks. All 42 checks including phase-2 e2e passed, and tide merged it the next afternoon. The ledger notes this PR as the first merge into the main repository since KubeVirt was chosen as our primary stack.

5. What I learned: Open-source contribution tips

• 「The operator's own metrics provide the best reproduction."" client request counters are left along every code path. The single line showing that the 404 count equals the reconcile count proves the assertion that it happens "exactly once per reconcile." There was no need to dig through logs or attach a debugger.
• 「Fix why the workaround exists before deleting the workaround code." Direct GET was created because the cache could not be trusted, and the reason the cache could not be trusted was a single missing line in the synchronization list. Adding that line made it safe to remove GET. The shortest justification was that another informer of the same type (ValidationWebhook) was already nearby.
• 「Comments are not written to persuade reviewers." I left a long explanation in the code comments for what should have been in the PR description, and I received feedback about it. Code comments should only contain statements that remain true unless the code changes, while the reasons for this change should go into the commit message and PR description.
• 「Untrusted contributors cannot run tests directly." The KubeVirt repository does not allow external contributors to trigger tests with /ok-to-test , so a collaborator must type /test all . Therefore, it was more important to include local test results (failure before fix, pass after fix) in numbers within the PR description.
• 「Check whether the fixed commit also includes a signature." When recreating a commit to address review feedback, the signature was omitted. DCO passed, but it violates the policy. A quick git log --show-signature -1 before pushing is enough to check.

TT's summary. The gatekeeper knocked on the warehouse door every 6 seconds and asked, "Is there a box here?" even though the box was never in the warehouse in the first place. He couldn't trust the records because that particular warehouse was missing from his notebook's list. Once we added that single line to the notebook, the knocking that happened over 840,000 times for two months disappeared. The long note explaining "why the knocks were necessary" was taken down after someone said, "You can remove that."

Links

All materials were accessed on 2026-10-29.

• PR: kubevirt/kubevirt#19499 (Merge commit 6443e168)
• Fixed source (base commit): admissionregistration.go lines 266–301, kubevirt.go lines 169–194
• PR introducing direct GET: kubevirt/kubevirt#16662
• 6-second reconcile: Issue #19342, PR #19417
• KubeVirt AI Contribution Policy: https://github.com/kubevirt/community/blob/main/ai-contribution-policy.md
• CDI PR from the same investigation on the same day: Part 7, containerized-data-importer#4319