The Day Public Access Dropped: MetalLB L2 Advertising Flapped, and the First Fix Made Everyone Stop Advertising

Status: Resolved (October 14, 2026), as of documentation time. The numbers for that day (such as the number of times the advertising node changed and the speaker restart count) were checked from working notes, and current settings and node labels were checked from the October 16, 2026 cluster read. What happened inside the router is written as an estimationbecause I could not view the router logs. Writing rules: Only facts from records, guesses are marked as 「estimation」, and unmeasured items are 「unmeasured」.

Table of Contents

• One-line Summary
• 1. Architecture: Router → Gateway IP → One node 「holding」 that IP
• 2. Symptoms: Failure only from the outside
• 3. Diagnosis: The advertising node changed over a thousand times
• 4. The first prescription was wrong
• 5. What was fixed: Two wired workers
• 6. But it was still invisible from the internal network: Hairpin
• 7. Next time I hear 「Access is down」
• 8. Remaining items and limitations
• References

One-line Summary

1. Architecture: Router → Gateway IP → One node 「holding」 that IP

Requests to the site go through port forwarding on the home router. The router forwards ports 80 and 443 to the cluster's gateway service IP, which is a LoadBalancer address assigned by MetalLB in L2 mode.

In L2 mode, that IP is not assigned to any machine's network card. When the router asks via ARP 「What is the MAC address for this IP?」, onenode running the MetalLB speaker answers 「Me」. MetalLB documentation explains how this node is determined (based on the documentation).

• All speakers generate a 「node+IP」 hash list using the same rule, and the first node advertises. Therefore, there is only one advertising node at a time.
• Node failures are detected via memberlist, and the surviving node takes over, broadcasting a gratuitous ARP to announce that the 「MAC has changed」.
• Limitations: All traffic concentrates on a single node, taking a few seconds to hand over, during which clients with slow ARP cache updates send to the old MAC.

2. Symptoms: Failure only from the outside

• Connecting to ports 80 and 443 using the site address from the outside No route to host.
• Sending directly to the gateway service from inside the cluster works normally. The public IP pointed to by the domain is correct, and DNS is normal.

The signal that 「it works inside」 was the most confusing part. The gateway and the web pods behind it were all healthy. The broken part was in front of them, between the router and the advertising node.

3. Diagnosis: The advertising node changed over a thousand times

Gateway service events log announcing from node "…" every time a speaker starts advertising (this is also the verification method recommended by MetalLB troubleshooting docs). Counting these events that day revealed that the advertising node changed 1,275 times(working notes).

• The speaker on one worker was in a crash loop. It had 360 restarts up to that day (the restart count read on October 16 was also 360 — it stopped after that). Since memberlist membership changes whenever a speaker dies and revives, according to the rule in section 1, the head of the hash list could change every time.
• Candidates for the advertising node even included a node connected via a laptop.

No route to host The path leading to is estimatedas follows. The router asks for the MAC of the target IP via ARP, but at that moment, if the advertising node is unreachable from the router (such as a laptop node behind wireless or a virtual network) or is in the middle of a handover, no reply comes back. Without a reply, the router returns a 「Host unreachable」 ICMP, and clients show it as No route to host . The router's ARP table and logs could not be viewed (unconfirmed).

4. The first prescription was wrong

If the problem is a flapping advertising node, the solution is to bind it to the most stable node. The first thought was the control plane set. Reboots are rare and workloads are low. I narrowed the nodeSelectors of L2Advertisement down to those three.

The result was worse. Even from the internal network , the site was cut off. This was because there was no node advertising at all this time.

The cause was a single node label.

• When kubeadm designates control plane nodes, it attaches the node-role.kubernetes.io/control-plane label along with node.kubernetes.io/exclude-from-external-load-balancers . The markcontrolplane step in the kubeadm source attaches both labels with empty values (blank strings), and the constant comment explains 「attached to control-plane for compatibility with old behavior」 (based on source). The official Kubernetes label list describes this label as 「a label that excludes nodes from the external load balancer's backend list」.
• MetalLB respects this label and does not advertise on nodes with it. Troubleshooting documents cover this details and the bypass flag (--ignore-exclude-lb). However, the section explaining nodeSelectors for L2 did not mention this.

Even if the label value is empty, if the key exists , it is excluded. Reading on October 16, all control plane sets had this label with empty values.

# 광고 노드를 고르기 전에 이것부터
kubectl get nodes -L node.kubernetes.io/exclude-from-external-load-balancers

nodeSelectors If all nodes chosen by metallb/metallb#3156(open as of October 16).

5. What was fixed: Two wired workers

Excluding the control plane, the remaining criteria are 「workers on the same wired LAN as the router and not frequently rebooted」. We pinned it to two such mini PCs.

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
spec:
  ipAddressPools: [<게이트웨이 풀>]
  nodeSelectors:
    - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values: [<워커 A>, <워커 B>]

The reason for using two is to let one take over if the other dies. Binding it to a single node causes the site to drop every time that node reboots. After rebooting one worker, HTTP 200 was verified from three external locations.

6. But it was still invisible from the internal network: Hairpin

Even after the fix, accessing via the public address (domain) from inside the house did not work. This is a different issue. This router does not support the path routing from internal network → public IP → back into the internal network (Hairpin NAT, NAT loopback). Therefore, it cannot report the external state of 「not working」 from inside the house, and conversely, 「working」 from inside the house does not guarantee it works outside.

Verification is split into two paths.

# 안: 공인 DNS 를 건너뛰고 게이트웨이 IP 로 바로 — 이름(SNI·Host)은 그대로 둔다
curl -sS -o /dev/null -w '%{http_code}\n' --resolve <도메인>:443:<게이트웨이 IP> https://<도메인>/

# 밖: 여러 나라에서 HTTP 요청을 보내 주는 외부 점검 서비스(check-host.net 등)로

7. Next time I hear 「Access is down」

1. Directly to the gateway IP (--resolve). If this fails, it is an internal cluster issue.
2. External monitoring service. If 1 works but 2 fails, the issue is between the router and the advertising node.
3. Check the current advertising node and flap count in the gateway service events (kubectl get events --field-selector involvedObject.name=<서비스>).
4. Check speaker logs for memberlist join failures and restart counts.
5. If the advertising node needs to be changed, check for exclusion labels first .

8. Remaining items and limitations

• The root cause of the worker whose speaker was in a crash loop was not covered in this post. That node is no longer an advertising candidate, so it has no impact on the site, but the underlying cause has not been fixed.
• The exact downtime in seconds when an advertising node changes was not measured (unmeasured). MetalLB documentation states it is a few seconds on most OSes.
• If two advertising nodes go down simultaneously , it drops again. Mechanisms to prevent this (such as anti-affinity reboot rules or alerts) do not yet exist.

A word from Titi. 「It works from the inside though?」 is the scariest phrase. Inside and outside have different paths, so a green light on one side does not guarantee the other. And when choosing nodes, look at labels before names! A single label might quietly be saying 「I'm out」.

References

All materials accessed on October 16, 2026.

• MetalLB L2 Concepts: https://metallb.universe.tf/concepts/layer2/
• MetalLB Troubleshooting (exclusion labels, --ignore-exclude-lb, checking advertising nodes): https://metallb.universe.tf/troubleshooting/
• Kubernetes Official Label List: https://kubernetes.io/docs/reference/labels-annotations-taints/#node-kubernetes-io-exclude-from-external-load-balancers
• kubeadm Source (Control Plane Designation Step): https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/markcontrolplane/markcontrolplane.go
• Documentation PR Sent Due to This Incident: https://github.com/metallb/metallb/pull/3156