Reading MetalLB: The controller that decides addresses, the speaker that raises its hand, and a one-hash election

Status: Draft (2026-10-17), partially measured. Reference version: metallb v0.16.1 (commit 791dfcc63ced). Since a small cluster is running this exact version (including controller, speaker, and frr-k8s for FRR mode), measurements were read-only obtained from that cluster on 2026-10-17 (reading CRs, node labels, services, events, and speaker logs). ARP packets were not directly captured. Writing convention: Measured data is the baseline; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. One-line Summary: The ticket counter that hands out numbers, and the person who raises their hand when called
• 2. Architecture Diagram
• 3. Code Map
• 4. Following the Core Path
  • 4.1 Controller: One IP per service
  • 4.2 Speaker: "Is it my turn?" for every service
  • 4.3 L2 Election: Filtering and lining up by hash
  • 4.4 Memberlist: The source of "is alive"
  • 4.5 Responding to ARP: Responder and gratuitous ARP
  • 4.6 BGP and FRR mode (Briefly)
• 5. The Author's Contributions and Where to Go Next
• 6. Measurement: Read from a small cluster
• 7. Where to Read First
• 8. Good Places for a First Contribution
• 9. Limitations
• References

1. One-line Summary: The ticket counter that hands out numbers, and the person who raises their hand when called

In the cloud, when you create a type: LoadBalancer service, the cloud attaches an external IP. Bare metal lacks that "cloud", so services remain forever as <pending> . MetalLB fills that gap.

It has two jobs. The ticket counter(controller) determines one IP from a pool for each service and writes it into the service's status. The person raising their hand (speaker) runs one per node, and when the network asks, "Whose IP is this?", only a designated single node answers "Mine" (L2 mode), or it announces the route to the router saying "Send this IP to me" (BGP mode).

The two jobs are separated because they require different permissions and locations. Choosing an IP must be done in one place while looking at the entire cluster to avoid collisions, and it requires no special privileges. Announcing must read and write ARP packets on each node's network card, so it must run on every node with hostNetwork and NET_RAW privileges. The side with greater privileges (speaker) does not choose IPs, and the side that chooses IPs (controller) does not touch packets.

2. Architecture Diagram

How to read the diagram:

• The controller on the top left looks only at the Service and IPAddressPool from the API server, decides the IP, and writes it to status.loadBalancer.ingress . It sends nothing over the network.
• The speakers in the middle read the IP written in the status and verify whether each other are alive via memberlist (port 7946). For a single service, all speakers independently perform the exact same calculation, and only the single node ranked first by the calculation result advertises. There is no separate consensus (leader election) process to choose a representative.
• The colors represent the current state of the small cluster. Worker A is advertising, Worker B is standing by, the control plane set is excluded from candidates due to an exclusion label, and GPU nodes are outside the nodeSelectors of L2Advertisement (Section 6).
• BGP on the bottom right is dashed. The small cluster is installed in FRR mode, but since there is no BGPPeer, it does not perform BGP announcements.

3. Code Map

4. Following the Core Path

4.1 Controller: One IP per service

The controller's SetBalancer, controller/main.go:52 is called whenever a service changes. After copying the service and matching it to the convergeBalancer, controller/service.go:49 as the "desired state", if anything has changed, it updates only the status.

The thought process of convergeBalancer is: "Can I keep the IP I currently have -> If not, get a new one". If the service already has an IP, it re-registers that IP with the allocator (no-op if it matches the configuration); if the pool has changed and it is no longer allowed, it deletes it and gets a new one. The place where it gets a new one is allocateIPs, controller/service.go:252 . If the user specified an IP (viametallb.io/loadBalancerIPs annotation, etc.), it uses that IP; if they specified a pool, it uses that pool; if nothing is specified, it loops through all pools andAllocate, internal/allocator/allocator.go:363finds an empty address.

Multiple services can share a single IP. The conditions are the same sharing key(metallb.io/allow-shared-ip annotation value), non-overlapping ports, and the same backend key (checkSharing, allocator.go:728). When the controller deletes a single service, it triggers "reprocess all services" (SyncStateReprocessAll). This is intended to let a service that was waiting because there was no slot take that empty slot.

The controller runs as a single Deployment without leader election (LeaderElection: false). Allocation state resides in memory, but upon restarting, it revives the state by re-registering the IP written in each service's status into the allocator (the first step of convergeBalancer). This design is possible because the source of truth is the API server's status.

4.2 Speaker: "Is it my turn?" for every service

The speaker's SetBalancer, speaker/main.go:385 is also called per service. If it is not a LoadBalancer, has no IP yet, or that IP does not belong to any pool, it withdraws its announcement. Otherwise, for each enabled protocol, it calls handleService, :446 , and if the protocol handler's ShouldAnnounce returns an empty string, it announces; if it returns a reason string, it withdraws the announcement with that reason (deleteBalancerProtocol, :495). When announcing, it leaves an info log serviceAnnounced and a service event nodeAssigned("announcing from node …"), and when withdrawing, it leaves serviceWithdrawn and the reason (notOwner, noMatchingAdvertisement …). The numbers in Section 6 are counts of these lines.

4.3 L2 Election: Filtering and lining up by hash

The L2 handler's ShouldAnnounce, speaker/layer2_controller.go:83 is the entirety of the election. Reading it in order goes like this:

1. If the service has no prepared endpoints, notOwner. An IP with no pods to receive traffic is advertised by no one.
2. If none of the L2Advertisements matching the service select my node, noMatchingAdvertisement (adsMatchNodeL2, :213). nodeSelectors are pre-converted into a set of node names when reading the configuration (l2AdvertisementFromCR, internal/config/config.go:743).
3. It builds the candidate node list (speakersForAds, :253).

	for s := range eligibleNodes {
		if k8snodes.IsNetworkUnavailable(nodes[s]) {
			level.Debug(l).Log("event", "skipping should announce l2", "service", name, "reason", "speaker's node has NodeNetworkUnavailable condition")
			continue
		}

		if !c.ignoreExcludeLB && k8snodes.IsNodeExcludedFromBalancers(nodes[s]) {
			level.Debug(l).Log("event", "skipping should announce l2", "service", name, "reason", "speaker's node has labeled 'node.kubernetes.io/exclude-from-external-load-balancers'")
			continue
		}

		if adsMatchNodeL2(ads, s) {
			res[s] = true
		}
	}

Source: metallb v0.16.1 speaker/layer2_controller.go, Apache-2.0

eligibleNodes are the speakers that memberlist currently considers alive (Section 4.4). Here, nodes with the NetworkUnavailable condition, nodes with exclusion labels, and nodes outside nodeSelectors are filtered out. The exclusion label check checks IsNodeExcludedFromBalancers, internal/k8s/nodes/nodes.go:30 , but it checks only for the presence of the key without looking at the value . Because kubeadm applies this label with an empty value, it is excluded even if the value is empty. Turning off this check is the speaker's --ignore-exclude-lb flag (speaker/main.go:99). 4. If the service is externalTrafficPolicy: Local , it keeps only nodes that have endpoints for that service. Since this is a policy to avoid forwarding across nodes unnecessarily, a node without endpoints must not seize the IP. 5. It lines up the remaining nodes.

	// Using the first IP should work for both single and dual stack.
	ipString := toAnnounce[0].String()
	// Sort the slice by the hash of node + load balancer ips. This
	// produces an ordering of ready nodes that is unique to all the services
	// with the same ip.
	sort.Slice(availableNodes, func(i, j int) bool {
		hi := sha256.Sum256([]byte(availableNodes[i] + "#" + ipString))
		hj := sha256.Sum256([]byte(availableNodes[j] + "#" + ipString))

		return bytes.Compare(hi[:], hj[:]) < 0
	})

	// Are we first in the list? If so, we win and should announce.
	if len(availableNodes) > 0 && availableNodes[0] == c.myNode {
		return ""

Source: metallb v0.16.1 speaker/layer2_controller.go, Apache-2.0

This sorting is the core of the design. Because every speaker calculates the same inputs(list of alive nodes, same configuration, same IP) using the same function, a single first-place winner is determined without asking each other. There is no voting and no locking. Since it hashes the node name appended with the IP, the winner differs per IP, and if there are multiple services, announcements are scattered across nodes (split into 4 and 6 in Section 6). Because the ingredient of the hash is the IP rather than the service, services sharing an IP will be advertised by the same node if their candidate lists are identical. This is because a single IP must not have two MAC addresses.

The tradeoff also stems from this sorting. If the input—the "list of alive nodes"—changes, the winner can change. If a node's speaker repeatedly dies and revives, the IPs for which that node was ranked first change owners every time. The event where "announcement nodes changed 1,275 times a day" recorded in "The Day Public Access Was Cut Off"is a natural consequence of this code. Therefore, narrowing down candidates with nodeSelectors to stable nodesserves as a remedy.

And there is a trap where step 3 and step 2 meet. If nodeSelectors select only nodes with the exclusion label attached, those nodes pass step 2 but noMatchingAdvertisement drop out at step 3, while other nodes drop out at step 2 into all speakers withdraw their advertisements. The only trace left is a single debug-level log line. This is what the PR in section 5 describes in the documentation.

4.4 memberlist: The Source of "Alive"

The input to the election, "alive speakers," is provided by HashiCorp memberlist. speakerlist.New, internal/speakerlist/speakerlist.go:57 uses the node name as the memberlist name (to match the endpoint's nodeName), binds to the pod IP and port 7946, and encrypts traffic with that hash if a secret key exists. If no bind address or label configuration is provided, it turns off memberlist; in that case, it assumes "speakers exist on all nodes" and uses the API server's node list as candidates as-is.

UsableSpeakers, :303 returns the current members of memberlist as-is. When members change (join/leave), memberlistWatchEvents, :339 leaves "node event - forcing sync" and forces recalculation of all services.

Why not just use Kubernetes' node Ready condition? It takes tens of seconds by default for the node controller to change a node to NotReady, during which a dead node remains holding its IP. Memberlist has nodes ping each other directly to suspect and mark failure within seconds. Instead, if there are nodes where speakers frequently die and revive, "recalculation" happens just as frequently. You can see traces of this in section 6.

4.5 Answering ARP: Responders and Gratuitous ARP

When winning an election, the L2 handler's SetBalancer, layer2_controller.go:131 registers the IP in Announce.SetBalancer, internal/layer2/announcer.go:258 . Packet processing runs separately.

• Which interface answers? updateInterfaces, announcer.go:61 periodically scans the node's network cards, creating an ARP responder for every interface that is up, not under a bridge, and not NOARP, and an NDP responder for every interface with an IPv6 link-local address. Exclusion lists (regex) are read from the ConfigMap metallb-excludel2 (parseAnnouncedInterfacesToExclude, speaker/main.go:572). If you specify interfaces in L2Advertisement, it responds only on that interface.
• Who asks? processRequest, internal/layer2/arp.go:73 responds with its own MAC address when it receives an ARP request that is either a broadcast or directed to it, provided the IP is registered and the interface matches (shouldAnnounce, announcer.go:237) . Because the speaker responds via raw sockets rather than the kernel, the node's IP configuration remains unchanged. This is why NET_RAW privileges are required.
• When the owner changes. For an IP that has newly started advertising, spamLoop, announcer.go:157 sends gratuitous ARPs every 1.1 seconds for 5 seconds. This is intended to quickly update the tables of switches and routers that remember the old MAC address. Comments note that the 1100ms interval is an empirical value from issue #172.

4.6 BGP and FRR Modes (Briefly)

The BGP handler's ShouldAnnounce, speaker/bgp_controller.go:155 has no elections. Every node satisfying the conditions announces routes to its router, and load balancing is handled by the router's ECMP. However, exclusion label checks are present here as well (:178).

The implementation that actually opens BGP sessions is selected via one of three choices in the environment variable METALLB_BGP_TYPE (speaker/main.go:109, newBGP, bgp_controller.go:492). frr-k8s , the speaker does not speak BGP directly and instead creates per-node FRRConfiguration CRs (updateConfig, internal/bgp/frrk8s/frrk8s.go:193). The frr-k8s daemonset then translates those CRs into FRR configurations. This separation allows things other than MetalLB (such as the node's own route advertisements) to share FRR on the same node.

5. Author's Contributions and References

metallb/metallb#3156(open as of 2026-10-17) is a documentation PR. It modifies two areas.

• Added a note to the nodeSelectors explanation in the L2 advanced configuration documentation: "Nodes with exclusion labels will not advertise even if they match nodeSelectors; kubeadm attaches this label to the control plane; selecting only those nodes results in no one advertising; the workaround is --ignore-exclude-lb(Helm speaker.ignoreExcludeLB)."
• Added this label to the "When speakers do not advertise services" list in the troubleshooting documentation.

The connection to the code remains as described in section 4.3. The label check is nodes.go:30, where L2 uses it is layer2_controller.go:268, and the remaining log is a single debug line at line 269 of the same file (the PR body notes this log was added in #2538 following issue #2505). The place where documentation lagged behind the code was the explanation that "nodeSelectors are a hard filter." While they are indeed a hard filter, it omitted the fact that there is another hard filter(exclusion labels) in front of it. The background story can be found in "The Day Public Access Was Cut Off".

6. Measurements: Insights from a Small Cluster

2026-10-17, all commands are read-only (kubectl get·logs). Node names have been replaced with roles, addresses masked, and raw data archived separately.

Configuration.

Even though installed in FRR mode, since there are no BGPPeers, no FRRConfigurations were created. The frr-k8s path from section 6 is currently idle in this small cluster configuration.

Exclusion labels are not visible via -L . kubectl get nodes -L node.kubernetes.io/exclude-from-external-load-balancers was blank across all eight nodes. Because the label value is an empty string, it looks identical to "non-existent." Checking by key existence with -l node.kubernetes.io/exclude-from-external-load-balancers revealed the control plane set. Combined with the check in section 4.3 that looks for the key rather than the value, this creates a combination where it appears as "no label" to human eyes, but "excluded" to code eyes. The NetworkUnavailable condition was False on all eight nodes.

Election Results. There were 10 ServiceL2Status entries left for each service being advertised by a speaker.

This is the result of 2 candidate nodes (step 4 in Figure 2) dividing the IPs in sha256 order. The service net sharing IPs had no L2 status, and all four nets had 0 ready endpoints. As per step 1 in section 4.3 (notOwner), no one advertises. Service events had announcing from node "<워커 A>" with protocol "layer2" logged 8 times per service (within the event retention period).

Memberlist and Responders. Startup logs of Worker A's speaker (current container started 10-14).

Although it is a node using only wired networking, ARP responders were attached to the wireless card and Cilium's VXLAN device as well , because there is no cilium_* in the default exclusion list and interfaces was not specified in L2Advertisement (section 4.5). No problems are apparent right now, but if you want to restrict advertisements to a specific interface, specifying interfaces is correct.

Traces of a Turbulent Day. Counted logs from Worker B's speaker (current container, approximately 22 days of logs).

Breaking down by date, memberlist suspect messages were 138 on 10-04, 39 on 10-05, 10 on 10-07, and 1 on 10-13, while advertisement lines on those same days were 1,343 on 10-04, 322 on 10-05, and around 10 per day thereafter. During the same period, GPU Node C's speaker restart count was 360 (last termination 10-08, exit code 1). Every time a speaker dies and restarts, memberlist membership changes (section 4.4), all speakers recalculate, and IPs for which that node was ranked first change owners (section 4.3). The noMatchingAdvertisement 2 entries (10-14) appear to be lines left when Worker B dropped out of candidates the moment nodeSelectors were changed to the control plane.

Speaker Restart Counts. Control plane set 0, Worker A 2, Worker B 10, GPU Node A 8, GPU Node B 0 (pod age 2 days), GPU Node C 360. Causes were not pursued in this article (unconfirmed).

Build String. Startup logs for both speaker and controller were MetalLB … starting version 0.16.1 (commit dev, branch dev) . Being an official image, the commit is recorded as dev . The first item in section 8 explains why.

7. Where to Start Reading

1. website/content/concepts/layer2.md. Grasp the concepts first (the documentation takes about 5 minutes to read).
2. speaker/layer2_controller.go:83. Read through the election function to the end. All five steps from section 4.3 are contained within it.
3. internal/speakerlist/speakerlist.go:57. Examine where election inputs come from, along with
4. internal/layer2/announcer.go:61 and arp.go:73. Where packets are actually answered after winning.
5. controller/service.go:49. The other side, where IPs are determined.
6. speaker/main.go:385. Finally, see how the two protocols fit into a single flow.

8. Good Places for a First Contribution

Things actually noticed while reading the source code. These have not been submitted anywhere yet.

• Release image commit shown as dev. The workflow that builds the image passes build-args in the form of 키: 값 (.github/workflows/publish.yaml:109).

          build-args: |
            GIT_BRANCH: ${{ github.ref_name }}
            GIT_COMMIT: ${{ github.sha }}

Source: metallb v0.16.1 .github/workflows/publish.yaml, Apache-2.0

docker/build-push-action ). Because build-args expects 키=값 per line, the Dockerfile default value ARG GIT_COMMIT=dev appears to remain as-is. Section 6 confirmed that two official v0.16.1 images display commit dev, branch dev . This is a candidate for a one-line fix to change it to = .

• Discrepancy Between Memberlist Configuration File Comments and Behavior. memberlistFileConfig Comments state that "if the file exists, all fields are mandatory, and missing fields cause an error" (speakerlist.go:357), but in reality, only time string fields trigger errors while integer and boolean fields quietly default to 0/false if omitted (e.g., IndirectChecks 0). Also, reading a file starts from LAN defaults regardless of --ml-wan-config (:397). internal/speakerlist has no test files at all, so adding tests would make a great first contribution.
• Error Messages. When backend keys differ, it says "does not match existing sharing key" (allocator.go:623). The unmatched quotation mark in the exclusion label function comment (nodes.go:29) falls into the same category.
• Silent "No Advertisement". The trap in Section 4.3 leaves only a single debug log. Leaving a service event when candidates become 0 due to exclusion labels is worth considering as a next step after documentation PR (#3156). Since it changes behavior, it is better to ask via an issue first.
• Default exclusion list and Cilium. In Section 6, we saw cilium_vxlan having an ARP responder attached. The default list (config/native/exclude-l2-config.yaml) contains Calico and Flannel device names, but for Cilium, there is only lxc . Verification is needed on whether there is any harm (unverified).

For the contribution process, CONTRIBUTING requires a DCO sign-off (Signed-off-by), and detailed guidance is deferred to the community page on the website. The PR template states that for non-trivial changes, you should open an issue first. As of 2026-10-17, there were 5 or more issues opened with the good first issue label (such as test execution documentation, container-level testing, etc.).

9. Limitations

• BGP mode was covered only at a high level. Session management, BFD, communities, and the internals of frr-k8s were not explored, and it could not be measured on a small cluster because there were no BGP peers (unmeasured).
• IPv6 and dual-stack assignment, avoidBuggyIPs, pool selectors (serviceAllocation), and webhook validation were not covered.
• How many seconds the connection actually drops when the advertising node changes and when gratuitous ARP fixes the router table were not measured by capturing packets (unmeasured).
• The log counts in Section 6 only count the current container logs of each speaker, so records prior to previous restarts are not included.
• The background and diagnostic sequence of the incident itself are covered in "The Day Public Access Was Lost".

TT's Word. If eight classmates each calculate "Whose turn is this ball?" using the same formula, exactly one person raises their hand without any meeting. But if you pick only the friends who have a "Count me out" sticker behind their name tags, the ball just rolls around on the floor!

References

All materials were accessed on 2026-10-17.

• metallb v0.16.1 source: https://github.com/metallb/metallb/tree/v0.16.1
• L2 concept documentation: https://github.com/metallb/metallb/blob/v0.16.1/website/content/concepts/layer2.md
• L2 advanced configuration documentation: https://github.com/metallb/metallb/blob/v0.16.1/website/content/configuration/_advanced_l2_configuration.md
• Troubleshooting documentation: https://github.com/metallb/metallb/blob/v0.16.1/website/content/troubleshooting/_index.md
• Contribution guide: https://github.com/metallb/metallb/blob/v0.16.1/CONTRIBUTING.md
• HashiCorp memberlist: https://github.com/hashicorp/memberlist
• Kubernetes official label list (exclusion labels): https://kubernetes.io/docs/reference/labels-annotations-taints/#node-kubernetes-io-exclude-from-external-load-balancers
• Author's documentation PR: https://github.com/metallb/metallb/pull/3156
• Same series: Reading CloudNativePG. Other series: The Day Public Access Was Lost

Quoted code and configurations belong to MetalLB and are licensed under the Apache License 2.0.