Hairpin NAT Measurement 2: When a Pod Calls Its Own Service

Status: Measured (2026-11-01). These values were measured on the TT Lab cluster (Kubernetes 1.34.10, Cilium 1.20.1). Cluster configurations were not changed, and no new Services were created. Among existing Services, one with a single endpoint was selected, a GET request was sent from inside that pod, and control measurements were performed using a temporary pod. Addresses have been rewritten to documentation examples (192.0.2.0/24), while Cilium's default value of 169.254.42.1 as well as times and ports are actual values. Writing rules: Measured data is the baseline; unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. Starting with the Conclusion
• 2. Why It Fails Out of the Box
• 3. Who Does What
  • kube-proxy: "SNAT if Self" Rule
  • kubelet hairpinMode: Present But Does Nothing Now
  • Per CNI
  • Cilium: Rewrites Source to 169.254.42.1
• 4. Measurement Environment and Method
• 5. Measurement Results
• 6. Limitations
• 7. Summary

1. Starting with the Conclusion

It is common for a web app to call its own API using its Service name. If there is a single replica, or if load balancing happens to pick itself even when there are multiple replicas, the request returns to the sending pod. Part 1's path has the exact same shape as a router.

• The root of the problem is the same. If you simply change the Service address to the pod address (DNAT), both the source and destination become the pod itself. Linux drops such packets when received from an external device, and bridges do not forward packets back out to the port they were received on.
• The solution is also the same: change the source address. kube-proxy's iptables mode inserts a rule to "masquerade if the source is that endpoint itself", and bridge-based CNIs turn on hairpin_mode on the bridge port. Cilium changes the source to 169.254.42.1 in eBPF.
• kubelet's hairpinMode does nothing now. Although the default value remains and the documentation is unchanged, since kubenet—which used this value—was removed in 1.24 along with dockershim, we could find no place in the source code reading it other than for configuration validation. Hairpin is a CNI responsibility.
• Measurement: A GET request sent by a pod to its own Service in Cilium 1.20.1 resulted in 200 (1.17ms), the source seen by the server was 169.254.42.1, and connection tracking had the LBLoopback mark attached. Three control requests sent from other pods displayed their pod addresses as is.

2. Why It Fails Out of the Box

Pod P (192.0.2.133) makes a request to Service S (192.0.2.96:80). S has only a single endpoint, which is P itself (port 8000). If the component handling the Service (whether kube-proxy's iptables rules or CNI's eBPF) changes only the destination, the packet looks like this.

바꾸기 전  출발 192.0.2.133:48766  →  도착 192.0.2.96:80   (Service)
DNAT 뒤    출발 192.0.2.133:48766  →  도착 192.0.2.133:8000 (자기 자신)

A packet with the same source and destination must re-enter the pod from outside the pod (toward the node). Two things block this here.

1. The bridge refuses to send it back. If the pods are attached to a Linux bridge (e.g., cni0), the new destination of the packet entering P's veth is once again P's veth. By default, a bridge does not forward packets out of the port they were received on. Allowing this is the hairpin setting of the bridge port. bridge(8) The explanation is as follows: it determines whether to send packets back out of the receiving port, also called reflective relay, and the default is off. In sysfs, it appears as /sys/class/net/<브리지>/brif/<포트>/hairpin_mode .
2. Packets whose source is their own address are treated with suspicion. From the perspective of the kernel inside the pod, a packet entering through an external device (eth0) has its own address as the source. Cilium source code comments also point this out, explaining that Linux drops such packets as martian sources unless they are received from the loopback device (bpf_lxc.c lines 259–264). Even if accepted, the response becomes a packet going "to itself" and terminates inside the pod, bypassing the reverse NAT (reverse transformation) back to the Service address. The sending socket awaits a response from S:80, but receives a response from P:8000 instead, failing to establish a connection just like the router in Part 1.

Therefore, the solution is the same as in Part 1: change the source address to a different oneso that the response returns to the component handling the Service and undergoes reverse transformation.

3. Who Does What

kube-proxy: "SNAT if Self" Rule

kube-proxy's iptables mode creates a chain for each Service endpoint and places at the very front a rule to attach a masquerade mark if the source matches that endpoint address . The source code comment says "Traffic returning to the originator is handled via SNAT" (proxier.go lines 1342–1346, v1.34.0). Even with this rule, if the pod is attached to a bridge, problem 1 (bridge not sending back) remains, so hairpin on the bridge port is also required.

kubelet hairpinMode: Present But Does Nothing Now

The kubelet configuration has hairpinMode(flag --hairpin-mode) with three possible values (KubeletConfiguration v1beta1 types.go lines 463–476, v1.34.0).

The place that actually used this value was the network plugin kubenet included in kubelet. Looking at the 1.23 source code, when kubenet is hairpin-veth , it injects hairpinMode: true into the bridge CNI configuration, or else sets the bridge to promiscuous (kubenet_linux.go, v1.23.0). kubenet was removed in 1.24 along with dockershim. In the current source code, we could find no places reading this value other than flag definition(cmd/kubelet/app/options/options.go), default value, and configuration validation (checked via GitHub code search on the default branch and confirming that kubelet.go·cmd/kubelet/app/server.go contains no mention of hairpin in v1.34.0, 2026-11-01). Nowadays, the CNI handles hairpins.

We also verified this on the measured cluster. The node's kubelet configuration (/configz) remained at its default value promiscuous-bridge , but the node had no bridge for pods (only an unused docker0 ), and the 14 pod veths were not attached to any bridge. The kubelet logs retained since July did not contain a single line with the word hairpin .

The Kubernetes official documentation section Debug Services "A Pod fails to reach itself via the Service IP" still guides users to check kubelet's hairpin-mode flag, the "Hairpin mode set to" log line, and the cbr0 bridge. Based on the source code above, this guidance is no longer accurate for clusters after version 1.24. If hairpins fail, you should inspect the CNI configuration rather than the kubelet.

Per CNI

Other CNIs (such as Calico) are not listed here as we did not open their source code this time.

Cilium: Rewrites Source to 169.254.42.1

Cilium handles Services in two places.

• socket LB: The moment an app calls connect() eBPF translates the Service address to the backend address. Because sockets connect to the backend from the beginning, packet-level NAT is unnecessary (kubeproxy-free documentation "Socket LoadBalancer Bypass in Pod Namespace").
• tc LB: The tc eBPF of the pod veth intercepts and modifies the Service for every packet. According to the same document, if socketLB.hostNamespaceOnly=true(agent flag bpf-lb-sock-hostns-only) is set, socket LB is bypassed inside the pod and tc LB is used. This is configured for cases like Istio sidecars that need to see the original ClusterIP inside the pod, or pods where socket LB does not listen, such as KubeVirt.

Hairpins are a matter for the tc LB side. bpf_lxc.c selects a backend and, if the backend address matches the packet's source address , attaches a loopback mark to that connection (bpf_lxc.c lines 249–266). Then, lb4_dnat_request changes the destination to the backend while modifying the source to the configured value service_loopback_ipv4 (lb.h lines 2355–2376). Its default value is --ipv4-service-loopback-address of agent flag 169.254.42.1 , and its description is "IPv4 source address used for SNAT when a pod communicates with itself via a Service" (cilium-agent command reference). IPv6 is fe80::1 .

There is one more thing. According to comments in the same file, traffic flows where an endpoint goes to itself via a Service bypass Network Policy checks. This is because otherwise, users would have to write separate policies stating that "pods are allowed to communicate with themselves" (bpf_lxc.c lines 1501–1508).

If socket LB is configured to be used inside the pod as well, connect() directly connects to its own address, so the hairpin path itself above likely will not be created (estimation; the measured cluster did not use socket LB inside the pod, so it was not measured).

4. Measurement Environment and Method

Creating a new Service was described as not performing cluster writes. Instead, we chose an existing Service with only a single endpoint. Since requests to this Service have only that single pod as a backend, sending a request from within that pod will invariably return to itself.

1. Hairpin: Sent a GET /healthz to the Service address (S:80) via a Python socket from inside the pod. Recorded the self-address and peer address seen by the socket (getsockname, getpeername), the response, and the elapsed time.
2. Server side: Immediately afterwards, we found the request line in that pod's access logs.
3. Connection tracking: Read the entry for that connection using cilium-dbg bpf ct list global on the same node's Cilium agent.
4. Control: Launched a temporary pod --rm on the same node, sent three identical requests to the same Service, and checked the source IP recorded in the server logs.

5. Measurement Results

Here is a single line from the server logs and the connection tracking entry (only the addresses have been modified for the example).

INFO:     169.254.42.1:48766 - "GET /healthz HTTP/1.1" 200 OK

TCP SVC 192.0.2.133:48766 -> 192.0.2.96:80  ... RevNAT=177 BackendID=216
TCP OUT 169.254.42.1:48766 -> 192.0.2.133:8000 ... Flags=0x001b [ RxClosing TxClosing LBLoopback SeenNonSyn ] RevNAT=177
TCP IN  169.254.42.1:48766 -> 192.0.2.133:8000 ... Flags=0x001b [ RxClosing TxClosing LBLoopback SeenNonSyn ]

How to read this:

• The socket knows it is connected to the Service. The client socket's peer was S:80. Since socket LB is not used inside the pod (Hostns-only), this means address translation occurred at the veth packet level rather than in the socket. If socket LB had been used, the peer would have appeared as the backend address.
• The server views the request as coming from 169.254.42.1. Just as described in the source code in Section 3, the source was changed to the loopback address. The port (48766) remains unchanged.
• Three entries were created in the connection tracking table. The original connection to the Service (SVC), and bidirectional entries (OUT/IN) going from the loopback address to the pod, with LBLoopback attached to the latter two. Responses are routed back to the Service address following this entry.
• The control request had its source unchanged. Requests from other pods were logged with the originating pod's address without SNAT. Modifying the source only happens in the case of hairpins.

Additionally, in the same logs, kubelet's health check requests were recorded as originating from the node's Cilium internal address (cilium_host).

Comparing this with the router in Part 1, two things are identical: the source must be changed so that the response returns via the same path, and as a price, the server cannot see the real client. The fact that requests sent by a pod to its own Service are logged as 169.254.42.1 is not a bug, but a result of this design. If you have features that filter by source address (allowlists, rate limiting), you need to be aware of this address.

6. Limitations

• Since the hairpin request was sent from within a running production pod, it was only sent once. The 1.17ms time is a single measurement value, and the difference from the control (3 attempts) should not be interpreted as a performance difference.
• Because configurations were not modified, actual failures when loopback SNAT is absent were not measured (unmeasured). The failure explanation in Section 2 comes from source code comments and Linux behavior.
• A comparison between the kube-proxy iptables mode and bridge CNI combination, with bridge port hairpin_mode turned off and on, was not measured because the measured cluster lacks a bridge (unmeasured).
• Cases using socket LB inside the pod were not measured (estimation in Section 3).
• Since network policy enforcement was disabled on the target pod, the source code explanation that hairpins bypass policy checks could not be confirmed via actual measurement (unmeasured).

7. Conclusion

Titi's Word. There is a mailbox labeled 'Letters for Titi' attached to my room door, and if I put a letter in it, it comes right back to my room. But if my name is still written in the sender field, I throw it away thinking, 'What a weird letter.' That's why Cilium the mailman changes the sender to '169.254.42.1 Post Office.' If you keep seeing that post office name in my diary, it means I wrote a letter to myself!