Hairpin NAT Measurement 1: Our Domain Only Fails to Open Inside the Office

Status: Measured (2026-11-01). Values measured from the TT Lab cluster behind a home router. No router, cluster, or network settings were changed; only reads and temporary pods were used. Addresses and domains were rewritten to documentation examples (192.0.2.0/24, 203.0.113.0/24, app.example.com), while times and counts are actual values. Writing Rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and interpretations whose causes could not be verified are marked as "Estimated".

Table of Contents

• 1. Starting with the Conclusion
• 2. What is the Problem?
  • When Only Changing the Destination
  • When Also Changing the Source: Hairpin NAT
  • What the Specification Requires
• 3. Four Solutions
• 4. Measurement Method
• 5. Measurement Results
• 6. What Happened at TT Lab
• 7. Limitations
• 8. Summary: Upon Hearing "It Only Fails Inside"

1. Starting with the Conclusion

We opened a web service behind a router and attached a domain. When accessed via mobile data, it opens fine, but it does not open on a laptop connected to the same Wi-Fi. The server is fine and the DNS is also correct. This is the hairpin NAT problem.

• The cause is the router. Requests sent to the public address by clients inside the private network must be sent back into the private network by the router (hairpin), but if it does not perform that processing or only changes the destination, the connection cannot be established.
• Hairpin NAT changes the source (SNAT) along with the destination (DNAT). This ensures that the server's reply returns via the router, allowing the router to revert the address and pass it to the client. The NAT specification states that this "MUST" be supported.
• If you cannot change what needs fixing, change the name. The most common answer is split DNS, which makes the same domain point to the internal address only within the private network. For a few devices, a hosts file also works.
• Measurement (No settings changed): When accessing the public address from three locations—laptop, cluster nodes, and pods—all 3 attempts resulted in a 5-second timeout, while going directly to the internal address with the same name returned 200. Three of the four external locations received 200.

A similar problem inside Kubernetes (when a pod requests its own Service) is covered in Part 2.

2. What is the Problem?

Routers in homes or small offices typically have a single public address and assign private addresses to internal devices. To send incoming requests from the outside to an internal server, you set up port forwarding. This is a rule like "Send traffic arriving at public address port 443 to server 192.0.2.10 port 443". The router changes the destination addressof incoming packets to the server address. This is called DNAT (Destination NAT).

This is sufficient for outside visitors. The server's reply follows the default route to the router, and the router uses the recorded connection information to revert the source to the public address and send it to the visitor. Since the visitor sent it to the public address and received a reply from the public address, there is no problem.

The problem is when the visitor is inside the same private network. When a laptop (192.0.2.20) looks up app.example.com , the public DNS returns the public address (203.0.113.7). The laptop sends a request to that address, and the request goes to the router, which is the default gateway. Here, there are three things the router can do:

1. Do nothing. Port forwarding is not applied to packets coming from the private network side. The router receives the address thinking it is its own, but since nothing is listening on port 443, it discards or rejects it.
2. Change only the destination (DNAT). This is the case in the diagram below.
3. Change both destination and source (DNAT + SNAT). This is hairpin NAT.

When Only Changing the Destination

When the router changes only the destination to the server address and forwards it, the source of the packet received by the server is the laptop's private address (192.0.2.20). From the server's perspective, the laptop is a neighbor on the same network, so it sends the reply directly to the laptop instead of routing it through the router. The source of the reply packet is the server address (192.0.2.10).

The laptop sent a SYN to establish a connection with port 443 of 203.0.113.7. However, the source of the returned SYN-ACK is 192.0.2.10. The laptop's TCP stack cannot match this as a reply to the connection it opened, so it treats it as an unknown packet (usually rejecting it with an RST). The laptop never receives the expected reply, sends a few more SYN packets, and eventually times out. Because the reply did not pass through the router, the router also has no opportunity to revert the address.

The key point is that requests and replies travel different paths. NAT requires both directions of a connection to pass through it in order to translate and revert addresses.

When Also Changing the Source: Hairpin NAT

When the router changes the source to its own address along with the destination (②), the server assumes the request came from the router and sends the reply back to the router (③). The router reverts both addresses according to the connection log and passes it to the laptop with the source address 203.0.113.7 (④). Since the laptop received a reply from the address it was waiting for, the connection is established. Because the packet enters the router and bends back like a hairpin to exit toward the same side, it is called hairpin NAT. Loopback NAT and NAT reflection are other names referring to the same feature.

There is also a cost. From the server's perspective, all connections from within the private network appear to come from a single router . Internal users cannot be distinguished using access logs, and access limits or rate limits based on source addresses count all internal users as a single person. Since internal traffic passes through the router twice, it also consumes router throughput.

What the Specification Regards

IETF documents defining NAT behavior do not leave hairpinning as optional.

• RFC 4787 (UDP) REQ-9: NATs MUST support hairpinning, and the source of a hairpinned packet must be an "External source IP address and port".
• RFC 5382 (TCP) Section 7.2 and REQ-8: The same requirement applies to TCP.

In the diagram, the source is depicted as the router's private-side address(192.0.2.1) for clarity, but what the specification requires is for the source to appear as a public address and externally used port. Either way, the core principle that the reply returns to the router remains the same. The "MASQUERADE" rule commonly used when building a software router with Linux—which triggers when the source is the private network and the destination is the server—translates to the private-side address approach. Among actual home routers, some products do not support hairpinning contrary to the specification or allow it to be disabled (the router measured this time also fell under 1 or 2 in Section 2; we will see which one in Section 5).

3. Four Solutions

If you can fix the router, that is the cleanest approach, but that is not always possible. There are broadly four methods, and in practice, two or more are often mixed.

A few additional notes:

• Split DNS goes well with TLS. Because the name remains unchanged, the certificate name, SNI, and Host header do not change. Typing the internal address directly as an IP causes certificate name mismatches, but split DNS and hosts files do not have this problem. To temporarily change it for a single command, curl --resolve app.example.com:443:192.0.2.10 does this job.
• Split DNS does not guarantee that "external access works". Just because it opens fine internally does not mean it opens externally. External status must be checked from the outside (Section 5).
• Router settings were not verified this time. Due to the rule of not changing settings, whether this router has a menu to enable hairpinning and its behavior when enabled remain unmeasured.

4. Measurement Method

We compared "accessing both the public address and the internal address using the same name" without changing settings across three locations.

The command structure looks like this. A 5-second limit was placed on each attempt.

# 공인 주소로 (DNS 를 건너뛰고 공인 주소를 강제, 이름은 그대로)
curl -sv --max-time 5 --resolve app.example.com:443:203.0.113.7 https://app.example.com/
# 내부 주소(게이트웨이)로
curl -sv --max-time 5 --resolve app.example.com:443:192.0.2.10 https://app.example.com/

External comparison was performed at four locations using a public checking service that sends HTTP requests from multiple countries (check-host.net HTTP check).

5. Measurement Results

It showed the exact pattern of the hairpin problem where it opens from the outside but not via the public address from the inside.We had noted "Internal hairpin blocked" in the 2026-10 incident log, and it remained the same on 2026-11-01. The connection refusal from one external location was deemed an issue on that specific location's end since the other three locations were normal, though the exact cause was not confirmed (estimated).

We measured a few more things.

• The router directly holds the public address. When doing a traceroute to the public address from the private network, the first hop was the public address itself, and ping replied 3/3 times with 0.65~1.5ms from the laptop and 0.37~0.50ms from the node. The packets reach the router, and the router recognizes the address as its own. This means another layer like carrier-grade NAT (CGNAT) is not involved.
• It was a timeout, not a refusal. We received neither RST nor ICMP errors on ports 443 and 80 of the public address. Among the three cases in Section 2, "Do nothing (drop)" and "Rewrite destination only" look identical to the client as a timeout.
• We could not distinguish which of the two it was (not measured). If the router only rewrites the destination, a SYN originating from the laptop's address should arrive at the gateway side. We checked the gateway node's flow logs (Hubble) and connection tracking table, but successful internal connections also left no L4 logs in these two places (due to the architecture where the gateway receives them as an L7 proxy), so they could not be used for determination. On the server side, tcpdump , but administrator privileges were required so we did not do it this time.
• The "internal address" measured from the node did not cross the private network. Requests sent from the node to the gateway address left a record of being processed by that node's own L7 proxy. This is because Cilium directly handles connections going to the LoadBalancer address within the node's host namespace inside the node. On the other hand, requests sent from the laptop left a record of arriving at the node responsible for the gateway address with the laptop's address. When verifying "it works with the internal address", it must be measured from an internal office device outside the clusterto verify the private network segment as well.

6. What Happened at TT Lab

Generalized, it comes down to this.

1. The service domain would not open only within the office. Since external guests had no issues, we initially suspected the DNS or browser cache of the office device. In reality, the router was not doing hairpinning.
2. Internal tests could not indicate external status. When an outage occurred where external access was genuinely broken, because it was a name that never opened properly from the inside anyway, "it doesn't work" did not provide new information. Conversely, "it works" with the internal address from the inside does not guarantee external access. Afterward, we split verification into two paths: check the inside directly via --resolve to the gateway address, and check the outside using an external monitoring service.
3. We used different solutions for different locations. Inside the cluster, we set up split DNS using CoreDNS's hosts plugin (this is why names resolved to internal addresses in pods). For development laptops, we pinned them using the hosts file. We did not touch the router settings.
4. There was a forgotten hosts file. One node's hosts file had the name mapped to the public addressinstead, so any connection targeting the domain from that node always failed. This is a clear demonstration of the drawback of hosts files: "they must be maintained manually per device, and they get forgotten."

If outbound traffic to its own public address is blocked from within the cluster, issues occur where cert-manager's HTTP-01 self-check fails, such as "it works from the outside, but internal verification fails." This was covered in How cert-manager Issues and Renews Certificates.

7. Limitations

• Low repetition count. 3 times per location, and 1 time for each of the 4 external locations.
• A single router. We could not distinguish whether the router "drops" or "rewrites destination only" (Section 5), and we did not measure whether there is a setting to enable hairpinning, its behavior when enabled, or the source address seen by the server when enabled (not measured).
• Only IPv4 was measured. IPv6 was not measured because there were no IPv6 addresses in the service names (not measured).
• External transit (proxies, tunnels, VPNs) was only listed in the comparison table and not measured (not measured).

8. Summary: Hearing "It only fails from the inside"

A word from TT. If I sent a letter to my own address and the post office said, "Oh, it's the same neighborhood," and immediately replied using the next-door neighbor's name, I'd throw it away as a letter for a stranger. The post office has to reply with their official post office stamp for me to accept it. If that doesn't work, you can just shout "Next room!" from inside the house (Split DNS)!