TT Lab
Blog Get started
Learn Learning paths Courses

Linux Network Diagnosis

Small Requests Work and Large Responses Stall

Continue in TT Lab

In one line

If curl gets the headers but stalls at the body, nine times out of ten it is an MTU black hole. Being able to connect and letting large packets through are different problems.

Why this exists

This kind of symptom keeps people stuck for a very long time.

Because "it connects", you do not suspect the firewall, and because "it works sometimes", you judge it is not DNS either. The cause is that large packets are silently dropped somewhere along the path.

MTU and PMTUD

The MTU (Maximum Transmission Unit) is the maximum size of a packet that can be sent at one time. Ethernet is usually 1500 bytes.

The problem is when there is a segment with a smaller MTU in the middle of the path.

client(1500) → router → VPN tunnel(1400) → server(1500)

TCP announces to each other when connecting, "I can receive up to 1460 bytes (MSS)". But this reflects only the two endpoints, not the segments in between.

If all is well, it works like this — when a 1500-byte packet reaches the 1400 segment, that router sends ICMP "Fragmentation Needed" to tell it "it's too big, reduce it to 1400". The sender reduces the size and sends again. This process is called Path MTU Discovery.

The black hole arises when ICMP is blocked

Many organizations say "ICMP is dangerous" and block all of it at the firewall. Then this happens.

  1. A large packet reaches the small-MTU segment
  2. The router drops it and sends ICMP
  3. The ICMP is blocked at the firewall
  4. The sender hears nothing and keeps retransmitting at the same size
  5. It fails forever. There is not even an error message

This is the PMTUD black hole. Small packets (the handshake, header requests) get through and only large packets (the body) vanish, which is why the symptom is so confusing.

Both end links are 1500 bytes but only the VPN tunnel in the middle is 1400 bytes. Small requests pass through the narrow segment as they are, but a 1500-byte packet is silently dropped at the entrance, and the ICMP fragmentation needed message that announces this is blocked by the firewall. The sender hears nothing and just keeps retransmitting at the same size

Diagnosis — how to check without privileges

Measuring while changing the size with ping -M do -s is the textbook method. ping runs either on a raw socket (NET_RAW) or on an ICMP datagram socket that Linux allows through net.ipv4.ping_group_range. This lab Pod has no NET_RAW, but that range is open, so ping -M do -s 1472 127.0.0.1 worked (measured 2026-11-06). Still, many environments block both, and when ICMP is blocked somewhere along the path, ping simply gets no answer without telling you why. So it pays to know a way that needs no permissions at all.

1. Check the path and the MTU

ip route get 8.8.8.8
# 8.8.8.8 via 10.0.0.1 dev eth0 src 10.0.0.5
ip -o link show          # mtu of each interface
cat /sys/class/net/eth0/mtu

2. Reproduce by varying the response size

curl -s -o /dev/null -w '%{size_download} %{time_total}\n' \
  'https://api.example.com/items?limit=1'
curl -s -o /dev/null -w '%{size_download} %{time_total}\n' \
  'https://api.example.com/items?limit=1000'

If only the small response succeeds, it is a strong signal.

3. Look at headers and body separately

curl -I https://target/            # headers only → works?
curl --max-time 5 https://target/  # full body → hangs?

4. Look at the socket state

ss -tin dst 10.0.3.20
# ... mss:1460 ... retrans:0/14 ...

If retrans keeps increasing, it means the same packet is being sent repeatedly. It is the typical fingerprint of an MTU black hole.

Response

Response Method Note
Allow ICMP Open at least type 3 code 4 on the firewall The fundamental fix
Lower the interface MTU ip link set eth0 mtu 1400 Requires privileges
MSS clamping Adjust the MSS on the router/firewall to fit the path A common field solution
Shrink application responses Pagination Only a workaround

What is used most often in the field is MSS clamping. If the gateway lowers the MSS value of the TCP handshake to fit the path MTU, large packets are never created in the first place.

Look at routing too

It may also be something other than MTU. If ip route get points to a different interface than you expected, the path itself is the problem.

ip route get 10.0.5.10
# if this says dev eth1, it is not going out eth0 as you assumed
ip rule show           # is there policy routing?
ip route show table all

It is common for the default route to change entirely when you turn on a VPN, and then traffic that should go to the internal network leaks out to the VPN, or the reverse.

What it looks like in the field

What to check in the quiz that follows

Check whether you can explain why the phenomenon of only small requests succeeding points to an MTU black hole, and whether you can tell the rise in retransmissions in ss -tin apart from other normal indicators.