How Does a Packet Choose Its Path

In one line

A host sends traffic directly if the destination is inside its own subnet, and to the default gateway if it is outside. The routing table is the list of rules for that decision, and the longest prefix wins.

Why it was needed

Reports like "ping works but I can't connect" or "only some ranges fail" mostly come from routing. But if you cannot read a routing table, you do not even know what to look at.

How it works

ip route
# default via 192.168.219.1 dev eth0
# 10.244.0.0/16 dev cilium_host scope link
# 192.168.219.0/24 dev eth0 proto kernel scope link src 192.168.219.50

The three lines mean the following.

192.168.219.0/24 dev eth0 — this range is directly attached. The host finds the peer's MAC with ARP and sends to it directly.
10.244.0.0/16 dev cilium_host — the Pod range goes out through the cilium interface.
default via 192.168.219.1 — everything else is left to the gateway.

If a destination matches several lines, the longest prefix wins. 10.244.1.5 matches both default (/0) and 10.244.0.0/16, but /16 is longer, so it wins. This is called longest prefix match.

To find the actual route for a particular destination, you can simply ask.

ip route get 8.8.8.8
ip route get 10.244.1.5

This command asks the kernel directly, "If I send to this address, where does it go out?" That is more accurate than reading the table by eye and reasoning about it.

The order for reading routes

Do not read the list from ip route from the top. The kernel picks the longest prefix (most specific) first. If the lengths are equal, it uses the one with the smaller metric.

10.0.5.0/24  dev eth1                  ← /24. 10.0.5.7 은 여기로
10.0.0.0/8   via 10.0.0.1 dev eth0     ← /8. 10.0.5.7 도 여기 포함되지만 진다
default      via 192.168.1.1 dev eth0  ← /0. 아무 데도 안 맞을 때

Rather than scanning the list by eye, it is more reliable to ask the kernel directly.

ip route get 10.0.5.7
10.0.5.7 dev eth1 src 10.0.5.2 uid 1000

src matters. Once the outgoing interface is decided, the source IP is decided with it, and if that address is not allowed by the peer's firewall, communication fails. Problems like "my IP is 203.x, so why is it going out as 10.x" show up here.

Rules → table → route

Linux has several routing tables, and ip rule decides which one to consult.

ip rule show
0:      from all lookup local        ← 자기 주소. 건드리지 않는다
32766:  from all lookup main         ← ip route 가 기본으로 보여 주는 것
32767:  from all lookup default

With VPNs, multiple uplinks and policy routing, rules get added here.

100:    from 10.8.0.0/24 lookup vpn   ← VPN 대역은 다른 테이블을 본다

In this case, if you look only at ip route (the main table), you end up with "the configuration is right but traffic doesn't go out." You have to look at that table separately with ip route show table vpn. Rules are evaluated from the smallest number first, and the table of the first matching rule is used.

ARP and the neighbor cache

If the destination is in the same subnet, traffic is sent directly by MAC address without going through a router. The table that maps addresses to MACs is the neighbor cache.

ip neigh show
10.0.5.1 dev eth1 lladdr 00:1a:2b:3c:4d:5e REACHABLE
10.0.5.9 dev eth1  FAILED                  ← 응답이 없다

FAILED means that no device has that IP or that it is not responding. Look here when routing is correct but communication fails. Right after you move an IP, the old MAC can stay in the cache and cause a few minutes of failure — clear it with ip neigh flush dev eth1.

Common misconceptions

The gateway must be in the same subnet. The default gateway is found with ARP, so it must be inside a directly attached range. If you enter an address from a different range as the gateway, you get Network is unreachable.

The belief that if ping works, the network is fine. Ping is ICMP, and services run on TCP. If a firewall opens only ICMP, ping works but connections do not. Conversely, if only ICMP is blocked, ping fails while the service works fine.

What really matters in practice

Inside a container, ip route is usually this simple.

default via 10.244.1.1 dev eth0
10.244.1.0/24 dev eth0

A Pod knows only its own node's gateway. The node knows the way to Pods on other nodes. So when Pod-to-Pod communication fails, you should look at the node's routing and the CNI, not inside the Pod.

Linux has several tables, and ip rule decides which one to consult. In a VPN or multi-uplink environment, if "the routing table is right but traffic doesn't go out," look at ip rule.