How a Name Becomes an Address, and the Day the Address Changed

In one line

A network problem is reported as "it doesn't work", but in reality it ends in one of three segments: name resolution, routing, or sockets. A diagnosis that does not say which segment is not a diagnosis.

Why this exists

curl failed. Suspecting DNS straight away is only half right. On a Linux host, name resolution is not just DNS's job. First, the hosts: line of /etc/nsswitch.conf decides in what order and where to ask. It is usually files dns, so /etc/hosts answers before DNS. Only after that does /etc/resolv.conf decide which server to ask and which search domains to append.

This structure determines how to choose a diagnostic tool. getent hosts goes through this whole path, while dig looks only at DNS. If their results differ, the problem is not the DNS server but the stub resolver segment. This one line cuts five minutes off an outage meeting.

There are values to remember in /etc/resolv.conf. nameserver entries are used up to 3 only. options timeout defaults to 5 seconds and options attempts defaults to 2 tries. So if the first nameserver is dead, the user feels the default 5 seconds as is — the reason it is slow even though you set up redundancy. options ndots is the threshold for how many dots a name must contain to be queried first as an absolute name without appending search domains, and the default is 1. In a container environment, if this value is set high, several failed queries go out ahead of the real one just to resolve one external name.

How it works

The routing table is not a list scanned from top to bottom but a lookup where the longest prefix wins. If both 10.0.0.0/24 and 0.0.0.0/0 exist, a packet going to 10.0.0.5 always takes the former. The default route is the one with prefix length 0, that is, the route for "when nothing matches". In the ip route output, via is the next hop, dev is the outgoing device, and src is the value to use as the source address.

Socket state is itself diagnostic information.

State	Meaning	If you see a lot of it
LISTEN	A server waiting for connections	Normal
ESTABLISHED	A two-way connection is established	Normal. A problem if the count hits a limit
SYN-SENT	Sent a connection request and waiting for a response	The peer is absent or it was dropped along the way
TIME-WAIT	The side that closed actively is waiting for the last packet	Short connections are being opened and closed in bulk
CLOSE-WAIT	The peer closed but this side has not	A sign of an application bug

A lot of TIME-WAIT is usually normal. It is the state that remains on the side that closed the connection first, and it is a mechanism that keeps late-arriving packets from contaminating a new connection. In contrast, CLOSE-WAIT piling up means the code is not closing sockets, so its nature is completely different.

What it looks like in the field

The author's homelab cluster stopped responding entirely one day. The logs repeated only dial tcp 10.0.0.111:6443: connect: no route to host. The diagnosis finished in 5 minutes — ip -4 addr show showed that the control plane node's address had changed from .111 to .120, and the output had scope global dynamic attached. It had received a different address during the DHCP lease renewal.

The decisive evidence was in the certificate. The apiserver certificate's SAN list had only 10.0.0.111 and not .120. So etcd and kube-apiserver failed trying to bind to the nonexistent .111 and fell into a restart loop, while kube-scheduler and controller-manager bind to 127.0.0.1, so their processes stayed alive but could do nothing. One address changed and the whole seven-node platform stopped.

The fix to prevent recurrence was to pin a static address with netplan. You give the interface dhcp4: false and specify the address, default route, and nameservers directly. But that alone was not enough. Because cloud-init regenerates the network configuration at boot and overwrites it, the following line had to be added as well.

echo 'network: {config: disabled}' > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

On top of this, you also have to set a DHCP reservation for that MAC on the router side or remove that address from the pool. If you fix only the host, another device could lease the same address and collide. You have to do all three for it to be "fixed". After applying, the confirmation signal was that the dynamic marker disappeared from the ip addr output.

There is one more lesson. This cluster had controlPlaneEndpoint pinned to a specific node's physical IP, not a VIP or a DNS name. So even after later expanding the control plane to 3 nodes and establishing an etcd quorum, if that one node died, all clients could not find the door even though the other two were healthy. If you do not make the address indirect, redundancy raises only data availability and leaves access availability unchanged.

What you will do in the next lab

You write a netplan static address file and a file that disables cloud-init yourself. You put an entry in /etc/hosts and confirm with getent that that path is actually taken, and read /etc/resolv.conf and /etc/nsswitch.conf to make a summary. You start one listener, observe sockets with ss, and parse the default route from ip route. Finally, you design firewall rules as a document and confirm why the order of rules decides life and death. In this environment iptables and ping are blocked, so applying rules and checking reachability are handled conceptually.