Bind to 127.0.0.1 and Nobody Outside Can Reach You

In a nutshell

The question of whether a port is open is actually two questions: who is listening and on which address is it listening. Most accidents in which curl localhost works inside the server but does not work from outside are decided by the second question.

Why this was needed

You deployed the application. You log in to the server and type curl http://localhost:8080/health, and a 200 comes back. Yet the load balancer keeps reporting health check failures. You check the firewall, and 8080 is open.

If you type ss -ltn, the answer comes out.

LISTEN  0  5  127.0.0.1:8080  0.0.0.0:*

127.0.0.1:8080 - it is bound only to the loopback address. This socket accepts only connections that came from within the same host. The kernel does not deliver packets coming from outside at all. It is a binding problem, not a firewall problem.

This distinction is also security design. Management ports and debug endpoints are deliberately bound only to loopback to block external exposure at the root. Conversely, if you accidentally bind a service port to loopback, you get the situation above.

How it works

ss is the successor of netstat and is the standard on recent distributions. There are four commonly used combinations.

ss -ltn          # TCP 리스닝 소켓만, 이름 해석 없이
ss -ltnp         # + 어느 프로세스가 잡고 있는지 (root 필요)
ss -tan state established
ss -s            # 상태별 요약

Recv-Q and Send-Q on a listening socket mean different things. On an established connection they are, respectively, "received data not yet read" and "sent data not yet acknowledged", but on a listening socket, Recv-Q is the current length of the completed queue waiting for accept, and Send-Q is the backlog maximum. So if the Recv-Q of a listening socket stays full, it means the application cannot keep up with accept, and you need to adjust the number of workers or the backlog.

There are two things of practical importance among the socket states.

TIME-WAIT: remains on the side that closed the connection first. It is a safety device to prevent late-arriving packets from mixing into a new connection, and the kernel cleans it up by itself after a certain time. It does not consume an fd.
CLOSE-WAIT: a state in which the peer sent a FIN but our side has not called close(). The kernel waits indefinitely and has no timeout. If they pile up, it is an application bug.

It is common to confuse these two and touch parameters such as net.ipv4.tcp_tw_reuse, but CLOSE-WAIT is not solved by kernel parameters.

If you can read /proc/net/tcp, you can diagnose even in environments without tools. Addresses and ports are written in hexadecimal (8080 is 1F90), and the states are also hexadecimal codes (0A = LISTEN, 01 = ESTABLISHED, 06 = TIME_WAIT, 08 = CLOSE_WAIT).

What it looks like in the field

"Address already in use". Someone already holds the port. Checking the PID with ss -ltnp is the first move, and it is usually the case where the previous process did not fully exit. Before you blindly do kill -9 here, you must check what that process is.

A socket is also an fd. It is exactly as we saw in the previous course. As connections grow, fds grow too, and you may hit the fd limit first. If you readlink ls /proc/PID/fd, it shows up in the form socket:[12345].

Inside a container, the namespace is different. ss inside a container shows only the sockets of that namespace. It is normal for the ports visible on the host and the ports visible inside the container to differ.

Why can't I use the port when it remains

bind: Address already in use usually arises not because someone else is using the port but because the me that just died is still holding the spot.

TIME_WAIT is not a bug. The side that closed the connection first has no way to confirm whether the last ACK reached the peer. So the kernel holds that socket's 4-tuple for 2*MSL (60 seconds on Linux). It is a device to prevent packets of an old connection that arrive late from getting mixed into a connection newly made with the same 4-tuple. You can see how many have piled up with ss -tan state time-wait | wc -l.

SO_REUSEADDR and SO_REUSEPORT solve different problems. The former lets you bind again to an address in the TIME_WAIT state. It is used so you do not have to wait 60 seconds on redeployment, and most server frameworks turn it on by default. The latter makes several processes listen on the same port at the same time. The kernel distributes incoming connections by a 4-tuple hash, so you can add workers without a proxy in front. The two names are similar and easy to mix up, but if you solve the redeployment problem with SO_REUSEPORT, you end up in a state in which the old process and the new process are both alive and sharing connections.

There are two receive queues. When a SYN arrives, the kernel puts it in the SYN queue and sends a SYN+ACK. When the handshake finishes, it moves it to the accept queue. If the application's accept() is slow, the accept queue fills, and from then on the kernel silently drops completed connections. On the client side it looks as though the connection was established but there is no response, a symptom whose cause is hardest to find. For a listening socket, ss -tlnp shows Recv-Q/Send-Q, which mean, respectively, the current accept queue length and the queue limit. If the two numbers are equal, the queue is overflowing, and nstat -az TcpExtListenDrops is the evidence.

Source ports are finite. If one server makes connections only to one port of another server, the usable 4-tuples are as many as the number of source ports. net.ipv4.ip_local_port_range defaults to 32768–60999, so that is about 28,000, and if the 60-second TIME_WAIT overlaps with that, you hit a wall at about 470 connections per second. This is the real reason to use a connection pool. Widening the ports or turning on tcp_tw_reuse only eases the symptom, and reusing connections is the answer.

What you will do in the next lab

You will start two local HTTP servers bound to different addresses and confirm with ss how the binding scopes differ. You will hold one connection to observe ESTABLISHED, and find the hexadecimal port yourself in /proc/net/tcp. You will read the backlog value of a listening socket, and finally write a script that answers "who is holding this port".