TCP — How Connection and Reliability Are Built

In a nutshell

TCP creates the illusion of "a byte stream that is neither lost nor reordered" with sequence numbers, acknowledgments, retransmission, and two kinds of windows.

Why this was needed

IP does its best but promises nothing. Packets can vanish, be reordered, and be duplicated. Yet most applications are built on the assumption "you receive what was sent". The layer that fills that gap is TCP.

How it works

Connection establishment. The client sends a SYN, the server answers with a SYN-ACK, and the client sends an ACK. Three exchanges are needed because each side must tell the other its own initial sequence number and have it acknowledged.

Connection termination. Each direction is closed separately, so FIN and ACK are needed twice each, four exchanges in all. The side that closed first remains in the TIME-WAIT state for 2MSL (twice the maximum segment lifetime). This is so that late-arriving segments do not contaminate the next connection using the same port pair, and so that it can retransmit if the last ACK was lost. It is normal for tens of thousands of TIME-WAIT sockets to pile up on a heavily loaded server, and if you recklessly touch kernel parameters to get rid of them, you invite other problems.

Reliability. The receiving side reports, with an ACK, the position up to which it has received properly. The sending side retransmits if there is no acknowledgment within a certain time (RTO), and if the same ACK is duplicated three times, it retransmits immediately without waiting for the timeout (fast retransmit).

Two windows. Flow control is a mechanism by which the receiving side announces with the advertised receive window (rwnd) "I can only take this much". Congestion control is a mechanism by which the sending side estimates, with the congestion window (cwnd) that it maintains itself, "the network can only bear this much". The two have different purposes. The actual amount sent is limited to the smaller of the two values. Congestion control grows exponentially with slow start, grows linearly once it passes the threshold, and cuts back sharply when loss is detected (AIMD).

What it looks like in the field

The difference between two connection failure messages decides half of the diagnosis.

Connection refused means the peer returned an RST. It is evidence that the packet went to the destination and came back, so routing, security groups, and network policies have already been passed. The remaining candidates are inside the destination host. Either the process died, it attached to a different port, it bound not to 0.0.0.0 but only to 127.0.0.1, or there is not a single healthy backend behind the load balancer. Here we must correct a common misconception. The explanation "refused, so it's the firewall" is mostly wrong. Firewalls in practice almost always have a policy of silently dropping (DROP), and a dropped packet gets no response, so it shows up as a timeout.

Connection timed out means there was no response at all until the SYN retransmissions were exhausted. By default Linux retransmits tcp_syn_retries times and doubles the interval each time. Adding 1+2+4+8+16+32+64 gives 127 seconds. If the timeout setting in the application log is 30 seconds but it actually failed after 127 seconds, it is a sign that setting is not being applied.

There are also cases where a timeout occurs even though there is a listener. If the accept queue is full, the kernel silently drops new SYNs. If you look at the listening sockets with ss -ltn, Recv-Q is the current queue length and Send-Q is the maximum size. If Recv-Q exceeds Send-Q, the cause is not the network but the application's throughput.

What to check in the quiz that follows

Check whether you can explain what packet exchange each of refused and timeout results from, and what the difference between flow control and congestion control is.