Watching many sockets from one seat

In one line

A multiplexing call watches many sockets at once, but an answer of "ready" only means that reading now will not block.

Why this was needed

To keep one connection from blocking the rest, you have to gather the place to wait into one spot instead of giving each connection its own. Giving each connection a thread is also a method, but each thread has its own stack and a context switch cost, so if you aim for thousands of connections you hit a different kind of limit first. Multiplexing goes the opposite way. You keep one flow of execution and leave only the waiting to the kernel.

How it works

The program tells the kernel a list of sockets and what it wants to know about each one: whether there will be something to read, or room to write. The kernel puts the program to sleep until at least one of the conditions is met, and when it wakes the program, it returns only those whose conditions are met. So no matter how many connections there are, there is one place to wait.

Linux has several calls that do this job, and starting from the oldest, their limits differ. select(2) carries a warning on the first line of its document. In the glibc implementation, fd_set has a fixed size, so you can watch only descriptors numbered below FD_SETSIZE, which is 1024, and this limit will not change, so modern applications should use poll or epoll. epoll(7) keeps an interest list and a ready list separately inside the kernel, and is built to scale well even when the number of watched descriptors grows.

In Python you don't have to choose among these yourself. DefaultSelector in the selectors module picks the most efficient implementation on that platform. The concepts the program deals with reduce to three.

selector.register(sock, selectors.EVENT_READ, data)
for key, events in selector.select(timeout=1.0):
    if events & selectors.EVENT_READ:
        ...
selector.modify(sock, selectors.EVENT_READ | selectors.EVENT_WRITE, data)

The thing most often misunderstood here is what "ready" means. Read readiness does not mean there is data; it means trying to read now will not block. You are also woken with read readiness when the other side has closed the connection, and then recv returns 0 bytes. 0 bytes is not an empty message; it is the signal that nothing more will come. If you mix these two up, you get a program that keeps watching a closed connection and spins its loop without rest.

Going further, the BUGS section of select(2) says that for a descriptor reported as read-ready, a subsequent read can still block. The example situation is one where data arrived but is discarded because the checksum turned out not to match, and the document recommends using O_NONBLOCK for sockets that must not block. In short, multiplexing does not replace non-blocking; you use them together. The readiness notification decides when to try, and non-blocking keeps the program from stopping when the attempt goes wrong.

There are rules for what to do after waking, too. If a listening socket woke with read readiness, there is no guarantee that exactly one connection is waiting. Several connections may be in the queue at a single wake-up, so you have to keep calling accept and emptying it until it returns EAGAIN or EWOULDBLOCK. The same document also says ECONNABORTED can occur for a connection that has already been aborted. You must not give up the rest of the queue because of that one.

What it looks like in the field

The incident usually shows up as 100 percent CPU usage. The number of connections has not grown, yet the event loop spins without rest. Following the cause, it is usually one of two things. Either a closed connection was not removed from the interest list, or write readiness is still being watched even though there is nothing to send. Room to write is almost always available, so write readiness is almost always true, and then select returns immediately. The next module takes this problem on directly.

There is an incident in the opposite direction too. It is making another blocking call while handling one event. That is when you read a file synchronously, send a request to another service in a blocking way, or do heavy computation on the spot. A multiplexing server has only one flow of execution, so this single stop becomes a stop of every connection being watched. Changing the structure does not make blocking disappear; it makes blocking far more expensive.

What you will do in the next quiz

Tell apart what a readiness notification guarantees and what it does not. The quiz covers how to handle each of these situations: recv returning 0 on a socket woken by read readiness, three waiting connections when a listening socket wakes once, and write readiness that stays true.