If every timer is identical, nobody wins

One-line summary

Consensus is not about deciding "who is right"; it is about deciding "whose turn it is to speak." Raft solves this with only two mechanisms: terms and majority votes. And for an election to finish, the timers must differ from one another.

Why this was needed

Suppose you store the same value on three servers. If a client sends writes to each of the three separately, network delay changes the arrival order and the records on the three servers diverge. So the practical answer is always the same: designate one place that accepts writes, and have the rest follow that one. The problem is when "that one" dies. With nobody telling the remaining servers who is next, they have to decide it among themselves.

The natural question is whether a human could just step in. For a long time that is exactly what was done, and many database replication setups today still leave promotion to a person. But human intervention takes minutes, and the service cannot accept writes in the meantime. Worse, humans make mistakes too: if the original leader was actually alive and only its network was cut, the moment you promote a new one there are two places accepting writes.

The Raft paper (Diego Ongaro, John Ousterhout) set out to solve this problem in a way that is understandable. The official site describes Raft as "a consensus algorithm designed to be easy to understand, equivalent to Paxos in fault tolerance and performance" (raft.github.io). The short version of the paper received the Best Paper award at USENIX ATC in 2014.

How it works

Time in Raft flows by terms. A term is an integer that only ever goes up by 1, and there is at most one leader per term. If there is no leader, the term goes up by one and a new election opens. Every message carries the sender's term, so when two nodes meet, the rule is simple: the side that sees the larger term raises its own term to that value and steps down to follower. A request with a smaller term is rejected, but the receiver does not lower its own term to match it.

An election finishes in three lines.

1. 리더 소식이 타임아웃만큼 끊기면 → 임기 +1, 후보가 되어 자기에게 한 표
2. 나머지에게 투표를 요청 → 받는 쪽은 '한 임기에 한 표' 만 준다
3. 과반을 받으면 리더. 과반은 절반 이상이 아니라 절반 초과다

The majority condition does exactly one thing: two majorities always overlap. However you pick two sets of 3 out of 5 nodes, at least one node falls in both. Because that node gives only one vote per term, two leaders in the same term can never happen. raft.github.io states this property as "the cluster makes progress as long as any majority of servers are up; for example, a 5-server cluster continues to operate even if 2 servers fail."

What remains is the timer. If the timeouts of the three nodes are identical, all three become candidates at the same moment, each gives its own vote to itself and none to the others. Nobody reaches a majority and only the term keeps rising. So Raft picks the timeout randomly within a fixed range. If the one that wakes up first collects votes before the others wake up, the election finishes in one round.

It is worth being clear about one thing here. An election is not a process of choosing "which node is better." It looks at neither performance, nor resources, nor recent load. It looks at only two things: who woke up first and (once a log exists) whose log is not behind. This simplicity is what makes the algorithm understandable, and at the same time it makes the answer to "why did that particular node become leader" anticlimactic. The answer is usually "because its timer fired first."

The leader also does less than you might expect. It receives values, passes them on to the others, and keeps sending a signal that it is alive. That signal is the heartbeat, and it is sent continuously even when there is nothing to say, because that is the only way to tell silence from death. A follower resets its timer to the beginning every time a heartbeat arrives. So the condition for an election to open again is not "when the leader dies" but "when a heartbeat fails to arrive within the timeout," and the difference between the two explains most operational incidents.

What it looks like in practice

In operations, this story usually shows up as a "term keeps rising" warning. The etcd documentation says that when a heartbeat does not arrive in time even though the leader is alive, a "spurious election" occurs, and it advises tuning heartbeat-interval and election-timeout in high-latency environments (etcd FAQ). When the disk slows down and writes back up, or a long GC pause occurs, the node is perfectly healthy but only its heartbeat is late, so elections repeat. Meanwhile the cluster cannot accept writes.

There is an incident in the opposite direction too. If you set the timeout too long, the cluster stays stopped that much longer when the leader really dies. The election timeout is a knob between "how quickly do we notice" and "how often are we needlessly startled," and both directions look like an outage.

What you will do in the next lab

You start three processes that differ only in port and implement the election yourself. The wiring, such as the HTTP server and timers, is provided; you write only the functions that make decisions: comparing terms, voting, and judging a majority. In the last step, you set the timeouts of all three nodes to the same value and watch for 9 seconds. You will see that no leader is elected and only the term rises.