When the leader dies: failover, split brain and two ATMs

01Twenty million from a ten-million account

Here is the scene that made me stop reading chapter 5 of Designing Data-Intensive Applications (first edition) and stare at the ceiling. An account holds 10 million dong. The cable between the Hanoi and Saigon data centres is cut. A leader in Hanoi approves a 10-million withdrawal at an ATM; a leader in Saigon, who also believes it is in charge, approves another 10-million withdrawal at another ATM. Twenty million leaves an account that only ever had ten. When the cable is mended, there is no sane way to reconcile the two balances.

When I first tried to describe this in my own words I wrote something clumsy: "the network splits in two, only one side takes the write, but the followers are following the other side, so the transfer is recorded and the goods never arrive." It was not precise, but the feeling was right, and it is the feeling I want to keep for the whole post. The thesis, in one line: automatic failover is a double-edged knife; some big outages were caused by the failover itself, not by the leader dying. And the way out is not cleverness, it is a majority vote. If you have met the single-machine version of this bug, two transactions overwriting each other, it is the last-ticket post; here the two writers are not threads but whole machines that cannot hear each other.

02Why followers exist, and what they cannot do

Two words get used interchangeably and they should not be. Replication keeps a copy of the whole dataset on several machines: it survives a dead machine and spreads read traffic. Partitioning (sharding, chapter 6) gives each machine a different slice of the data: it spreads write traffic and holds data bigger than one disk. Real systems usually do both: every shard is itself replicated. This post is about the first one, and specifically the most common flavour, one leader and several followers.

Here is the part I got wrong for years. Adding followers feels like adding capacity, so surely it helps writes too? It does not. A follower has to swallow 100% of the writes in order to stay in sync; every write the leader accepts is replayed on every follower. Ten followers means ten machines each doing the full write workload, plus the leader. Followers buy you availability and read throughput, never write throughput. If the writes are the bottleneck, you partition.

write load each follower still carries

1×

steps in a failover

3

votes needed in a 5-node cluster

3 / 5

the leaderless rule of thumb

W+R>N

03Sync, async, semi-sync: what did the leader promise?

Before we watch a leader die, ask a simpler question: when it tells a client "committed", how many machines actually hold that write? The answer is a dial, and every setting is a different promise.

On this one my instinct was right the first time: a bank wants a synchronous guarantee (in practice, semi-sync), a social network is happy with async. Nobody cares if a like disappears; everybody cares if a transfer does. In PostgreSQL this dial is a couple of lines:

# semi-sync: COMMIT returns once ANY 1 of these two standbys has flushed the WAL
# (standbys not listed here stay asynchronous)
synchronous_standby_names = 'ANY 1 (replica_a, replica_b)'
synchronous_commit        = on     # on = wait for that standby to flush the WAL to disk
# per transaction, for low-value writes: SET LOCAL synchronous_commit = local;

Notice ANY 1: whichever standby answers first satisfies the promise, so a single slow replica does not freeze the primary. (PostgreSQL calls this quorum commit.) The price is paid in latency on every commit, one round trip to a follower, which is why you do not make everything synchronous by default. And that extra round trip lands right in the slow tail of your response times, the subject of the percentiles post.

04The three steps of failover

My picture for this is a company whose CEO collapses in the middle of a quarter. The board has to do three things, in order, under pressure:

1. Detect that the CEO is gone. There is no oracle. The only tool is a timeout: if nobody has heard from the leader for N seconds, assume it is dead.
2. Elect a new CEO. Usually the follower with the most up-to-date data, because every write it is missing is a write that disappears.
3. Redirect everyone. Clients, and the other followers, must send writes to, and replicate from, the new leader. The old one must be forced to stand down if it ever comes back.

Which of these three steps is the dangerous one? All of them, as it turns out. Each step hides a decision that humans used to make at 3 a.m. with a coffee. Automating them removes the human, and with it the human's ability to say "wait, that looks wrong".

05Three disasters

1. Lost writes, and the orphan who comes back

With asynchronous replication, the CEO's last few decisions were never written down anywhere else. The board elects a successor who never heard them. Worse, the old CEO may recover and walk back into the building holding those "orphan" decisions. The cluster now has to throw them away, silently, or reconcile them by hand.

2. Split brain

The Hanoi and Saigon branches lose contact. Each side gives up on the CEO and elects its own, and now two CEOs issue contradictory orders. This is two nodes both believing they are the leader and both accepting writes. Nothing here requires a bug: a perfectly correct failover script, run on each side of a network cut, produces exactly this. It is the two-ATM story from the top of the post, and without a way to decide who wins, the data diverges for good.

3. The wrong timeout

The CEO went to the bathroom for five minutes and the board has already elected a replacement. A leader that is only slow (a long garbage collection, a busy disk, a congested link) looks identical to a dead one from outside. Declare it dead too early and you trigger a false alarm: an unnecessary failover that itself loads the cluster, and loaded clusters are slower, which causes more false alarms. Declare it dead too late and the system sits frozen, with writes failing, for a long time. More on this dial below.

06Quorum: majority, or silence

The fix is a rule so simple it sounds like a joke: a node may be leader only if more than half of the nodes in the cluster agree. That is a quorum. For five nodes the quorum is ⌊5/2⌋ + 1 = 3 votes. Now cut the network into a group of three and a group of two.

The mathematics is one line: any two groups that each hold more than half of the nodes must share at least one node, and a node votes only once per election. So two different leaders can never both win in the same term. Side B is not broken; it is honest about being in the minority, and it pays for that honesty with unavailability. This is a deliberate choice: when the network splits, a quorum system sacrifices availability on the small side to protect consistency. It also explains why two data centres are not enough for the Hanoi and Saigon bank: cut the link and neither side holds a majority, so a third vote has to live somewhere else, in a third site.

A vote alone does not stop an old leader that never noticed it was replaced, say one frozen by a long GC pause. So production systems add two more tools. Term numbers (Raft calls them terms, other systems call them epochs): every election bumps a counter, and any message stamped with an older term is simply ignored; chapter 9 is where this becomes the Raft and Paxos story. And fencing: a token that increases with each leadership, checked by the storage layer itself, so a deposed leader's writes are rejected even if it still thinks it is in charge. The blunt version is STONITH ("shoot the other node in the head"), which cuts the old leader's power or network before the new one starts. It sounds brutal and it is exactly how the old leader is made to stop.

# sentinel.conf, on each of 3 sentinels
# quorum 2: two sentinels must agree the master is down; a failover then also
# needs a majority of all sentinels to authorise the one that runs it
sentinel monitor mymaster 10.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000    # the timeout: 5 s of silence = suspect
sentinel failover-timeout mymaster 60000

# redis.conf, on the master: refuse writes when too few replicas are keeping up
min-replicas-to-write 1     # need at least 1 replica connected...
min-replicas-max-lag 10     # ...whose last ack is no older than 10 seconds

Look at the last two lines: a cousin of the quorum idea, shrunk to one node. A master with no replica that has acknowledged within the last 10 seconds stops accepting writes, choosing silence over becoming one of the two ATMs. It is a bound, not a cure: replication stays asynchronous, so an isolated master can still accept up to roughly 10 seconds of writes that will become orphans.

07The timeout dial

Is a 5-second timeout or a 30-second one better? I caught myself reaching for a number, then realised the honest answer is neither, they are two different costs. At 5 seconds a long GC pause or a hiccup in the cloud network is enough to evict a healthy leader, and the unnecessary failover loads the cluster further. At 30 seconds a real crash means half a minute of failed writes, and a bigger window of unreplicated data in async setups.

There is no value that is right everywhere; there is only the one that matches your business. A payments service would rather freeze for thirty seconds than approve twice, a feed would rather fail over fast and lose a few posts. The pragmatic habit: set the timeout from measured numbers (your p99.9 pause plus margin, see the percentiles post), page a human whenever a failover happens instead of discovering it later in the logs, and rehearse the failover in a quiet hour, not for the first time at 3 a.m.

08Leaderless: W + R > N

One way to dodge the whole problem is to not have a leader. In Dynamo-style stores such as Cassandra, a write goes to W of the N replicas and a read asks R of them. The rule: if W + R > N, the set of replicas you wrote to and the set you read from must overlap in at least one node, so a read always touches a copy of the latest write. The same pigeonhole logic as the quorum.

09Read-your-own-writes

One last ghost lives in even a healthy cluster. You change your display name; the write goes to the leader; the page reloads and reads from a follower that is two seconds behind; your old name is back. You assume the save failed and hit it again. This is a violation of read-your-own-writes, and the common fix is as unglamorous as it sounds: read the data a user may have just changed from the leader, and everything else from followers. Variants: read from the leader for a minute after any write by that user, or remember the timestamp of their last write and only use a follower that has caught up to it.

10Closing: the mantras

A leader is not whoever believes they are one. A leader is whoever a majority agrees on.

Duy Anh, after chapter 5

• Followers give you availability and read capacity, never write capacity. To scale writes, partition.
• Pick sync, async or semi-sync by asking what you can afford to lose.
• Failover is three steps (detect, elect, redirect) and each one can fail. Automatic failover is a double-edged knife.
• Cure split brain with a majority: the minority side must go quiet. Add term numbers and fencing so a deposed leader cannot do damage.
• Leaderless: W + R > N makes reads overlap writes, but it is a dial, not a proof.

Back to the Hanoi and Saigon ATMs. With a third site holding the deciding vote, one of them goes dark for a few minutes and apologises to the customer; the other pays out ten million, once. A slightly unavailable bank is an inconvenience; a bank that paid twice is a disaster. A majority vote is only the opening move, though: how do machines that can crash and fall silent actually agree on anything? That is consensus, which chapter 9 answers properly and the reason Raft and Paxos exist.

Before consensus, though, comes a plainer way to hurt a database: ask it the wrong question. Next up in this series is one missing index and the road to deadlock.

Written by Vũ Duy Anh, backend engineer in Hà Nội. Questions or corrections are welcome by email.