Some jobs must be done by exactly one machine: running a nightly batch, compacting a table, consuming a partition, accepting writes.
Run three instances of the service and all three will try. You need them to agree on one.
The naive version, and why it is not enough
const gotIt = await redis.set('leader', myId, {NX: true, EX: 30});if (gotIt) becomeLeader();Atomic, and it looks correct. The expiry is there so a crashed leader does not hold the lock forever.
That expiry is also the bug.
Fencing tokens
The fix is to stop trusting the leader and start checking at the resource.
Every lock acquisition returns a monotonically increasing token. The leader includes it with every write, and storage rejects anything with a token lower than the highest it has seen.
- A A acquires the lock token 33
- — A pauses — a long GC
- B A’s lease expires; B acquires token 34
- B B writes token 34
- A A wakes, still believes it holds the lock, writes token 33
A's write is accepted and overwrites B's. Both nodes followed the lock protocol perfectly; the lock service was correct; the data is corrupt. A lease alone cannot prevent this, because a paused process cannot be told its lease expired.
Use something that already does this
Leader election is consensus, and consensus is hard. The correct move is almost always to delegate:
etcd, ZooKeeper, Consul. Purpose-built, Raft underneath, and they hand you a monotonic revision number that works as a fencing token.
Kubernetes leases. If you are already on Kubernetes, the
coordination.k8s.io/Lease API is etcd with a friendly interface, and the
client-go leaderelection package implements the loop correctly.
Your database. A row with a version column, updated conditionally, gives you both the lock and the fencing token in one place — and if that database is already the resource being protected, this is genuinely the simplest correct design.
Kafka consumer groups. Partition assignment is leader election, already solved, if your work is naturally partitioned.
Design so the leader matters less
The best systems make leadership cheap to lose.
Make the work idempotent. If running the batch twice is harmless, a double-election is a performance problem rather than a data problem. See idempotency — this is by far the highest-leverage mitigation.
Keep terms short and heartbeats frequent, so a dead leader is replaced in seconds.
Partition the work. One leader per shard rather than one leader for everything. A failure then affects a fraction of the system, and no single node is a throughput ceiling.
Handle losing it gracefully. A leader must check it still holds the lease before each significant action, and stop cleanly when it does not — the loop that never re-checks is the one that becomes a second leader.