Raft: The Consensus Algorithm Behind etcd and Kubernetes

⏱️ Lectura: 14 min

When an etcd node crashes mid-write and Kubernetes keeps running without corrupting its state, there’s a named algorithm working behind the scenes: Raft. Without it, the control plane of Kubernetes, CockroachDB, Consul, and Vault couldn’t guarantee that every node sees exactly the same data after a crash.

📑 En este artículo

1. TL;DR
2. What the Raft algorithm is and why it matters
3. How Raft works under the hood
   1. The three states of a node
   2. Log replication: how a write gets confirmed
   3. Safety: why the new leader never loses committed data
4. Practical examples: Raft in code
5. Getting started: setting up a 3-node etcd cluster
   1. Checking which node is the leader
6. Real-world use cases
7. Common mistakes and best practices
8. Comparison with alternatives
9. Going deeper: advanced details
10. Frequently Asked Questions
    1. Are Raft and Paxos compatible with each other?
    2. What’s the minimum number of nodes needed to tolerate a failure?
    3. What happens if the leader crashes right when it’s confirming a write?
    4. Is Raft useful for single-instance databases?
    5. Why etcd and not raw Raft for Kubernetes?
11. References

The Raft algorithm was published in 2014, when Diego Ongaro and John Ousterhout, from Stanford, presented a paper with an explicit goal: achieve the same fault-tolerance guarantees as Paxos, but in a way an engineer could understand without a PhD in distributed systems. That focus on understandability is why Raft, not Paxos, is the consensus protocol behind most new distributed databases today.

TL;DR

• You’ll understand why a cluster needs to elect a single leader before accepting writes.
• You’ll be able to distinguish the three states of a Raft node: follower, candidate, and leader.
• You’ll learn how a log entry gets replicated until the majority confirms it (commit).
• You’ll be able to set up a 3-node etcd cluster and check which node is the leader with etcdctl.
• You’ll be able to code a Raft node in Go using the hashicorp/raft library.
• You’ll understand the difference between Raft, Paxos, Zab, and 2PC, and when each one makes sense.
• You’ll be able to identify the most common mistakes when deploying Raft clusters in production.

What the Raft algorithm is and why it matters

Raft is a distributed consensus algorithm: a protocol that lets a group of machines (a cluster) agree on a sequence of values, even if some of those machines fail or the network partitions. Agreeing means something concrete: every surviving node ends up with the same operation log, in the same order, even if the current leader crashes mid-write.

This problem shows up in any system that needs replication with strong consistency: a distributed database that can’t afford to have two nodes believe they wrote different values for the same key, or a configuration store like etcd, where Kubernetes keeps the state of every pod, deployment, and secret in the cluster. If that store splits into two versions of the truth (known as split brain), the entire cluster ends up in an inconsistent state.

Before Raft, the reference algorithm was Paxos, described by Leslie Lamport in 1989. Paxos is correct and deeply studied, but its own community acknowledges it’s hard to implement without subtle bugs: the original paper is notorious for being nearly unreadable, and most production implementations diverge from the pure algorithm just to be reasoned about. Raft solves the same problem by breaking it into three independent subproblems: leader election, log replication, and safety, each with simple rules to follow.

How Raft works under the hood

The three states of a node

Every node in a Raft cluster is always in one of three states: follower, candidate, or leader. On startup, every node begins as a follower. A follower is passive: it only responds to requests sent by the leader or a candidate, never initiating anything on its own.

If a follower doesn’t hear from the leader for a while (the election timeout, randomized per node), it assumes the leader died or became unreachable and turns into a candidate: it increments its term (an epoch counter that never goes backward) and requests votes from the rest of the cluster.

stateDiagram-v2
    [*] --> Follower
    Follower --> Candidate: election timeout
    Candidate --> Leader: majority of votes
    Candidate --> Follower: discovers leader with higher term
    Candidate --> Candidate: split vote, new round
    Leader --> Follower: detects higher term in response

A candidate wins the election if it gets votes from the majority of nodes in the cluster, not the majority of those that respond: the majority of the total configured nodes. In a 5-node cluster, 3 votes are needed; in a 3-node cluster, 2. This majority requirement is what makes Raft fault-tolerant: as long as the majority of nodes are alive and able to communicate, the cluster can elect a leader and keep accepting writes.

The randomized timeout trick prevents two followers from becoming candidates at exactly the same time and tying the vote (split vote) over and over. If a tie happens, each candidate waits for a new random timeout and tries again.

Log replication: how a write gets confirmed

Once there’s a leader, every write in the cluster goes through it. The client sends the operation to the leader, the leader appends it to its local log as a new (not yet confirmed) entry, and sends it in parallel to all followers via AppendEntries messages.

sequenceDiagram
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    L->>F1: AppendEntries(log)
    L->>F2: AppendEntries(log)
    F1-->>L: ack
    F2-->>L: ack
    Note over L,F2: majority confirmed, entry committed

When the majority of nodes (leader included) have confirmed writing that entry to their log, the leader marks it as committed and only then applies the operation to its state machine and responds to the client. Followers learn which entries are committed in the next AppendEntries (which also works as a heartbeat when there are no new writes) and apply that entry to their own state machine.

This sequence (propose, replicate, wait for majority, commit, apply) guarantees that no data is considered saved until it survives on more than half the cluster. If the leader crashes right after writing the entry but before the majority confirms it, that entry might be lost or might survive, but it never ends up in an ambiguous state: the newly elected leader already has, by construction, every entry committed up to that point.

Safety: why the new leader never loses committed data

Raft guarantees this with a simple election rule: a node only votes for a candidate if that candidate’s log is at least as up to date as its own. A candidate with a shorter or older log than the majority of the cluster can’t win an election. This means the elected leader always contains, at minimum, every entry that was already committed before the election.

Practical examples: Raft in code

To see how this translates into code, the simplest example is modeling the minimal state any Raft node needs: what state it’s in, what term it’s on, and who it voted for.

type NodeState int

const (
    Follower NodeState = iota
    Candidate
    Leader
)

type LogEntry struct {
    Term    int
    Command []byte
}

type RaftNode struct {
    state       NodeState
    currentTerm int
    votedFor    string
    log         []LogEntry
    commitIndex int
}

This snippet doesn’t implement anything yet: it just defines the shape of the data each node needs to persist. currentTerm and votedFor must be written to disk before responding to any vote or AppendEntries, because if the node restarts and forgets them, it could vote twice in the same term and break the single-majority guarantee.

In practice almost nobody implements Raft from scratch: you use a proven library. The most widely used one in Go is hashicorp/raft, the same one running inside Consul and Vault.

import (
    "os"
    "time"

    "github.com/hashicorp/raft"
    raftboltdb "github.com/hashicorp/raft-boltdb"
)

func setupRaft(nodeID, dataDir string, fsm raft.FSM) (*raft.Raft, error) {
    config := raft.DefaultConfig()
    config.LocalID = raft.ServerID(nodeID)

    logStore, err := raftboltdb.NewBoltStore(dataDir + "/raft-log.db")
    if err != nil {
        return nil, err
    }

    snapshots, err := raft.NewFileSnapshotStore(dataDir, 2, os.Stderr)
    if err != nil {
        return nil, err
    }

    transport, err := raft.NewTCPTransport(
        "127.0.0.1:8300", nil, 3, 10*time.Second, os.Stderr,
    )
    if err != nil {
        return nil, err
    }

    return raft.NewRaft(config, fsm, logStore, logStore, snapshots, transport)
}

This code assembles the four pieces hashicorp/raft needs to run a node: a logStore to persist the log in BoltDB, a snapshots store to compact the old log, a TCP transport to talk to the rest of the cluster, and an fsm (Finite State Machine) that the developer implements themselves: the code that applies each committed entry to the application’s actual state.

Getting started: setting up a 3-node etcd cluster

The fastest way to see Raft in action without writing a line of Go is to spin up an etcd cluster, which exposes its leader state directly through its API.

# download the official binary
ETCD_VER=v3.5.17
curl -L "https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz" -o etcd.tar.gz
tar xzf etcd.tar.gz && cd "etcd-${ETCD_VER}-linux-amd64"

# node 1 of 3, static cluster on localhost
./etcd --name node1 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --listen-peer-urls http://127.0.0.1:2380 \
  --listen-client-urls http://127.0.0.1:2379 \
  --advertise-client-urls http://127.0.0.1:2379 \
  --initial-cluster node1=http://127.0.0.1:2380,node2=http://127.0.0.1:2381,node3=http://127.0.0.1:2382 \
  --initial-cluster-state new

You repeat the same command for node2 and node3, changing the peer ports (2381, 2382) and client ports. When all three start up, the exact election process described above kicks in: each begins as a follower, one of them wins the random timeout, requests votes, and becomes the leader.

⚠️ Heads up: a Raft cluster with an even number of nodes (4, 6) doesn’t gain extra fault tolerance over the odd number right below it (3, 5): it still needs the same absolute majority, but pays more network latency per replicated node. The standard recommendation is to always use an odd number.

flowchart TD
    A["Client"] --> B["Leader node (etcd)"]
    B --> C["Follower node 1"]
    B --> D["Follower node 2"]
    C --> E[("Replicated log")]
    D --> F[("Replicated log")]
    subgraph "Raft Cluster"
    B
    C
    D
    end

Checking which node is the leader

With the cluster up, etcdctl lets you confirm live which node won the election:

etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:2379,http://127.0.0.1:2379 \
  endpoint status --cluster --write-out=table

The table it returns includes an IS LEADER column: only one endpoint has it set to true. If you kill that node’s process and run the same command again against the two remaining nodes, you’ll see that shortly another node switches to true: that’s leader election happening in real time.

Real-world use cases

Raft isn’t an academic exercise: it underpins the state of systems running in production at large scale.

• etcd, the key-value store Kubernetes uses as the database for its control plane: every object (Pod, Deployment, Secret) you see with kubectl get lives in a Raft log replicated across etcd nodes.
• HashiCorp Consul and Vault, which use hashicorp/raft to replicate the service catalog and secrets across nodes.
• CockroachDB, which partitions its data into ranges and assigns an independent Raft group to each range, so different parts of the database can have different leaders in parallel.
• TiKV, the storage layer of TiDB, with a range and Raft group design very similar to CockroachDB’s.

Common mistakes and best practices

The most common mistake when deploying a Raft cluster in production is underestimating the impact of disk latency on the election timeout. Every vote and every AppendEntries needs to be persisted before responding (fsync), so if the disk is slow, the leader’s heartbeat can take longer than the followers’ timeout and trigger unnecessary elections constantly. etcd’s recommendation is to run on SSD and monitor the log’s fsync latency.

Another common mistake is confusing majority of configured nodes with majority of responding nodes: Raft always calculates the majority based on the total configured cluster size, not the nodes alive at that moment. If a 5-node cluster loses 3, it doesn’t matter that the remaining 2 agree with each other: they don’t form a majority, and the cluster stops accepting writes until quorum is restored. This is a deliberate design decision: prefer to stop writing rather than risk inconsistency.

💡 Tip: modern Raft implementations (including etcd’s) add a pre-vote phase before the actual election: a node that lost contact with the leader first checks whether it would win the election without incrementing its term yet, preventing a node isolated from the network (but still alive) from forcing unnecessary elections as soon as it reconnects.

Changing a cluster’s membership (adding or removing a node) live is another delicate point: doing it naively, by replacing the old configuration with the new one all at once, can create a window where two simultaneous majorities and two leaders exist at the same time. Raft solves this with a joint configuration mechanism (joint consensus), where the cluster briefly operates under the union of the old and new configuration before switching over completely.

Comparison with alternatives

Algorithm	Used in	Main advantage	Limitation
Raft	etcd, Consul, CockroachDB, TiKV	Designed to be understandable and unambiguously implementable	Requires a live majority; doesn’t work with clusters split down the middle
Paxos (multi-Paxos)	Google Chubby, variants in Spanner	Decades of formal study, very flexible in its variants	The original paper is hard to implement without subtle bugs
Zab (ZooKeeper Atomic Broadcast)	Apache ZooKeeper	Optimized for the single-leader pattern with ordered broadcast	Tightly coupled to ZooKeeper’s internal design, not very reusable elsewhere
Two-Phase Commit (2PC)	Classic distributed transactions	Simple to understand for a single transaction	Blocking: if the coordinator crashes, participants are left hanging

Going deeper: advanced details

A Raft log can’t grow forever: every node would take longer and longer to restart and replay the entire history. That’s why each node periodically generates a snapshot: a complete picture of its state machine’s state at a given point, and discards log entries before that point. A follower that’s fallen too far behind doesn’t receive the whole log from scratch: the leader sends it the most recent snapshot directly and continues from there with AppendEntries.

Another detail that separates a toy implementation from one ready for production is how it handles read-only reads. Returning a read directly from the leader’s local state without any additional check is risky: that node might have stopped being the real leader (say, due to a network partition) without knowing it yet. Serious implementations solve this with a heartbeat exchange with the majority before answering a read (read index), or with time-based lease reads, where the leader only trusts its own state during a short window after the last confirmed heartbeat.

📖 Summary on Telegram: View summary

Your next step: spin up the three etcd nodes from the command above, kill the process shown as leader, and confirm with etcdctl endpoint status --cluster --write-out=table that another node takes its place in under a second.

Frequently Asked Questions

Are Raft and Paxos compatible with each other?

Not directly: they’re different protocols with their own message formats and guarantees, even though both solve the same consensus problem and both tolerate minority failures. A cluster can’t mix nodes speaking Raft with nodes speaking Paxos.

What’s the minimum number of nodes needed to tolerate a failure?

Three. With 3 nodes, the majority is 2, so the cluster tolerates losing 1 node and keeps accepting writes. With 5 nodes it tolerates 2 simultaneous failures, at the cost of replicating every write to more replicas.

What happens if the leader crashes right when it’s confirming a write?

It depends on whether that entry already reached the majority before the crash. If it did, the new leader (which, by the voting rule described above, has that entry in its log) keeps it and applies it. If it didn’t, that write is lost, but the client never got confirmation that it was saved, so it can retry it.

Is Raft useful for single-instance databases?

It doesn’t make sense there: Raft exists to coordinate multiple copies of the same data. If there’s only one node, there’s nothing to replicate and no network failure to tolerate.

Why etcd and not raw Raft for Kubernetes?

etcd is a complete Raft implementation with a key-value API on top (watch, transactions, TTLs). Kubernetes doesn’t implement Raft itself: it uses etcd as a dependency and relies on the consistency guarantees etcd already handles internally.

References

• raft.github.io: the algorithm’s official site with an interactive visualization of leader election and log replication.
• etcd.io/docs: official etcd documentation, the Raft implementation Kubernetes uses.
• github.com/hashicorp/raft: Go implementation used by Consul and Vault, with FSM examples.
• Wikipedia: Raft (algorithm): historical summary and comparison with Paxos.

📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Imagen destacada: Foto de Kier in Sight Archives en Unsplash