Distributed Consensus in System Designing

expert · System Designing

In a distributed system, network partitions will happen, servers will crash, and messages will be dropped. Despite these failures, machines must often agree on a single source of truth—such as who the current leader is, or whether a transaction was committed. Distributed Consensus is the protocol that allows a cluster of independent machines to agree on a state or a sequence of values safely, ensuring data consistency even when some nodes are offline. 1. The Core Rules of Consensus To be considered mathematically correct, a consensus algorithm must satisfy three foundational properties: Agreement: All non-faulty nodes must decide on the exact same value. Validity (Integrity): The value that is agreed upon must have been proposed by one of the nodes in the system (the system cannot invent data out of thin air). Termination (Liveness): Every non-faulty node must eventually reach a decision; the system cannot get stuck in an infinite loop forever. 2. Paxos vs. Raft While there are several consensus frameworks, Paxos and Raft are the two dominant protocols used to manage state machine replication. A. Paxos — The Classic Proposed by Leslie Lamport, Paxos is the foundational mathematical standard for distributed consensus. It breaks nodes into roles: Proposers , Acceptors , and Learners , navigating a multi-phase voting process to achieve agreement. The Problem: Paxos is notoriously abstract, difficult to conceptualize, and highly complex to implement in actual production software. Building a bug-free implementation of Multi-Paxos is considered one of the hardest engineering tasks in systems architecture. B. Raft — Deconstructed for Understandability Introduced by Diego Ongaro and John Ousterhout, Raft was designed specifically as an alternative to Paxos with a primary goal: Understandability . It achieves consensus by decomposing the problem into two distinct, highly managed phases: Leader Election and Log Replication . 3. How Raft Works Under the Hood In a Raft cluster (typically consisting of 3 or 5 nodes), a node can exist in one of three states: Follower , Candidate , or Leader . Step 1: Leader Election All nodes start as Followers . They expect to receive a continuous "heartbeat" from a Leader. If a follower stops hearing the heartbeat because the leader died, its internal Election Timeout expires. The follower transitions to a Candidate , increments the cluster's evolutionary timeline (called a Term ), votes for itself, and fires off "RequestVote" messages to all other nodes. If a candidate receives votes from a majority of nodes ( $N/2 + 1$ ), it is officially elected as the new Leader and begins broadcasting heartbeats to assert dominance. The Secret Weapon: Randomized Timeouts To prevent a split-vote deadlock (where two followers become candidates at the exact same millisecond and vote for themselves), Raft randomizes election timeouts (e.g., each node picks a random timeout between 150ms and 300ms). One node will naturally time out first, initiate the vote, and claim the leadership seat cleanly. Step 2: Log Replication Once a leader is established, all client requests must go directly through it. A client sends a command to the Leader (e.g., SET x = 10 ). The leader appends this command to its local, uncommitted log file. The leader broadcasts an AppendEntries message containing the log entry to all Follower nodes. Once a majority of followers successfully write the entry to their local disks and acknowledge it back to the leader, the entry is considered Committed . The leader officially applies the change to its local state machine, returns a success code to the client, and notifies the followers to commit the entry during the next heartbeat. 4. Real-World Implementations Distributed consensus is too complex to build from scratch for standard applications. Instead, systems integrate specialized, battle-tested consensus engines: etcd (Uses Raft): A strictly consistent, distributed key-value store that acts as the backbone brain of Kubernetes , tracking cluster state, secrets, and configuration details. Apache ZooKeeper (Uses Zab — a Paxos variant): A centralized service for maintaining configuration information, naming, and providing distributed synchronization across massive big-data ecosystems (like Hadoop and Kafka clusters). CockroachDB (Uses Raft + Multi-Raft): A next-generation, globally distributed SQL database that breaks tables into small shards and runs independent Raft consensus loops across those shards to ensure ACID compliance across global data centers. 5. The CAP Connection By design, distributed consensus engines choose Consistency (C) over Availability (A) under the CAP Theorem. If a 5-node cluster suffers a network partition that cuts it perfectly into a group of 2 nodes and a group of 3 nodes: The group of 2 cannot form a majority (

Careeroza — One-stop Zone for Aspirants

Study material, Careeroza mentorship, tech jobs, and career guidance on careeroza.com.

Public study materials

< 3$ ). It will refuse to elect a leader and will block all incoming write requests. The group of 3 can form a majority (

\geq 3$ ). It will operate normally. If the network breaks further and a majority can no longer be reached anywhere, the entire consensus layer locks down completely, preferring to reject writes rather than risk corrupting or fracturing your data truth.