Distributed System Concepts in System Architecture

Core Distributed System Concepts Building reliable distributed systems is fundamentally about managing state across a network where individual nodes can crash, network partitions can occur, and clocks can drift. These four concepts are the "primitives" used to solve those complex problems. 1. Distributed Consensus Distributed Consensus is the process by which a cluster of nodes agrees on a single data value or state, even if some nodes fail or the network is unreliable. It is the heart of any reliable distributed system. The Problem: In a system of 5 nodes, how do you decide which one should commit a transaction if no node is "in charge"? Common Algorithms: Paxos: The original, mathematically proven consensus protocol. Raft: Designed for understandability; it decomposes consensus into leader election and log replication. Production Use: Used by services like Etcd (the backbone of Kubernetes) to store cluster state and by Apache ZooKeeper for configuration management. 2. Leader Election In many distributed systems, it is inefficient or dangerous for all nodes to perform the same task simultaneously (e.g., you don't want two nodes both sending out the same email notification). Leader Election ensures that only one node (the Leader) performs the critical action at any given time. The Process: Nodes compete to acquire a "lock" or "lease." The winner becomes the Leader. The Leader sends periodic "heartbeats" to the other nodes to signal, "I am still alive." If the heartbeats stop, the nodes immediately trigger a new election. Production Use: Distributed job schedulers, stream processors (ensuring only one partition consumer is active), and coordination service. 3. Distributed Locks A Distributed Lock provides a mutual exclusion mechanism across different servers. It ensures that only one process can access a shared resource (like a specific database row or a file) at a time. Implementation: Usually built on top of consensus services like Redis (via the Redlock algorithm) or Zookeeper. The "Fencing" Problem: A common pitfall is the "Stop-the-world" GC pause . If a node acquires a lock, then hangs due to a garbage collection pause, the lock might expire and be given to another node. When the first node wakes up, it thinks it still has the lock. Fencing tokens (increasing numbers) are used to detect and reject these stale writes. 4. Clock Synchronization In a distributed system, every server has its own physical hardware clock. Due to environmental factors, these clocks "drift"—one server might be milliseconds ahead of another. The Problem: If Node A saves a file at 10:00:01 and Node B saves a file at 10:00:02 , but their clocks are out of sync, the system might incorrectly conclude that Node B's update happened before Node A's. Solutions: NTP (Network Time Protocol): Syncs clocks via the network, but still has a margin of error. Logical Clocks (Lamport Timestamps): Instead of wall-clock time, nodes use a counter to determine the causal ordering of events. Google Spanner (TrueTime): Uses specialized hardware (GPS and Atomic clocks) in data centers to provide an extremely tight bound on clock uncertainty, allowing for globally ordered transactions. Distributed Systems Primitive Matrix Concept Primary Goal Reliability Mechanism Consensus Agreement on state Raft / Paxos Leader Election Single source of authority Heartbeats / Leases Distributed Locks Mutual exclusion Fencing Tokens / Etcd/Redis Clock Sync Temporal ordering NTP / Logical Clocks / Atomic Clocks