CAP Theorem in System Designing

The CAP Theorem (also known as Brewer's Theorem) is a fundamental law of distributed systems. It states that any distributed data store can simultaneously provide at most two of three core guarantees: Consistency , Availability , and Partition Tolerance . In a real-world network, wires get cut, routers crash, and data centers lose connectivity. The CAP theorem provides the mathematical framework for how a system must behave when these network failures occur. 1. Decoding the Three Components (C, A, P) Consistency (C) Every read request receives the most recent write or an error. Think of it as Single-Copy Consistency . It guarantees that no matter which node a client connects to in the cluster, they will see the exact same data at the exact same millisecond. Availability (A) Every non-failing node returns a non-error response for every request—but without any guarantee that it contains the absolute most recent write. Essentially, the system is always up for reads and writes , even if some nodes are serving slightly stale data. Partition Tolerance (P) The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network between nodes. A network partition is a reality of distributed computing; nodes will inevitably lose the ability to talk to each other. 2. The Hard Truth: You Cannot Choose "CA" A common misconception is that you can pick any combination of the three: CP, AP, or CA. In reality, you must always choose Partition Tolerance (P). Network partitions are a physical certainty, not an architectural option. Therefore, the CAP theorem is better stated as: In the presence of a network partition, a system must choose either Consistency (C) or Availability (A). 3. The Architectural Choices Imagine a database split across two nodes (Node 1 and Node 2) due to a network partition. They can no longer communicate with each other. A client writes new data to Node 1. A. CP Systems (Consistency + Partition Tolerance) If the system prioritizes consistency, it knows that Node 2 cannot be updated right now because the network link is broken. To prevent Node 2 from serving old, incorrect data, the system will block or return an error for any subsequent read/write requests hitting Node 2. The Result: The data remains perfectly synchronized across operational nodes, but parts of the system become completely unavailable. Real-World Tools: HBase, Apache ZooKeeper, Google Spanner, MongoDB (in default configurations). Best Used For: Financial transactions, banking ledgers, and authentication systems where showing the wrong balance or old password is a critical failure. B. AP Systems (Availability + Partition Tolerance) If the system prioritizes availability, Node 1 accepts the write, and Node 2 cheerfully continues to accept read and write requests from users, despite being disconnected from Node 1. The Result: The system boasts 100% uptime, but users hitting Node 2 will read stale data until the network partition heals and background synchronization catches up ( Eventual Consistency ). Real-World Tools: Apache Cassandra, Amazon DynamoDB, Couchbase. Best Used For: Social media feeds (it's fine if a friend sees a tweet 5 seconds late), shopping carts, and real-time streaming analytics. 4. Beyond CAP: The PACELC Theorem The CAP theorem only describes what happens when there is a network partition . But partitions are rare; systems spend 99% of their time running normally. To describe the trade-offs during normal operations, engineers use the PACELC Theorem . It states: If there is a P artition, choose between A vailability or C onsistency; E lse (when the system is running normally), choose between L atency or C onsistency. PC/EC (e.g., MongoDB): In a partition, it chooses consistency. In normal operations, it still chooses consistency (making the client wait until all internal replicas confirm the write, which increases latency). PA/EL (e.g., DynamoDB/Cassandra): In a partition, it chooses availability. In normal operations, it chooses low latency (returning success to the client instantly, syncing replicas asynchronously in the background). 5. System Design Takeaway Modern enterprise systems rarely pick a single database for everything. Microservice architectures allow you to mix and match trade-offs per use case: Use a CP system (like Postgres or a strictly consistent NoSQL engine) to manage user accounts, billing, and order placement. Use an AP system (like Cassandra or Redis) to manage product search recommendations, tracking metrics, and notification histories.