Caching in System Designing
medium · System Designing
Caching is the process of storing copies of frequently accessed data in a high-speed data storage layer (typically RAM) so that future requests can be served much faster than fetching it from the primary source. In system design, caching is the ultimate tool for reducing latency, easing database load, and scaling your application to handle massive spikes in traffic. 1. The Levels of Caching A robust system architecture places caches at multiple points along the request-response lifecycle to optimize performance at every layer. A. Browser Cache (Client Side) How it works: The browser stores static assets (HTML, CSS, JS, images) directly on the user's local machine. Mechanism: Controlled via HTTP cache headers like Cache-Control and ETag sent by your backend. It prevents the client from making unnecessary network requests entirely. B. Content Delivery Network (CDN) Cache (Edge Side) How it works: CDNs (like Cloudflare or Akamai) are distributed networks of proxy servers positioned geographically close to your users. Mechanism: Instead of a user in London hitting your primary server in Virginia for a static image, the request hits the nearest London CDN edge node. C. In-Memory Cache (Application Side) How it works: Dedicated, ultra-fast data stores living entirely in RAM inside your backend infrastructure, like Redis or Memcached . Mechanism: Used to cache complex database query results, computed metrics, or session states. Reading from RAM takes microseconds (
0^{-6}$ seconds), whereas reading from a traditional disk-based database takes milliseconds (
0^{-3}$ seconds). 2. Cache Eviction Policies Memory (RAM) is expensive and limited. When a cache fills up, the system must decide which old data to throw away to make room for new data. LRU (Least Recently Used): Discards the data that hasn't been accessed for the longest period of time. This is the industry standard for most web applications. LFU (Least Frequently Used): Counts how many times an item is requested. It discards the data with the lowest hit count, regardless of when it was last accessed. TTL-Based (Time-To-Live): Data automatically expires and deletes itself after a predefined countdown (e.g., 3600 seconds). 3. Caching Strategies (Data Patterns) How your application interacts with the cache and database determines your system's data consistency. A. Cache-Aside (Lazy Loading) The application orchestrates the entire flow. It looks at the cache first. If there's a Cache Miss , it queries the database, writes the data to the cache, and returns it to the user. Pros: Cache only contains data that users are actively requesting. Cons: A cache miss results in three network hops, causing a slight delay for that specific user. B. Write-Through The application writes updates directly to the cache first, and the cache immediately writes it down to the primary database. Pros: Fast subsequent reads; the cache is always fresh. Cons: High write latency because you must write to two storage layers before completing the request. 4. The Hardest Problem: Cache Invalidation "There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton Cache Invalidation is the process of declaring cached data inaccurate or obsolete when the underlying primary data changes. If a user updates their profile name from "Alice" to "Bob", but your application cache still serves "Alice" for the next hour, your data is stale . Common Invalidation Techniques: Purge/Evict on Write: Whenever your application executes an update or delete query on the database, it explicitly deletes the corresponding key from Redis. The next read will force a cache miss and fetch the fresh data. Short TTLs: If absolute real-time accuracy isn't critical (like a view counter on a video), you can rely on a short TTL (e.g., 10 seconds). The data fixes itself automatically when the timer runs out.