Big Data Systems in System Architecture

Big Data Systems: Processing Petabytes at Scale When your data exceeds the capacity of a single machine or a traditional relational database, you move into the realm of Big Data . These systems are designed to partition data across hundreds or thousands of nodes, processing them in parallel to turn raw noise into actionable intelligence. 1. Hadoop: The Foundation Apache Hadoop was the first framework to make Big Data accessible. It solved two fundamental problems: Storage (HDFS) and Compute (MapReduce) . HDFS (Hadoop Distributed File System): Splits massive files into "blocks" and distributes them across a cluster of commodity servers. It replicates these blocks (usually 3x) to ensure that if a server dies, no data is lost. The Philosophy: "Move the computation to the data. " Instead of moving gigabytes of data to a central processor, Hadoop sends the processing code to the server where that specific data block resides. 2. MapReduce: The Original Paradigm MapReduce is the programming model used by Hadoop to process large datasets. It consists of two main functions: Map: The input data is split into chunks, and each node processes its chunk to produce a set of key-value pairs. Reduce: The framework aggregates all the outputs from the Map phase that share the same key and summarizes or combines them into a final result. The Limitation: MapReduce is "disk-heavy. " After the Map phase, it writes results to the physical disk before the Reduce phase can start. This makes it very slow for complex, iterative processes like Machine Learning. 3. Apache Spark: The Evolution Apache Spark was created to replace MapReduce by moving the processing from the disk into In-Memory RAM. Speed: By caching data in memory, Spark can run jobs up to 100x faster than MapReduce. DAG (Directed Acyclic Graph): Unlike MapReduce's two-step rigid process, Spark builds a DAG of tasks, allowing it to chain multiple operations together in a single pipeline without writing to disk between every step. Use Case: Iterative algorithms (Machine Learning), real-time streaming, and interactive data analysis. 4. Search Ranking: The "Relevance" Engine Once you have indexed millions of documents, the final challenge is Ranking : deciding which documents are the "best" result for a user's query. TF-IDF (Term Frequency-Inverse Document Frequency): A classic algorithm that calculates relevance based on: TF: How often a word appears in a specific document. IDF: How rare that word is across the entire corpus. (If a word appears in every document, it’s not useful for ranking). Modern Ranking (Learning to Rank): Today's systems use Machine Learning models. The model analyzes hundreds of "features"—user location, click-through rates, document freshness, and user intent—to predict the probability that a user will click on a specific result. Big Data Reference Matrix Tool / Concept Core Function Primary Philosophy Hadoop (HDFS) Distributed Storage Reliability through redundancy. MapReduce Data Processing Disk-based, fault-tolerant parallel processing. Spark Data Processing In-memory, high-speed, DAG-based engine. Search Ranking Relevance Scoring ML-driven prediction of user intent.