Streaming Systems in System Architecture

Data Motion: Streaming vs. Batch Architectures In modern distributed systems, data processing is divided into two fundamental paradigms: Batch , where we process large "chunks" of historical data, and Streaming , where we process individual events in real-time. Choosing between these dictates your system's latency, cost, and complexity. 1. Batch Processing Batch Processing involves collecting data over a period (e.g., hours or days) and processing it all at once in a single large job. Logic: It prioritizes throughput over latency. Systems like Apache Spark or Hadoop MapReduce are optimized to crunch terabytes of data efficiently. The Benefit: High cost-efficiency and fault tolerance. If a batch job fails, you simply restart it from the beginning. Best Use Case: Generating end-of-day financial reports, training large machine learning models, or data warehousing ETL. 2. Stream Processing Stream Processing handles data continuously as it is generated, one event at a time, with minimal latency. Logic: Instead of waiting for a batch, events flow through a pipeline (e.g., Apache Kafka to Apache Flink ). The Benefit: Immediate action. You can detect fraud in a transaction the moment it happens rather than the next day. Best Use Case: Real-time fraud detection, live user activity monitoring, and dynamic pricing adjustments. 3. Live Streaming Architecture Live video streaming is a massive challenge in scalability because it requires moving high-bandwidth data to millions of users simultaneously. Ingestion: Video is captured and pushed to a media server via protocols like RTMP (Real-Time Messaging Protocol). Transcoding: The raw stream is converted into multiple resolutions (4K, 1080p, 720p, etc.) so users with slow internet can still watch. Distribution (CDN): The transcoded video is pushed to a Content Delivery Network (CDN) . CDNs have edge servers globally that cache the video segments near the end user, preventing the "bottleneck" of everyone downloading from a single origin server. 4. Multiplayer Game Architecture Multiplayer games are arguably the most demanding real-time systems, requiring ultra-low latency, strict synchronization, and massive concurrency. Authoritative Server: To prevent cheating, the server acts as the "source of truth." Clients send inputs (e.g., "move left"), and the server calculates the result and broadcasts the new state to all players. Client-Side Prediction & Interpolation: To hide network latency (lag), the client predicts movement locally. If the server disagrees, the client "snaps" to the correct position (reconciliation). Interest Management: Servers only send data about nearby objects to a player to reduce bandwidth (e.g., you don't need to know where a player is if they are on the other side of the game map). Processing Paradigms Matrix Feature Batch Processing Stream Processing Data Latency High (Hours/Days) Low (Milliseconds/Seconds) Data Scope Fixed, historical sets Continuous, infinite streams Complexity Lower Higher (State management is hard) Primary Goal Deep analytics/Reporting Immediate reaction/Action