Streaming Systems

Data Motion: Streaming vs. Batch Architectures

In modern distributed systems, data processing is divided into two fundamental paradigms: Batch, where we process large "chunks" of historical data, and Streaming, where we process individual events in real-time. Choosing between these dictates your system's latency, cost, and complexity.

1. Batch Processing

Batch Processing involves collecting data over a period (e.g., hours or days) and processing it all at once in a single large job.

• Logic: It prioritizes throughput over latency. Systems like Apache Spark or Hadoop MapReduce are optimized to crunch terabytes of data efficiently.

• The Benefit: High cost-efficiency and fault tolerance. If a batch job fails, you simply restart it from the beginning.

• Best Use Case: Generating end-of-day financial reports, training large machine learning models, or data warehousing ETL.

2. Stream Processing

Stream Processing handles data continuously as it is generated, one event at a time, with minimal latency.

• Logic: Instead of waiting for a batch, events flow through a pipeline (e.g., Apache Kafka to Apache Flink).

• The Benefit: Immediate action. You can detect fraud in a transaction the moment it happens rather than the next day.

• Best Use Case: Real-time fraud detection, live user activity monitoring, and dynamic pricing adjustments.

3. Live Streaming Architecture

Live video streaming is a massive challenge in scalability because it requires moving high-bandwidth data to millions of users simultaneously.

• Ingestion: Video is captured and pushed to a media server via protocols like RTMP (Real-Time Messaging Protocol).

• Transcoding: The raw stream is converted into multiple resolutions (4K, 1080p, 720p, etc.) so users with slow internet can still watch.

• Distribution (CDN): The transcoded video is pushed to a Content Delivery Network (CDN). CDNs have edge servers globally that cache the video segments near the end user, preventing the "bottleneck" of everyone downloading from a single origin server.

4. Multiplayer Game Architecture

Multiplayer games are arguably the most demanding real-time systems, requiring ultra-low latency, strict synchronization, and massive concurrency.

• Authoritative Server: To prevent cheating, the server acts as the "source of truth." Clients send inputs (e.g., "move left"), and the server calculates the result and broadcasts the new state to all players.

• Client-Side Prediction & Interpolation: To hide network latency (lag), the client predicts movement locally. If the server disagrees, the client "snaps" to the correct position (reconciliation).

• Interest Management: Servers only send data about nearby objects to a player to reduce bandwidth (e.g., you don't need to know where a player is if they are on the other side of the game map).

Processing Paradigms Matrix