Streams in nodejs

In Node.js, Streams are a fundamental concept for handling data. Instead of reading an entire file or dataset into memory at once, streams break the data into small pieces (chunks) and process them bit by bit. This is exactly how video streaming works—you don't wait for the entire 2GB movie to download before you start watching; you watch it chunk by chunk. 1. Why use Streams? Memory Efficiency: You can process files larger than your available RAM (e.g., a 10GB log file on a 4GB machine). Time Efficiency: You can start processing data as soon as the first chunk arrives, rather than waiting for the entire payload. 2. The Four Types of Streams A. Readable Streams Used for operations where data can be read from a source. Examples: fs.createReadStream() , process.stdin , or an HTTP request ( req ). Events: data (emitted when a chunk is available), end (emitted when no more data is left). const fs = require('fs'); const readable = fs.createReadStream('large-video.mp4'); readable.on('data', (chunk) => {   console.log(`Received ${chunk.length} bytes of data.`); }); B. Writable Streams Used for operations where data can be written to a destination. Examples: fs.createWriteStream() , process.stdout , or an HTTP response ( res ). Methods: write() to send data, end() to signal you are finished const writable = fs.createWriteStream('output.txt'); writable.write('Hello '); writable.write('World!'); writable.end(); C. Duplex Streams Streams that are both Readable and Writable (e.g., a network socket). You can send and receive data simultaneously. Example: A TCP socket connection ( net.Socket ). D. Transform Streams A special type of Duplex stream where the output is calculated based on the input. It modifies or transforms the data as it passes through. Example: zlib.createGzip() (compresses data as it flows) or crypto.createCipheriv() (encrypts data). 3. Piping: The Secret Sauce The most efficient way to use streams is by Piping . The .pipe() method takes a Readable stream and connects it to a Writable stream, automatically managing the flow of data so the destination isn't overwhelmed (a concept known as backpressure ). Example: Compressing a file const fs = require('fs'); const zlib = require('zlib'); // Built-in compression module const readStream = fs.createReadStream('input.txt'); const gzip = zlib.createGzip(); // Transform Stream const writeStream = fs.createWriteStream('input.txt.gz'); // Connect the streams: Read -> Compress -> Write readStream.pipe(gzip).pipe(writeStream); console.log("File compressed successfully."); 4. Buffers vs. Streams Buffer: A small "waiting room" in memory for a single chunk of data. Stream: The actual "conveyor belt" that moves these buffers from one place to another. In your projects (like Careeroza), you would use streams if you were implementing a feature to allow students to upload large study materials or video lectures. This ensures your server doesn't crash from high memory usage when multiple people upload at once.