Advanced filesystem in nodejs

When handling large files (like high-resolution videos or massive database dumps) in Node.js, the Stream API is the only way to process them without crashing your server. If you try to use fs.readFile() on a 4GB file, Node.js will attempt to load the entire 4GB into your RAM. Since the default memory limit for a Node.js process is often much lower (around 1.5GB to 4GB depending on the system), your application will throw an ERR_STRING_TOO_LONG or FATAL ERROR: Ineffective mark-compacts near heap limit . 1. The "Chunking" Concept Streams break the file into small chunks (usually 64KB by default). Instead of one massive block of data, you get a continuous flow of small pieces. Comparison: ReadFile vs. Streams    Feature  fs.readFile (Buffer)  fs.createReadStream (Stream)  Memory Usage    Equal to the size of the file.  Very low and constant (fixed chunk size).  Speed   Must wait for the whole file to be read.  Can start processing as soon as chunk 1 arrives.  Safety  High risk of "Out of Memory" crashes.  Safe for files of any size. 2. Reading a Large File Using createReadStream , you can listen to the data event to process each piece as it arrives. const fs = require('fs'); // Creating a read stream for a 5GB file const stream = fs.createReadStream('very-large-video.mp4'); stream.on('data', (chunk) => {     // This chunk is a Buffer. By default, it's 64KB.     console.log(`Received chunk of size: ${chunk.length} bytes`); }); stream.on('end', () => {     console.log('Finished reading the entire file.'); }); stream.on('error', (err) => {     console.error('An error occurred:', err.message); }); 3. Writing a Large File Similarly, you can use createWriteStream to write data to a file piece by piece. const writable = fs.createWriteStream('destination.zip'); // You can manually write chunks writable.write(someDataChunk); writable.end(); 4. The Efficient Way: pipeline If you are moving a large file from one place to another (e.g., from a file to a network response or from a file to a compressed version), use pipeline . It handles backpressure —meaning if the writing side is slower than the reading side, it tells the reader to "pause" so memory doesn't overflow. const { pipeline } = require('stream/promises'); const fs = require('fs'); const zlib = require('zlib'); async function compressLargeFile() {   try {     await pipeline(       fs.createReadStream('huge-data.log'),  // Source       zlib.createGzip(),                     // Transform (Compression)       fs.createWriteStream('huge-data.gz')   // Destination     );     console.log('Compression successful!');   } catch (err) {     console.error('Pipeline failed:', err);   } } 5. Real-World Use Cases Video Streaming: Sending video data to a browser chunk by chunk so the user can start watching immediately. Log Processing: Searching through gigabytes of logs for a specific error without loading the whole file. CSV Parsing: Using a stream to parse a million-row CSV file and insert records into a database like MongoDB.