Node.js › File System (fs) Module

How would you read a large file efficiently using a stream rather than loading it entirely into memory with fs.readFile()?

Intermediate

Answer

fs.createReadStream() reads a file incrementally in chunks, emitting 'data' events (or usable via .pipe() or async iteration) as each chunk becomes available, rather than buffering the entire file's contents in memory at once -- essential for processing files larger than available memory, or simply for reducing memory pressure and improving responsiveness when handling large files like log files or video uploads.

const fs = require('node:fs');

const readStream = fs.createReadStream('huge-log-file.txt', { encoding: 'utf-8', highWaterMark: 64 * 1024 });

for await (const chunk of readStream) {
  processChunk(chunk); // processes incrementally, without loading the whole file into memory
}

Real-world example A log-analysis tool that previously used fs.readFile() and crashed with an out-of-memory error on multi-gigabyte log files is rewritten to use fs.createReadStream(), processing the file incrementally in manageable chunks and successfully handling files of any size regardless of available memory.

Common follow-ups: What does the highWaterMark option control, and how does it affect the tradeoff between memory usage and the number of 'data' events emitted?;How would you combine a read stream with a Transform stream to process and write output incrementally in a single pipeline?

Streams & Buffers;Performance Optimization & Profiling