Streams, Buffers & Data Processing
Streaming, backpressure, piping, binary data, compression, and efficient large-data processing.
Q18: What are Streams in Node.js and why are they used?
Streams are data-handling channels used to read or write data sequentially. They conserve memory by processing data in chunks rather than reading entire files into memory at once.
- Streams process large datasets in small chunks to reduce memory consumption.
- Four core types exist: Readable, Writable, Duplex, and Transform.
Which stream type allows modifying data as it is being read and written?
Q19: What is backpressure in Node.js streams?
Backpressure happens when data is emitted faster than a Writable stream can process it. Node.js manages this by buffering data and signaling the reader to pause until the drain event is emitted.
- Backpressure prevents high memory usage from unconsumed stream data.
- The
drainevent notifies readers to resume writing.
Which event signals that a stream buffer is empty and writing can resume?
Q20: What is stream piping in Node.js?
Piping (readable.pipe(writable)) connects the output of a Readable stream directly into the input of a Writable stream, automatically handling flow control and backpressure.
- Connects readable source output directly into writable destinations.
- Manages data flow and backpressure buffering automatically.
Which stream method connects a Readable stream output to a Writable destination?
Q21: Why should pipeline() from stream/promises be preferred over readable.pipe()?
While readable.pipe() routes data between streams, it fails to handle error forwarding across all stream stages automatically, often leaving streams open and causing memory or file descriptor leaks. The stream.pipeline() helper properly forwards errors, cleans up unclosed underlying file descriptors, and supports Promise syntax via stream/promises.
pipe()can leak file descriptors and stream handles when errors occur mid-stream.pipeline()cleans up destroyed streams safely and provides built-in Promise support.
What is the primary advantage of stream.pipeline() over the traditional stream.pipe() method?
Q22: What is the Buffer class in Node.js?
The Buffer class provides instances to store raw binary data outside the V8 heap in fixed-length memory allocations.
- Buffers store raw binary data in memory.
- Useful when working with file systems or network streams.
Where is memory allocated for Buffer instances in Node.js?
Q23: What are Zero-Copy transfers and how are they achieved with Buffer.allocUnsafe()?
Buffer.alloc() initializes memory by zero-filling the allocated space, preventing accidental exposure of sensitive lingering data. Buffer.allocUnsafe() allocates memory without zero-filling, making execution faster but introducing potential data leak security risks if uninitialized memory is read directly. Zero-copy operations avoid unnecessary memory copying by pointing multiple abstractions or operations to the exact same underlying memory reference using typed arrays (ArrayBuffer) or network stream descriptors.
Buffer.allocUnsafe()skips memory zero-filling for higher performance, but requires immediate overwriting to avoid data leaks.- Zero-copy mechanisms share existing memory references to eliminate data duplication overhead during I/O.
What security risk arises when using Buffer.allocUnsafe() without writing new data over it immediately?
Q24: Why should string_decoder be used instead of buffer.toString() when parsing multi-byte streams?
When streaming multi-byte UTF-8 encoded characters (e.g., emojis or non-Latin scripts), a data chunk boundary may split a multi-byte character sequence in half. Converting chunks using buffer.toString() outputs corrupt character symbols (``). The string_decoder module preserves incomplete multi-byte sequences at chunk boundaries until the next chunk arrives.
buffer.toString()corrupts multi-byte UTF-8 characters split across stream chunk boundaries.StringDecoderbuffers incomplete multi-byte sequences until remaining bytes arrive.
What issue occurs when using buffer.toString('utf8') on raw stream chunks containing split multi-byte characters?
Q25: How is stream compression implemented using the zlib module?
The zlib core module provides synchronous and asynchronous stream compression using Gzip, Deflate, or Brotli algorithms. Because zlib stream transforms (createGzip(), createBrotliCompress()) return Transform streams, they can be piped directly between file streams or HTTP response streams for low-memory data transfer.
zliboffers Transform streams for Gzip, Deflate, and Brotli compression.- Piping compression streams minimizes heap memory consumption during file or HTTP transfers.
Which stream class type do zlib compression methods inherit from in Node.js?
Q26: How does the readline module handle streaming line-by-line file input?
The readline module reads input streams line-by-line without buffering entire files into memory. By pairing readline.createInterface() with a readable file stream (fs.createReadStream()), applications process massive log files line-by-line via for await (const line of rl) iterations with minimal memory footprints.
readlineprocesses text input line-by-line efficiently.- Ideal for parsing large log files streamingly without loading full files into memory.
Which method constructs a readline interface attached to standard or file input streams?