Buffers, Streams, and Binary Data

Understand binary data, Buffers, readable and writable streams, piping, backpressure, and efficient data processing.

Understanding Buffers

A Buffer represents a fixed-size sequence of bytes. Buffers are important when working with files, network protocols, cryptography, images, compressed data, and other binary content.

Buffers are built on top of JavaScript typed-array capabilities but provide Node-specific APIs for binary data.

Creating and Reading a Buffer

const buffer = Buffer.from('Hello Node.js', 'utf8');

console.log(buffer);
console.log(buffer.toString('utf8'));

💡 Buffer.from() converts text into bytes, while toString() converts those bytes back into text.

Key Takeaways

Buffers represent binary byte sequences.
Buffers are commonly used by Node.js I/O APIs.
Always know the encoding when converting between bytes and strings.

Knowledge Check

What does a Node.js Buffer represent?

Readable and Writable Streams

Streams process data incrementally instead of requiring the entire dataset to be loaded into memory. Readable streams produce data, writable streams consume data, and duplex or transform streams can do both.

Streams are particularly valuable for large files, HTTP requests, HTTP responses, compression, and network communication.

Streaming a File

import { createReadStream } from 'node:fs';

const stream = createReadStream('large-file.txt', {
  encoding: 'utf8'
});

stream.on('data', (chunk) => {
  console.log('Received chunk:', chunk.length);
});

stream.on('end', () => {
  console.log('Finished');
});

💡 The file is delivered in chunks rather than loaded completely into memory.

Key Takeaways

Streams reduce memory usage for large datasets.
Readable streams produce chunks of data.
Writable streams consume chunks of data.

Knowledge Check

What is a major advantage of streams?

Piping and Backpressure

Backpressure occurs when a writable destination cannot consume data as quickly as a readable source produces it. Node.js streams provide mechanisms for controlling this flow.

The pipe() method connects streams and handles important flow-control behavior automatically. Modern applications can also use pipeline() from node:stream/promises to compose streams while handling errors more safely.

Prefer pipeline for Complex Stream Chains

pipeline() provides cleaner error handling and lifecycle management when connecting multiple streams.

Using pipeline

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';

await pipeline(
  createReadStream('input.txt'),
  createWriteStream('output.txt')
);

console.log('Copy completed');

💡 pipeline() connects the streams and provides Promise-based completion and error propagation.

Key Takeaways

Backpressure prevents fast producers from overwhelming slow consumers.
Use pipeline() when composing multiple streams.
Stream errors must be handled explicitly.

Knowledge Check

What does backpressure protect against?

Understanding Buffers

Readable and Writable Streams

Piping and Backpressure

STAY UPDATED WITH BYTES!