ExtendedBuffer

ExtendedBuffer is a growable binary buffer built on top of Node.js Buffer. It keeps an internal read pointer (similar to a stream cursor) and supports appending data at the end or prepending data at the start.

Install

npm install extended-buffer

Browser usage (bundlers)

ExtendedBuffer works in browsers as long as a Buffer polyfill is available. Most bundlers can use the buffer package as a drop-in implementation.

Install the polyfill:

npm install buffer

If your bundler does not expose Buffer globally, add a small shim in your app entry:

import { Buffer } from "buffer";

if (!globalThis.Buffer) {
  globalThis.Buffer = Buffer;
}

Webpack 5

// webpack.config.js
const webpack = require("webpack");

module.exports = {
  resolve: {
    fallback: {
      buffer: require.resolve("buffer/")
    }
  },
  plugins: [
    new webpack.ProvidePlugin({
      Buffer: ["buffer", "Buffer"]
    })
  ]
};

Vite

// vite.config.ts
import { defineConfig } from "vite";

export default defineConfig({
  resolve: {
    alias: {
      buffer: "buffer"
    }
  },
  optimizeDeps: {
    include: ["buffer"]
  }
});

Rollup

// rollup.config.js
import resolve from "@rollup/plugin-node-resolve";
import commonjs from "@rollup/plugin-commonjs";
import inject from "@rollup/plugin-inject";

export default {
  plugins: [
    resolve({ browser: true, preferBuiltins: false }),
    commonjs(),
    inject({
      Buffer: ["buffer", "Buffer"]
    })
  ]
};

Notes:

• BigInt read/write methods require a polyfill that supports Node’s BigInt Buffer APIs.
• If your tooling already provides a Buffer global, you can skip the shim/inject steps.

Quick start

import { ExtendedBuffer } from 'extended-buffer';

const b = new ExtendedBuffer();

b.writeString("OK");          // append
b.writeUInt16BE(1337);        // append

console.log(b.readString(2)); // "OK"
console.log(b.readUInt16BE()); // 1337

Core concepts

Stored data vs readable data

The buffer stores a contiguous region of bytes. A separate read pointer tracks how many bytes were already consumed.

• length — total stored bytes (including already-read bytes).
• getReadableSize() — unread bytes remaining.
• pointer / getPointer() — current read pointer (0…length).
• nativePointer() — absolute index inside the underlying Buffer for the next read.

Views

• nativeBufferView — a Buffer view of all stored bytes (from the start of stored data to the end).
• bufferView — a new ExtendedBuffer instance that maps to the same bytes as nativeBufferView (zero-copy).
  The new instance starts with pointer = 0, so you can read/parse without consuming the parent buffer.

If you need only unread bytes, you can derive it:

const unread = b.nativeBufferView.subarray(b.pointer);

Example: parse without touching the parent pointer:

import { ExtendedBuffer } from 'extended-buffer';

const b = new ExtendedBuffer();
b.writeString("OK");
b.writeUInt16BE(1337);

const v = b.bufferView;
console.log(v.readString(2));   // "OK"
console.log(v.readUInt16BE());  // 1337

console.log(b.pointer);         // 0 (parent is untouched)

Notes:

• bufferView shares memory with the parent. In-place mutations (e.g. v.nativeBufferView[0] = 0xff) will be visible in both.
• The view usually has no spare head/tail capacity, so calling v.write*() will likely trigger a reallocation (copy) and then the two instances will diverge.

Construction and options

type ExtendedBufferOptions = {
  capacity?: number;            // initial native buffer size (bytes)
  capacityStep?: number;        // how much to grow when resizing
  nativeAllocSlow?: boolean;    // using Buffer.allocUnsafeSlow() when initializing ExtendedBuffer
  nativeReallocSlow?: boolean;  // using Buffer.allocUnsafeSlow() for further reallocations
  initNativeBuffer?: Buffer;    // use an existing Buffer as the initial native buffer (no copy)
};

Default values:

• capacity: 16 * 1024 bytes (16 KiB)
• capacityStep: same as capacity
• nativeAllocSlow: false
• nativeReallocSlow: false

Example:

const b = new ExtendedBuffer({
  capacity: 1024 * 1024,
  capacityStep: 1024 * 1024,
  nativeAllocSlow: true,
  nativeReallocSlow: true
});

Wrap an existing Buffer (initNativeBuffer)

If you already have a Node.js Buffer (from a socket, file, etc.) and want to parse it with ExtendedBuffer without copying, pass it as initNativeBuffer.

• The buffer is not copied — it becomes the internal _nativeBuffer.
• The instance starts with pointer = 0 and length = initNativeBuffer.length.

import { ExtendedBuffer } from 'extended-buffer';

const packet = Buffer.from([0x00, 0x02, 0x4f, 0x4b]); // 2, "OK"

const b = new ExtendedBuffer({ initNativeBuffer: packet });

const len = b.readUInt16BE();
console.log(b.readString(len)); // "OK"

You can also reuse an instance:

b.initExtendedBuffer(packet);

Note: when you construct from initNativeBuffer, the buffer is treated as already filled. If you later call write*(), it will typically require a reallocation (copy) to make room.

Writing data

Most write methods accept an optional unshift?: boolean:

• unshift = false (default): append to the end
• unshift = true: prepend to the start

Buffers and strings

b.writeNativeBuffer(Buffer.from([1, 2, 3]));
b.writeBuffer(Buffer.from([4, 5, 6]));      // alias that also accepts ExtendedBuffer
b.writeString("hello", "utf8");

Prepend example:

b.writeString("payload");
b.writeUInt16BE(7, true); // prepend length/header

Integers

Variable-width (size must be 1…6 bytes):

b.writeIntBE(-10, 3);
b.writeUIntLE(5000, 4);

Fixed-width helpers:

• writeInt8, writeUInt8
• writeInt16BE, writeInt16LE, writeUInt16BE, writeUInt16LE
• writeInt32BE, writeInt32LE, writeUInt32BE, writeUInt32LE

BigInt (64-bit integers)

If your runtime supports BigInt and Node's Buffer.readBig* / Buffer.writeBig* APIs, you can read/write 64-bit integers as bigint values (always 8 bytes):

• writeBigInt64BE, writeBigInt64LE — signed 64-bit
• writeBigUInt64BE, writeBigUInt64LE — unsigned 64-bit

import { ExtendedBuffer } from 'extended-buffer';

const b = new ExtendedBuffer();

b.writeBigUInt64BE(2n ** 63n); // 9223372036854775808n
b.writeBigInt64LE(-42n);

b.setPointer(0);
console.log(b.readBigUInt64BE()); // 9223372036854775808n
console.log(b.readBigInt64LE());  // -42n

If BigInt is not supported, these methods throw ExtendedBufferUnsupportedError('EXECUTION_ENVIRONMENT_NOT_SUPPORT_BIG_INT').

Floating point

• writeFloatBE, writeFloatLE (4 bytes)
• writeDoubleBE, writeDoubleLE (8 bytes)

Reading data

All read* methods advance the internal read pointer (consume bytes). If there aren’t enough readable bytes, they throw ExtendedBufferRangeError('SIZE_OUT_OF_RANGE').

Checking before reading

if (b.isReadable(4)) {
  const x = b.readUInt32BE();
}

Read a native Buffer or another ExtendedBuffer

// Copy out as a native Buffer
const chunk: Buffer = b.readBuffer(10, true);

// Copy out as a new ExtendedBuffer (same capacity/capacityStep/nativeAllocSlow/nativeReallocSlow by default)
const eb: ExtendedBuffer = b.readBuffer(10);

Strings

const s = b.readString(5, "utf8");

Integers

Variable-width (size 1…6 bytes):

const a = b.readIntBE(3);
const u = b.readUIntLE(4);

Fixed-width helpers:

• readInt8, readUInt8
• readInt16BE, readInt16LE, readUInt16BE, readUInt16LE
• readInt32BE, readInt32LE, readUInt32BE, readUInt32LE

BigInt (64-bit integers)

• readBigInt64BE, readBigInt64LE — signed 64-bit
• readBigUInt64BE, readBigUInt64LE — unsigned 64-bit

const b = new ExtendedBuffer();

b.writeBigInt64BE(-1n);
b.writeBigUInt64BE(18446744073709551615n); // 2^64 - 1

b.setPointer(0);
console.log(b.readBigInt64BE());  // -1n
console.log(b.readBigUInt64BE()); // 18446744073709551615n

Note: Node's Buffer will throw a native RangeError if the value doesn't fit into signed/unsigned 64-bit range.

Floating point

• readFloatBE, readFloatLE
• readDoubleBE, readDoubleLE

Pointer control (peeking / rewinding)

Save pointer, read, then restore (peek)

const p = b.pointer;
const header = b.readUInt16BE();

// decide what to do...
b.setPointer(p); // rewind back to before header

Move relative to current position

b.offset(4);   // skip 4 bytes
b.offset(-2);  // go back 2 bytes (must stay within 0…length)

If you try to set the pointer outside [0, length], it throws ExtendedBufferRangeError('POINTER_OUT_OF_RANGE').

Transactions (atomic changes)

Sometimes you want to perform a multi-step read/write and either:

• commit everything if it succeeds, or
• rollback the buffer to the exact previous state if something fails.

ExtendedBuffer.transaction() wraps your code in a transaction:

const result = b.transaction(() => {
  // any reads/writes/offsets/etc.
  return 123;
});

Rules:

• If the callback returns normally, changes are kept (committed).
• If the callback throws, the buffer is restored (rolled back) and the error is re-thrown.
• Transactions are re-entrant: nested transaction() calls do not create extra snapshots.

What gets rolled back:

• stored payload bytes
• pointer (read pointer)
• internal start/end offsets and the original native Buffer (even if the buffer was reallocated during the callback)

Example: "try parse" without consuming bytes

This is useful for protocols where you might receive partial data and want to retry later.

import { ExtendedBuffer } from 'extended-buffer';

function tryReadFrame(b: ExtendedBuffer): Buffer | null {
  try {
    return b.transaction(() => {
      // (1) read header
      const len = b.readUInt16BE();

      // (2) not enough bytes yet -> rollback and let the caller wait for more data
      if (!b.isReadable(len)) {
        throw new Error('INCOMPLETE_FRAME');
      }

      // (3) success -> commit
      return b.readBuffer(len, true);
    });
  } catch {
    return null;
  }
}

Example: rollback on validation error

b.transaction(() => {
  const magic = b.readUInt32BE();
  if (magic !== 0xdeadbeef) {
    throw new Error('BAD_MAGIC');
  }

  const version = b.readUInt8();
  if (version !== 1) {
    throw new Error('UNSUPPORTED_VERSION');
  }
});

Performance note

transaction() snapshots the current payload (it copies the stored bytes) before running the callback. That makes rollbacks safe, but can be expensive for very large buffers. Use it for small/medium payloads, or when the safety/ergonomics is worth the extra copy.

Memory management

Discard already-read data

If you continuously read from the buffer, you can drop the consumed prefix:

b.discardReadData();

This moves the internal start forward by the number of read bytes and resets pointer to 0.

Shrink free capacity (gc())

b.gc();

gc() first discards read data, then may shrink the underlying native Buffer when free space exceeds capacityStep.

Reset everything

b.clean(); // alias for initExtendedBuffer()

Errors

The library defines these error classes:

• ExtendedBufferError
• ExtendedBufferTypeError
• ExtendedBufferRangeError
• ExtendedBufferUnsupportedError

Common error codes you may see:

• SIZE_OUT_OF_RANGE: reading more bytes than available
• POINTER_OUT_OF_RANGE: setting pointer outside 0…length
• INVALID_INTEGER_SIZE_VALUE_TYPE: size is not a safe integer
• INVALID_INTEGER_SIZE_VALUE_RANGE: integer size not in 1…6
• INVALID_INSTANCE_STATE: internal invariant check failed
• INVALID_BUFFER_TYPE: attempt write invalid buffer type
• VALUE_MUST_BE_AN_INTEGER: value not a safe integer
• VALUE_MUST_BE_AN_UNSIGNED_INTEGER: value is not a safe integer or less than 0
• VALUE_MUST_BE_AN_BIG_INTEGER: value is not a bigint
• VALUE_MUST_BE_AN_UNSIGNED_BIG_INTEGER: value is not a bigint or less than 0
• EXECUTION_ENVIRONMENT_NOT_SUPPORT_BIG_INT: BigInt methods are not supported in the current runtime
• EXCEEDING_MAXIMUM_BUFFER_SIZE: allocation exceeds Node’s kMaxLength or os.totalmem()

Caveats

Prepending (unshift) after reading

unshift=true prepends bytes by moving the internal start pointer, but the read pointer is not adjusted automatically. If you prepend after consuming bytes, you may get surprising results (e.g., some previously read bytes can become readable again, or newly prepended bytes may be skipped).

A safe pattern is:

b.discardReadData();
b.writeUInt16BE(123, true);

nodeGc() is Node-specific

nodeGc() calls global.gc() if it exists. In Node.js it requires starting the process with --expose-gc. In non-Node runtimes, global may not exist.

Reference: full public API (names)

Properties:

• length, capacity, pointer, nativeBufferView, bufferView

Core:

• initExtendedBuffer(initNativeBuffer?), assertInstanceState(), clean()
• nativePointer(), getWritableSizeStart(), getWritableSizeEnd(), getWritableSize(), getReadableSize()
• transaction(callback)
• allocStart(size), allocEnd(size)
• writeNativeBuffer(buf, unshift?), writeBuffer(bufOrEB, unshift?), writeString(str, enc?, unshift?)
• Pointer: setPointer(p), getPointer(), offset(n), isReadable(size)
• Maintenance: discardReadData(), gc(), nodeGc()

Numbers:

• Write: writeIntBE/LE, writeUIntBE/LE, writeInt8, writeUInt8, writeInt16BE/LE, writeUInt16BE/LE, writeInt32BE/LE, writeUInt32BE/LE, writeBigInt64BE/LE, writeBigUInt64BE/LE, writeFloatBE/LE, writeDoubleBE/LE
• Read: readBuffer, readString, readIntBE/LE, readUIntBE/LE, readInt8, readUInt8, readInt16BE/LE, readUInt16BE/LE, readInt32BE/LE, readUInt32BE/LE, readBigInt64BE/LE, readBigUInt64BE/LE, readFloatBE/LE, readDoubleBE/LE

License

MIT