Package Exports

@phroun/paged-buffer
@phroun/paged-buffer/lib/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (@phroun/paged-buffer) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

@phroun/paged-buffer

A high-performance, byte-level buffer system for editing massive files with intelligent memory management and undo/redo capabilities.

Requirements

Node.js: This is a Node.js-only library that uses native modules (fs, crypto, os)
Environment: Server-side or desktop applications (not browser-compatible)

Features

📁 Massive File Support: Edit multi-gigabyte files with minimal memory usage
💾 Intelligent Memory Management: LRU page eviction with configurable memory limits
↩️ Smart Undo/Redo: Transaction-based undo with intelligent operation merging
🔄 File Change Handling: Automatic detection and intelligent merging of external changes
⚡ High Performance: ~6MB memory usage for 10GB files, instant loading
🛡️ Binary & UTF-8 Support: Automatic mode detection with appropriate handling
🔔 Event Notifications: Comprehensive notification system for monitoring operations
🏗️ Minimal API: Focused byte-level operations, letting higher-level libraries handle line/character semantics

Quick Start

npm install @phroun/paged-buffer

const { PagedBuffer, FilePageStorage } = require('@phroun/paged-buffer');

// Create buffer with file-based storage
const storage = new FilePageStorage('/tmp/editor-cache');
const buffer = new PagedBuffer(64 * 1024, storage, 100);

// Load a massive file (loads instantly)
await buffer.loadFile('huge-file.txt');

// Enable undo system
buffer.enableUndo({ maxUndoLevels: 50 });
buffer.undoSystem.configure({
  mergeTimeWindow: 15000,     // Merge operations within 15 seconds
  mergePositionWindow: 1000   // Merge operations within 1000 bytes
});

// Edit the file at byte level
await buffer.insertBytes(1000, Buffer.from('Hello World'));

// Use transactions for complex operations
buffer.beginUndoTransaction('Find and Replace');
await buffer.deleteBytes(2000, 2010);
await buffer.insertBytes(2000, Buffer.from('replacement'));
buffer.commitUndoTransaction();

// Undo changes
await buffer.undo(); // Undoes entire transaction

// Save changes
await buffer.saveFile();

Architecture Philosophy

This library focuses on high-performance byte-level operations and leaves higher-level concerns to the calling application:

This Library Handles:

✅ Byte-level file operations (insertBytes, deleteBytes, getBytes)
✅ Memory management with intelligent paging
✅ File change detection and conflict resolution
✅ Transaction-based undo/redo system
✅ Binary and UTF-8 file mode detection
✅ Minimal line information for UTF-8 files

Your Application Handles:

🎯 Line/character positioning and conversion
🎯 Marks/bookmarks system with position adjustment
🎯 Line-based editing operations
🎯 Text-specific features (syntax highlighting, etc.)

Performance Characteristics

Operation	10GB File	Memory Usage	Time
Load	✅	~6MB	<2s
Random Access	✅	Per page (64KB)	<100ms
Large Insert	✅	Affected pages only	<500ms
Undo/Redo	✅	Operation delta only	<200ms

Core API Reference

Buffer Operations

// Loading and saving
await buffer.loadFile(filename, mode);    // Load file (auto-detects mode)
buffer.loadContent(string);              // Load from string
await buffer.saveFile(filename);         // Save to file
await buffer.saveAs(filename, force);    // Save detached buffer

// Reading data (byte-level)
const data = await buffer.getBytes(start, end);
const size = buffer.getTotalSize();
const mode = buffer.getMode();           // 'binary' or 'utf8'
const state = buffer.getState();        // Buffer state

// Modifying data (byte-level)
await buffer.insertBytes(position, data);
const deleted = await buffer.deleteBytes(start, end);
const original = await buffer.overwriteBytes(position, data);

Line Information (UTF-8 Mode Only)

For higher-level libraries that need line/character positioning:

// Get line start positions (byte offsets)
const lineStarts = await buffer.getLineStarts();
// Returns: [0, 15, 32, 48, ...] - byte positions where each line starts

// Get total line count
const lineCount = await buffer.getLineCount();

// Position conversion helpers
const bytePos = await buffer.lineCharToBytePosition({line: 10, character: 5}, lineStarts);
const lineChar = await buffer.byteToLineCharPosition(1500, lineStarts);

// Convenience methods for line-based operations
const result = await buffer.insertTextAtPosition({line: 10, character: 5}, 'Hello', lineStarts);
const {deletedText, newLineStarts} = await buffer.deleteTextBetweenPositions(
  {line: 5, character: 0}, 
  {line: 5, character: 10}, 
  lineStarts
);

Undo/Redo System

// Enable undo with configuration (two-step process)
buffer.enableUndo({ maxUndoLevels: 1000 });
buffer.undoSystem.configure({
  mergeTimeWindow: 15000,     // Merge operations within 15 seconds  
  mergePositionWindow: 1000   // Merge operations within 1000 bytes
});

// Basic undo/redo
const canUndo = buffer.canUndo();
const canRedo = buffer.canRedo();
await buffer.undo();
await buffer.redo();

// Named transactions
buffer.beginUndoTransaction('Complex Operation');
// ... multiple operations ...
buffer.commitUndoTransaction();

// Transaction rollback
await buffer.rollbackUndoTransaction();

// Transaction status
const inTransaction = buffer.inUndoTransaction();
const txInfo = buffer.getCurrentUndoTransaction();

// Get undo system statistics
const stats = buffer.undoSystem.getStats();
console.log(`Undo levels: ${stats.undoGroups}, Memory: ${stats.memoryUsage} bytes`);

File Change Handling

// Configure change strategies
buffer.setChangeStrategy({
  noEdits: 'rebase',      // Auto-rebase if no local edits
  withEdits: 'warn',      // Warn but continue if local edits exist
  sizeChanged: 'detach'   // Always detach if file size changed
});

// Manual change detection
const changeInfo = await buffer.checkFileChanges();
console.log('File changed:', changeInfo.changed);
console.log('Size changed:', changeInfo.sizeChanged);
console.log('Modified time changed:', changeInfo.mtimeChanged);
console.log('File deleted:', changeInfo.deleted);

// The buffer automatically handles file changes based on your configured strategy
// There's no separate handleFileChanges() method - changes are processed internally

Storage Backends

// File-based storage (recommended for large files)
const fileStorage = new FilePageStorage('/tmp/editor-cache');

// Memory storage (for testing/small files)
const memoryStorage = new MemoryPageStorage();

// Custom storage implementation
class CustomStorage extends PageStorage {
  async savePage(pageId, data) { /* implement */ }
  async loadPage(pageId) { /* implement */ }
  async deletePage(pageId) { /* implement */ }
  async pageExists(pageId) { /* implement */ }
}

Memory Management & Monitoring

// Get detailed memory statistics
const stats = buffer.getMemoryStats();
console.log(`Total pages: ${stats.totalPages}`);
console.log(`Loaded pages: ${stats.loadedPages}/${stats.maxMemoryPages}`);
console.log(`Dirty pages: ${stats.dirtyPages}`);
console.log(`Memory used: ${stats.memoryUsed} bytes`);
console.log(`Undo memory: ${stats.undo.memoryUsage} bytes`);

// Advanced page management happens automatically:
// - LRU eviction when memory limits are reached
// - Automatic page splitting when pages grow too large
// - Integrity verification for file-backed pages

Building a Text Editor on Top

Here's how you might build line-based operations in your text editor:

class TextEditor {
  constructor() {
    this.buffer = new PagedBuffer();
    this.lineStarts = []; // Cache for performance
    this.marks = new Map(); // Your marks implementation
  }

  async loadFile(filename) {
    await this.buffer.loadFile(filename);
    this.lineStarts = await this.buffer.getLineStarts();
    
    // Enable undo with custom configuration
    this.buffer.enableUndo({ maxUndoLevels: 500 });
    this.buffer.undoSystem.configure({
      mergeTimeWindow: 5000,    // Shorter window for responsive editing
      mergePositionWindow: 100  // Merge nearby character operations
    });
  }

  async insertLine(lineNum, text) {
    const position = {line: lineNum, character: 0};
    const result = await this.buffer.insertTextAtPosition(
      position, 
      text + '\n', 
      this.lineStarts
    );
    
    // Update cached line starts
    this.lineStarts = result.newLineStarts;
    
    // Update your marks
    this.updateMarksAfterInsertion(position, [text]);
    
    return result.newPosition;
  }

  setBookmark(name, line, character) {
    this.marks.set(name, {line, character});
  }

  updateMarksAfterInsertion(insertPos, insertedLines) {
    // Your sophisticated marks adjustment logic
    for (const [name, markPos] of this.marks) {
      if (markPos.line >= insertPos.line) {
        // Adjust mark based on your rules
        markPos.line += insertedLines.length;
        this.marks.set(name, markPos);
      }
    }
  }
}

Advanced Usage

Large File Editing

// Configure for very large files
const buffer = new PagedBuffer(
  1024 * 1024,    // 1MB pages for large files
  fileStorage,
  20              // Keep 20 pages in memory (~20MB)
);

// Monitor memory usage
const stats = buffer.getMemoryStats();
console.log(`Memory: ${stats.memoryUsed} bytes`);
console.log(`Pages: ${stats.loadedPages}/${stats.totalPages}`);

Batch Operations with Transactions

// Group related operations
buffer.beginUndoTransaction('Batch Replace');

// Use lineStarts cache for efficiency
const lineStarts = await buffer.getLineStarts();

for (const {line, character, oldText, newText} of replacements) {
  const pos = {line, character};
  await buffer.deleteTextBetweenPositions(
    pos, 
    {line, character: character + oldText.length}, 
    lineStarts
  );
  await buffer.insertTextAtPosition(pos, newText, lineStarts);
}

buffer.commitUndoTransaction(); // Single undo step

// Or rollback if something goes wrong
// await buffer.rollbackUndoTransaction();

Error Handling

try {
  await buffer.loadFile('massive-file.txt');
} catch (error) {
  console.error('Failed to load:', error.message);
}

// Handle detached buffers
if (buffer.getState() === 'detached') {
  console.warn('Buffer is detached from file - saving to backup');
  await buffer.saveAs('backup-file.txt', true);
}

// Handle corrupted state
if (buffer.getState() === 'corrupted') {
  console.error('Buffer integrity compromised');
  // Handle data recovery scenario
}

Memory Management

The buffer uses sophisticated memory management:

Page-based Loading: Only active pages are kept in memory
LRU Eviction: Least recently used pages are evicted when memory limit is reached
Lazy Evaluation: Operations are deferred until actually needed
Delta Storage: Undo operations store only changes, not entire content
Automatic Page Splitting: Large pages are automatically split to maintain performance
Integrity Verification: File-backed pages are verified against checksums

For a 10GB file with default settings:

Initial Load: ~6MB memory (metadata only)
Active Editing: ~6-50MB memory (depending on access patterns)
Undo History: Proportional to actual changes made

File Change Scenarios

The buffer intelligently handles external file changes:

Append-Only (Log Files)

// Original file: 5GB log file
// External process appends 100MB
// Buffer detects append and merges gracefully
// Your edits preserved + new content added

Conflict Resolution

// You edit at byte position 1000
// External process edits at byte position 2000
// Buffer preserves both changes

// You edit at byte position 1000
// External process also edits at byte position 1000  
// Buffer detaches and requires manual resolution

Notifications

buffer.onNotification((notification) => {
  console.log(`${notification.severity}: ${notification.message}`);
  
  switch(notification.type) {
    case 'file_modified_on_disk':
      // Handle external file changes
      break;
    case 'undo_transaction_committed':
      // Transaction completed
      break;
    case 'buffer_detached':
      // Buffer conflicts with file
      break;
    case 'memory_pressure':
      // System is evicting pages due to memory limits
      break;
    case 'page_conflict_detected':
      // Page integrity issues detected
      break;
  }
});

// Clear notifications
buffer.clearNotifications(); // Clear all
buffer.clearNotifications('memory_pressure'); // Clear specific type

// Get all notifications
const notifications = buffer.getNotifications();

Testing Support

The library includes features for testing environments:

// Custom clock for deterministic testing
buffer.undoSystem.setClock(() => mockTimestamp);

// Reset operation counters (for testing)
const { resetOperationCounter } = require('@phroun/paged-buffer');
resetOperationCounter();

// Memory storage for fast tests
const buffer = new PagedBuffer(1024, new MemoryPageStorage(), 10);

Testing

# Run all tests
npm test

# Run with coverage
npm run test:coverage

# Run specific test suite
npm test -- --grep "Undo System"

# Run integration tests (requires more time/disk space)
npm test -- --grep "Integration"

Contributing

Fork the repository
Create a feature branch: git checkout -b feature-name
Make changes and add tests
Run tests: npm test
Submit a pull request

License

MIT License - see LICENSE file for details.

Changelog

0.1.4

Improved documentation

0.1.3

Solved an issue preventing saving of files larger than 2GB

0.1.2

Refactored to make the Virtual Page Manager its module with full tests

0.1.0

Initial release
Core paged buffer functionality with byte-level operations
Transaction-based undo/redo system with intelligent merging
File change detection and handling with configurable strategies
Line information helpers for UTF-8 files
Comprehensive notification system
Multiple storage backend support
Advanced memory management with LRU eviction
Comprehensive test suite
Full documentation