@hce/encode

HCE (Hierarchical Columnar Encoding) encoder for efficient data compression.

Installation

npm install @hce/encode
# or
pnpm add @hce/encode

Usage

import { HCEEncoder } from '@hce/encode';

const data = [
  { type: 'user', id: 1, name: 'Alice', role: 'admin' },
  { type: 'user', id: 2, name: 'Bob', role: 'user' }
];

const encoder = new HCEEncoder({
  fieldDelimiter: ',',
  recordDelimiter: '|',
  typeField: 'type',
  autoDetectGrouping: true
});

const hce = encoder.encode(data, 'users');
console.log(hce);

Options

• fieldDelimiter - Delimiter between fields (default: ',')
• recordDelimiter - Delimiter between records (default: '|')
• nestedDelimiter - Delimiter for nested arrays (default: ';')
• missingValue - Placeholder for missing/null values (default: ' ')
• flattenNested - Whether to flatten nested objects (default: true)
• typeField - Name of the type field (default: 'type')
• indent - Indentation size in spaces (default: 2)
• autoDetectGrouping - Auto-detect best grouping field (default: true)
• preferTypeGrouping - Prefer type-only grouping for uniform schemas (default: true)
• minGroupSizeForSecondaryGrouping - Minimum average group size (default: 5)
• schemaUniformityThreshold - Schema uniformity threshold 0-1 (default: 0.9)

Features

• Automatic grouping optimization based on schema uniformity
• 48-60% compression ratio on typical JSON data
• Intelligent field detection and encoding
• Support for nested objects and arrays
• Type-safe TypeScript implementation

License

MIT