Package Exports

vectoriadb
vectoriadb/package.json

Readme

VectoriaDB

A lightweight, production-ready in-memory vector database for semantic search in JavaScript/TypeScript

VectoriaDB is a fast, minimal-dependency vector database designed for in-memory semantic search. Powered by transformers.js, it's perfect for applications that need to quickly search through documents, tools, or any text-based data using natural language queries.

Features
Installation
Why VectoriaDB?
Quick Start
Core Concepts
API Reference
Advanced Usage
Error Handling
Performance
Use Cases
Testing
Comparison
Limitations
Roadmap
Contributing
License

Features

🚀 Fast: In-memory storage with optimized HNSW indexing for O(log n) search
🪶 Lightweight: Minimal dependencies, small footprint
🔍 Semantic Search: Natural language queries using state-of-the-art embeddings
🎯 Type-Safe: Full TypeScript support with generics
⚡ Batch Operations: Efficient bulk insert and search
🔧 Flexible Filtering: Custom metadata filtering with type safety
📊 Scalable: HNSW index for 100k+ documents with sub-millisecond search
💾 Persistent: File & Redis adapters for caching across restarts
🔄 Smart Updates: Incremental updates without re-embedding (instant metadata updates)
🛡️ Production-Ready Error Handling: Typed error classes with specific error codes
📦 Battle-Tested: Used in production at FrontMCP

Installation

npm install vectoriadb
# or
yarn add vectoriadb
# or
pnpm add vectoriadb

Requirements:

Node.js 18+ (for transformers.js compatibility)
TypeScript 5.0+ (if using TypeScript)

Why VectoriaDB?

Use VectoriaDB when you need:

🎯 Semantic search without complex infrastructure (no external services required)
⚡ Fast in-memory search with HNSW indexing (handles 100k+ documents)
🔒 Privacy-first - all embeddings generated locally, no API calls
🚀 Production-ready vector search with minimal setup
📦 Embedded search in Node.js applications, CLIs, or desktop apps

Skip VectoriaDB if you need:

💾 Persistent storage (use Pinecone, Weaviate, or Qdrant)
🌐 Distributed architecture (use Weaviate or Milvus)
📊 Multi-million document scale (use specialized distributed vector DBs)

Quick Start

import { VectoriaDB } from 'vectoriadb';

// Create and initialize the database
const db = new VectoriaDB();
await db.initialize();

// Add documents
await db.add('doc-1', 'How to create a user account', {
  id: 'doc-1',
  category: 'auth',
  author: 'Alice',
});

await db.add('doc-2', 'Send email notifications to users', {
  id: 'doc-2',
  category: 'notifications',
  author: 'Bob',
});

// Search
const results = await db.search('creating new accounts');
console.log(results[0].metadata); // { id: 'doc-1', category: 'auth', ... }
console.log(results[0].score); // 0.87

Core Concepts

Documents

Each document in VectoriaDB consists of:

id: Unique identifier
text: The text content to search
metadata: Custom metadata (type-safe with generics)

Embeddings

VectoriaDB automatically generates embeddings (vector representations) of your documents using transformers.js. The default model is Xenova/all-MiniLM-L6-v2 (22MB, 384 dimensions), which provides a great balance of size, speed, and accuracy.

Search

Search uses cosine similarity to find the most semantically similar documents to your query.

API Reference

Constructor

const db = new VectoriaDB<MetadataType>(config?)

Config Options:

interface VectoriaConfig {
  modelName?: string; // Default: 'Xenova/all-MiniLM-L6-v2'
  dimensions?: number; // Auto-detected from model
  defaultSimilarityThreshold?: number; // Default: 0.3
  defaultTopK?: number; // Default: 10
}

Methods

`initialize(): Promise<void>`

Initialize the embedding model. Must be called before using the database.

await db.initialize();

`add(id: string, text: string, metadata: T): Promise<void>`

Add a single document to the database.

await db.add('doc-1', 'Document content', { id: 'doc-1', category: 'tech' });

`addMany(documents: Array<{id, text, metadata}>): Promise<void>`

Add multiple documents in batch (more efficient).

await db.addMany([
  { id: 'doc-1', text: 'Content 1', metadata: { id: 'doc-1' } },
  { id: 'doc-2', text: 'Content 2', metadata: { id: 'doc-2' } },
]);

`search(query: string, options?): Promise<SearchResult<T>[]>`

Search for documents using semantic similarity.

const results = await db.search('machine learning', {
  topK: 5, // Return top 5 results
  threshold: 0.5, // Minimum similarity score
  filter: (metadata) => metadata.category === 'tech', // Custom filter
  includeVector: false, // Include vector in results
});

`get(id: string): DocumentEmbedding<T> | undefined`

Get a document by ID.

const doc = db.get('doc-1');

`has(id: string): boolean`

Check if a document exists.

if (db.has('doc-1')) {
  // Document exists
}

`remove(id: string): boolean`

Remove a document.

db.remove('doc-1');

`removeMany(ids: string[]): number`

Remove multiple documents.

const removed = db.removeMany(['doc-1', 'doc-2']);

`clear(): void`

Remove all documents.

db.clear();

`size(): number`

Get the number of documents.

const count = db.size();

`filter(filterFn): DocumentEmbedding<T>[]`

Get documents by filter (without semantic search).

const techDocs = db.filter((metadata) => metadata.category === 'tech');

`getStats(): VectoriaStats`

Get database statistics.

const stats = db.getStats();
console.log(stats.totalEmbeddings);
console.log(stats.estimatedMemoryBytes);

Advanced Usage

Type-Safe Metadata

Use TypeScript generics for type-safe metadata:

interface MyMetadata extends DocumentMetadata {
  id: string;
  category: 'tech' | 'business' | 'science';
  author: string;
  tags: string[];
}

const db = new VectoriaDB<MyMetadata>();

await db.add('doc-1', 'Content', {
  id: 'doc-1',
  category: 'tech', // Type-checked!
  author: 'Alice',
  tags: ['ai', 'ml'],
});

const results = await db.search('query', {
  filter: (metadata) => {
    // metadata is fully typed!
    return metadata.category === 'tech' && metadata.tags.includes('ai');
  },
});

Custom Embedding Models

Use any Hugging Face model compatible with transformers.js:

const db = new VectoriaDB({
  modelName: 'Xenova/paraphrase-multilingual-MiniLM-L12-v2', // Multilingual support
});

Batch Operations

For better performance with large datasets:

const documents = [
  { id: '1', text: 'Doc 1', metadata: { id: '1' } },
  { id: '2', text: 'Doc 2', metadata: { id: '2' } },
  // ... thousands more
];

// Much faster than calling add() in a loop
await db.addMany(documents);

HNSW Index for Production Scale

For production applications with large datasets (>10k documents), enable HNSW (Hierarchical Navigable Small World) indexing for faster approximate nearest neighbor search:

const db = new VectoriaDB({
  useHNSW: true,
  hnsw: {
    M: 16, // Max connections per node (higher = better recall, more memory)
    M0: 32, // Max connections at layer 0
    efConstruction: 200, // Construction quality (higher = better quality, slower build)
    efSearch: 50, // Search quality (higher = better recall, slower search)
  },
});

await db.initialize();

// Add documents - HNSW index is built automatically
await db.addMany(documents);

// Search uses HNSW for O(log n) instead of O(n) complexity
const results = await db.search('query');

HNSW Benefits:

Speed: O(log n) search vs O(n) brute-force
Scalability: Handles 100k+ documents efficiently
Accuracy: >95% recall with proper tuning
Production-Ready: Battle-tested algorithm used by major vector databases

Parameter Tuning:

Parameter	Lower Value	Higher Value	Default
M	Faster build, less memory	Better recall, more memory	16
efConstruction	Faster build, lower quality	Better quality, slower build	200
efSearch	Faster search, lower recall	Better recall, slower search	50

When to use HNSW:

✅ Dataset > 10,000 documents
✅ Search latency is critical
✅ Have memory for the graph structure (~50-100 bytes per document per connection)
❌ Dataset < 1,000 documents (overhead not worth it)
❌ Need exact nearest neighbors (HNSW is approximate)

Complex Filtering

Combine semantic search with complex metadata filters:

interface SecurityMetadata extends DocumentMetadata {
  id: string;
  category: string;
  tags: string[];
  author: string;
  priority: 'low' | 'medium' | 'high';
}

const db = new VectoriaDB<SecurityMetadata>();

const results = await db.search('user authentication', {
  topK: 10,
  threshold: 0.4,
  filter: (metadata) => {
    return (
      metadata.category === 'security' &&
      metadata.tags.includes('auth') &&
      metadata.author === 'security-team' &&
      metadata.priority === 'high'
    );
  },
});

Persistence with Storage Adapters

Cache embeddings across restarts to avoid recalculation. VectoriaDB supports multiple storage backends:

In-Memory (Default)

No persistence - data is lost on restart:

const db = new VectoriaDB(); // Uses MemoryStorageAdapter by default

File-Based Persistence

Perfect for local development - caches to disk with automatic invalidation when tools change:

import { VectoriaDB, FileStorageAdapter, SerializationUtils } from 'vectoriadb';

const documents = [
  { id: 'tool-1', text: 'Create user account', metadata: { id: 'tool-1' } },
  { id: 'tool-2', text: 'Send email notification', metadata: { id: 'tool-2' } },
];

// Create tools hash for cache invalidation
const toolsHash = SerializationUtils.createToolsHash(documents);

const db = new VectoriaDB({
  storageAdapter: new FileStorageAdapter({
    cacheDir: './.cache/vectoriadb',
    namespace: 'my-app', // Separate cache per namespace
  }),
  toolsHash, // Cache invalidated when tools change
  version: '1.0.0', // Cache invalidated when version changes
});

await db.initialize(); // Automatically loads from cache if valid

// Add documents (only on first run or after invalidation)
if (db.size() === 0) {
  await db.addMany(documents);
  await db.saveToStorage(); // Manually save to cache
}

// Subsequent runs will load from cache instantly

Redis for Distributed Caching

Share embeddings across pods in distributed environments:

import { VectoriaDB, RedisStorageAdapter, SerializationUtils } from 'vectoriadb';
import Redis from 'ioredis'; // or your Redis client

const documents = [
  /* your documents */
];
const toolsHash = SerializationUtils.createToolsHash(documents);

const redis = new Redis({
  host: 'localhost',
  port: 6379,
});

const db = new VectoriaDB({
  storageAdapter: new RedisStorageAdapter({
    client: redis,
    namespace: 'my-app-v1', // Namespace by app + version
    ttl: 86400, // 24 hours (default)
  }),
  toolsHash,
  version: process.env.APP_VERSION,
});

await db.initialize(); // Loads from Redis if cache is valid

if (db.size() === 0) {
  await db.addMany(documents);
  await db.saveToStorage();
}

// Don't forget to close when shutting down
await db.close();

Cache Invalidation:

The cache is automatically invalidated when:

toolsHash changes (documents added/removed/modified)
version changes (application version updated)
modelName changes (different embedding model)

Best Practices:

Local dev: Use FileStorageAdapter to speed up restarts
Production: Use RedisStorageAdapter for multi-pod deployments
Tools hash: Create from document IDs + texts for automatic invalidation
Namespace: Use app name + version to prevent cache conflicts
Manual save: Call saveToStorage() after adding documents

Incremental Updates (Production-Ready)

Update documents efficiently without re-embedding when only metadata changes:

Update Metadata Only (Instant)

// Update metadata without re-embedding (instant operation)
db.updateMetadata('doc-1', {
  id: 'doc-1',
  category: 'updated-category',
  priority: 'high',
  lastModified: new Date(),
});

Smart Update (Auto-Detection)

// Only re-embeds if text actually changed
const reembedded = await db.update('doc-1', {
  text: 'Updated content', // If different, will re-embed
  metadata: { id: 'doc-1', category: 'updated' },
});

console.log(reembedded); // true if re-embedded, false if text was same

Batch Updates (Efficient)

// Update many documents - only re-embeds those with text changes
const result = await db.updateMany([
  {
    id: 'doc-1',
    text: 'New content for doc 1', // Will re-embed
    metadata: { id: 'doc-1', category: 'tech' },
  },
  {
    id: 'doc-2',
    metadata: { id: 'doc-2', category: 'food' }, // No text = no re-embedding
  },
  {
    id: 'doc-3',
    text: 'Same text as before', // Smart detection = no re-embedding
    metadata: { id: 'doc-3', category: 'science' },
  },
]);

console.log(`Updated ${result.updated} documents`);
console.log(`Re-embedded ${result.reembedded} documents`); // Only what changed

Force Re-Embedding

// Force re-embed even if text hasn't changed (e.g., new embedding model)
await db.update('doc-1', { text: 'same text' }, { forceReembed: true });

// Force re-embed all in batch
await db.updateMany(docs, { forceReembed: true });

Performance Benefits:

Operation	Speed	Re-embedding
`updateMetadata()`	Instant	Never
`update()` (metadata)	Instant	No
`update()` (text)	~100-200ms	Only if changed
`updateMany()` (mixed)	Batched	Only what changed

Use Cases:

Metadata updates: Change categories, tags, priorities instantly
Partial text updates: Only re-embed documents that actually changed
Dynamic content: Update frequently changing metadata without performance hit
Bulk operations: Efficiently update thousands of documents

Error Handling

VectoriaDB provides production-ready error handling with specific error types that can be caught and handled individually.

Error Classes

All errors extend the base VectoriaError class with a code property for programmatic error handling:

import {
  VectoriaError, // Base error class
  VectoriaNotInitializedError, // DB not initialized
  DocumentValidationError, // Invalid document data
  DocumentNotFoundError, // Document doesn't exist
  DocumentExistsError, // Document already exists
  DuplicateDocumentError, // Duplicate in batch or existing
  QueryValidationError, // Invalid search query/params
  EmbeddingError, // Embedding generation failure
  StorageError, // Storage operation failure
  ConfigurationError, // Invalid configuration
} from 'vectoriadb';

Error Types

VectoriaNotInitializedError

Thrown when operations are attempted before calling initialize():

const db = new VectoriaDB();

try {
  await db.add('doc-1', 'text', { id: 'doc-1' });
} catch (error) {
  if (error instanceof VectoriaNotInitializedError) {
    console.log(error.code); // 'NOT_INITIALIZED'
    console.log(error.message); // 'VectoriaDB must be initialized before adding documents...'
    await db.initialize(); // Fix: initialize first
  }
}

DocumentValidationError

Thrown when document data is invalid:

try {
  // Empty text
  await db.add('doc-1', '', { id: 'doc-1' });
} catch (error) {
  if (error instanceof DocumentValidationError) {
    console.log(error.code); // 'DOCUMENT_VALIDATION_ERROR'
    console.log(error.documentId); // 'doc-1'
  }
}

try {
  // Metadata.id mismatch
  await db.add('doc-1', 'text', { id: 'doc-2' });
} catch (error) {
  if (error instanceof DocumentValidationError) {
    console.log(error.message); // 'Metadata id "doc-2" does not match document id "doc-1"'
  }
}

DocumentNotFoundError

Thrown when attempting to update a non-existent document:

try {
  await db.update('nonexistent', { text: 'new' });
} catch (error) {
  if (error instanceof DocumentNotFoundError) {
    console.log(error.code); // 'DOCUMENT_NOT_FOUND'
    console.log(error.documentId); // 'nonexistent'
  }
}

DocumentExistsError

Thrown when adding a document with an ID that already exists:

await db.add('doc-1', 'text', { id: 'doc-1' });

try {
  await db.add('doc-1', 'duplicate', { id: 'doc-1' });
} catch (error) {
  if (error instanceof DocumentExistsError) {
    console.log(error.code); // 'DOCUMENT_EXISTS'
    console.log(error.documentId); // 'doc-1'
    // Fix: use remove() first or choose different ID
    db.remove('doc-1');
    await db.add('doc-1', 'duplicate', { id: 'doc-1' });
  }
}

DuplicateDocumentError

Thrown when batch operations contain duplicates:

try {
  await db.addMany([
    { id: 'doc-1', text: 'first', metadata: { id: 'doc-1' } },
    { id: 'doc-1', text: 'second', metadata: { id: 'doc-1' } }, // Duplicate in batch
  ]);
} catch (error) {
  if (error instanceof DuplicateDocumentError) {
    console.log(error.code); // 'DUPLICATE_DOCUMENT'
    console.log(error.context); // 'batch' or 'existing'
    console.log(error.documentId); // 'doc-1'
  }
}

QueryValidationError

Thrown when search parameters are invalid:

try {
  await db.search(''); // Empty query
} catch (error) {
  if (error instanceof QueryValidationError) {
    console.log(error.code); // 'QUERY_VALIDATION_ERROR'
  }
}

try {
  await db.search('query', { topK: -5 }); // Invalid topK
} catch (error) {
  if (error instanceof QueryValidationError) {
    console.log(error.message); // 'topK must be a positive number'
  }
}

try {
  await db.search('query', { threshold: 1.5 }); // Invalid threshold
} catch (error) {
  if (error instanceof QueryValidationError) {
    console.log(error.message); // 'threshold must be between 0 and 1'
  }
}

EmbeddingError

Thrown when embedding generation fails:

try {
  // This would only happen with internal errors
  await db.addMany(documents);
} catch (error) {
  if (error instanceof EmbeddingError) {
    console.log(error.code); // 'EMBEDDING_ERROR'
    console.log(error.details); // Additional error details
  }
}

Production Error Handling Patterns

Catch Specific Errors

try {
  await db.add('doc-1', text, metadata);
} catch (error) {
  if (error instanceof DocumentExistsError) {
    // Handle duplicate: maybe update instead
    await db.update(error.documentId, { text, metadata });
  } else if (error instanceof DocumentValidationError) {
    // Handle validation: log and skip
    console.error(`Invalid document ${error.documentId}:`, error.message);
  } else if (error instanceof VectoriaNotInitializedError) {
    // Handle initialization: retry after init
    await db.initialize();
    await db.add('doc-1', text, metadata);
  } else {
    // Unknown error: rethrow
    throw error;
  }
}

Catch by Error Code

try {
  await db.search(query);
} catch (error) {
  if (error instanceof VectoriaError) {
    switch (error.code) {
      case 'NOT_INITIALIZED':
        await db.initialize();
        break;
      case 'QUERY_VALIDATION_ERROR':
        console.error('Invalid query:', error.message);
        break;
      default:
        throw error;
    }
  }
}

Batch Operations with Error Recovery

async function addDocumentsSafely(documents: Array<{ id: string; text: string; metadata: T }>) {
  try {
    await db.addMany(documents);
  } catch (error) {
    if (error instanceof DuplicateDocumentError) {
      // Remove duplicate and retry
      const uniqueDocs = documents.filter((doc) => doc.id !== error.documentId);
      await db.addMany(uniqueDocs);
      console.warn(`Skipped duplicate: ${error.documentId}`);
    } else if (error instanceof DocumentValidationError) {
      // Log validation error and continue with valid documents
      console.error(`Invalid document ${error.documentId}:`, error.message);
      // Filter out invalid document and retry
      const validDocs = documents.filter((doc) => doc.id !== error.documentId);
      await db.addMany(validDocs);
    } else {
      throw error; // Unexpected error
    }
  }
}

Graceful Degradation

async function searchWithFallback(query: string) {
  try {
    return await db.search(query);
  } catch (error) {
    if (error instanceof QueryValidationError) {
      // Fallback to default search
      console.warn('Invalid query, using default search');
      return await db.search('default query', { threshold: 0.1 });
    } else if (error instanceof VectoriaNotInitializedError) {
      // Initialize and retry
      await db.initialize();
      return await db.search(query);
    }
    throw error;
  }
}

Error Codes Reference

Error Class	Code	When Thrown
`VectoriaNotInitializedError`	`NOT_INITIALIZED`	Operation before `initialize()`
`DocumentValidationError`	`DOCUMENT_VALIDATION_ERROR`	Empty text, metadata mismatch
`DocumentNotFoundError`	`DOCUMENT_NOT_FOUND`	Update/get non-existent document
`DocumentExistsError`	`DOCUMENT_EXISTS`	Add document with existing ID
`DuplicateDocumentError`	`DUPLICATE_DOCUMENT`	Duplicate in batch or existing document
`QueryValidationError`	`QUERY_VALIDATION_ERROR`	Empty query, invalid topK/threshold
`EmbeddingError`	`EMBEDDING_ERROR`	Embedding generation failure
`StorageError`	`STORAGE_ERROR`	Storage operation failure
`ConfigurationError`	`CONFIGURATION_ERROR`	Invalid configuration

Best Practices

Always catch specific errors instead of generic Error
Use error codes for programmatic handling
Access error properties (documentId, context, etc.) for debugging
Implement retry logic for VectoriaNotInitializedError
Log validation errors with context for debugging
Graceful fallbacks for production resilience

Performance

Memory Usage

Memory efficient with Float32 arrays:

Embeddings: ~1.5KB per document (384 dimensions × 4 bytes)
Metadata: ~1KB per document (estimated)

Example: 10,000 documents ≈ 25 MB

Search Speed

Without HNSW (brute-force):

Complexity: O(n) where n = number of documents
Performance: <10ms for 10,000 documents on modern hardware
Best for: <10,000 documents

With HNSW (approximate nearest neighbor):

Complexity: O(log n) approximate search
Performance: Sub-millisecond for 100,000+ documents
Accuracy: >95% recall with default parameters
Best for: >10,000 documents

Embedding Generation

Model: Xenova/all-MiniLM-L6-v2 (22MB)
Speed: ~100-200 embeddings/second (hardware dependent)
Batch optimization: 32 documents per batch

Use Cases

1. Tool Discovery

interface ToolMetadata extends DocumentMetadata {
  id: string;
  toolName: string;
  category: string;
}

const db = new VectoriaDB<ToolMetadata>();
await db.initialize();

await db.addMany([
  { id: 'tool-1', text: 'Create user accounts', metadata: { id: 'tool-1', toolName: 'create_user', category: 'auth' } },
  { id: 'tool-2', text: 'Send emails', metadata: { id: 'tool-2', toolName: 'send_email', category: 'notification' } },
]);

const results = await db.search('how to add new users');
// Returns: [{ metadata: { toolName: 'create_user', ... }, score: 0.89 }]

2. Documentation Search

interface DocMetadata extends DocumentMetadata {
  id: string;
  title: string;
  section: string;
  url: string;
}

const db = new VectoriaDB<DocMetadata>();
// Add documentation pages
// Search with natural language

3. Product Search

interface ProductMetadata extends DocumentMetadata {
  id: string;
  name: string;
  category: string;
  price: number;
}

const db = new VectoriaDB<ProductMetadata>();
// Add products with descriptions
// Search: "affordable wireless headphones"

Testing

VectoriaDB comes with comprehensive tests covering all major functionality:

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

The test suite includes:

Embedding Service Tests: Verify embedding generation and model initialization
Vector Database Tests: Test CRUD operations, search, and filtering
Similarity Tests: Validate cosine similarity calculations

All tests use mocked transformers.js to avoid downloading models during CI/CD, making tests fast and reliable.

Comparison with Other Vector Databases

Feature	VectoriaDB	Pinecone	Weaviate	ChromaDB
In-memory	✅	❌	❌	✅
Lightweight	✅ (22MB)	❌	❌	⚠️
Type-safe	✅	⚠️	⚠️	⚠️
Zero config	✅	❌	❌	✅
Production-ready	✅	✅	✅	✅
Persistence	❌	✅	✅	✅
Distributed	❌	✅	✅	❌

VectoriaDB is ideal for:

Small to medium datasets (<100k documents)
Fast in-memory search without external dependencies
Embedded applications that need semantic search
Development and testing before scaling to production DBs

Limitations

Single process: Not distributed (use Redis adapter for multi-pod setups)
HNSW is approximate: ~95% recall vs 100% with brute-force (use brute-force for exact results)
In-memory primary: Persistence via adapters (cache strategy, not database)

Roadmap

Comprehensive test suite with mocked dependencies
HNSW indexing for faster search (>100k documents)
Persistence adapters (Redis, File, Memory)
Incremental updates without re-embedding
Production-ready error handling with typed exceptions
Compression for stored embeddings
Multi-vector embeddings per document

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

License

Apache-2.0

Credits

Built with:

transformers.js by Xenova

vectoriadb