warpvector 🌌

[!NOTE] 🌍 日本語のドキュメント: 🇯🇵 日本語版の README はこちらからお読みいただけます

Warp your vector space at runtime — no retraining, no Python, just TypeScript.

WarpVector is a lightweight, zero-dependency TypeScript middleware that dynamically transforms vector spaces based on search context and user intent, without retraining AI models or running expensive re-inference.

✨ Project Highlights

• ⚡️ Blazing Fast (Edge Ready): Sub-millisecond inference directly on Cloudflare Workers or in-browser via WASM.
• 🧠 Dynamic & Smart: Instantly warps the vector space in real-time based on user intent, boosting search accuracy.
• 💸 Cost-Effective: Slashes Vector DB storage and memory costs by up to 96.9% using Int8/Binary quantization.
• 🔄 Zero-Downtime Migration: Translate vector spaces on the fly (e.g., 1536D to 512D) to eliminate vendor lock-in without re-indexing.
• 📦 Zero-Python (Pure TS): No heavy ML frameworks. Bring advanced machine learning directly into your JS/TS backend.

Experience real-time vector space transformation and quantization in your browser via WASM.

💡 Why WarpVector?

Traditional vector search is static — it depends entirely on pre-generated embedding distances. When you need context-aware tuning, your only options have been metadata filtering or expensive re-inference with instruction-tuned models (usually requiring a Python backend).

WarpVector changes this. It acts as a "magic filter" without ever touching the base embedding model.

🔄 Before / After: Evolution of Search Architecture

graph TD
    subgraph "❌ Before (Traditional Static Search)"
        B_Query[User Query<br/>e.g., 'Apple'] --> B_LLM[Embedding Model<br/>ada-002, etc.]
        B_LLM -->|Static Vector| B_DB[(Vector DB)]
        B_DB -.->|Problem| B_Result[Mixes up fruit & company.<br/>Noisy search results.]
    end

    subgraph "✨ After (Dynamic Search with WarpVector)"
        A_Query[User Query<br/>e.g., 'Apple'] --> A_LLM[Embedding Model<br/>ada-002, etc.]
        A_LLM -->|Static Vector| A_WV{⚡️ WarpVector Middleware<br/>Applies 'IT Domain' Intent}
        A_WV -->|Optimized Vector| A_DB[(Vector DB)]
        A_DB -.->|Solution| A_Result[Space is warped!<br/>Apple Inc. instantly rises to top.]
    end

🌍 Uniqueness in the Ecosystem (Why WarpVector?)

WarpVector occupies a highly unique position in the current ecosystem by being edge-native, zero-dependency, and purely TypeScript.

• vs. Heavy LLM Frameworks (LlamaIndex / LangChain) While some massive frameworks have concepts like "Embedding Adapters", they come with huge dependencies. WarpVector extracts this concept as an ultra-lightweight, WASM-accelerated middleware designed to run sub-millisecond as a standalone utility on edge environments like Cloudflare Workers.
• vs. Backend ML Libraries (Faiss / Sentence-Transformers) Advanced vector optimization—such as Whitening, contrastive learning, and quantization—traditionally required heavy Python/PyTorch or C++ infrastructure. WarpVector rebuilds these complex mathematical optimizations natively in TypeScript, liberating them for the frontend and edge runtimes.

🎯 5 Key Use Cases

Integrating WarpVector into your RAG or vector search systems solves the following challenges:

• 🎯 1. Intent-Aware Personalized Search

  Standard embeddings can't distinguish "Apple" (fruit) from "Apple" (company). WarpVector lets you switch intents to instantly warp the vector space toward the right domain.

• 🔄 2. Log-Driven Online Learning (Separation of Concerns)

  Collect user click/skip logs at the edge, run online learning in your backend, and instantly deploy only the lightweight transformation matrices to the edge — keeping inference lightning fast.

• 📐 3. Auto-Correction of Embedding Anisotropy

  Many models produce vectors that are all too similar (anisotropy bias). WhiteningAdapter automatically learns and removes this bias via streaming PCA, dramatically improving search resolution.

• 💾 4. 75–97% Memory Reduction via Quantization

  Add .setFinalStage("quantize", ...) to your pipeline to compress vectors from Float32 to Int8 or Binary format, shrinking DB costs without sacrificing accuracy.

• 🔄 5. Zero-Downtime Model Migration (No Re-indexing)

  Upgrading from ada-002 to text-embedding-3? Train an AlignmentAdapter with just 100 pairs, and translate new queries into your old vector space instantly, eliminating vendor lock-in.

• 🚀 6. Drop-in Integration — Just a Few Lines of TS

  No Python or heavy ML frameworks needed. Pure TypeScript + WASM. Integrates cleanly with LangChain, LlamaIndex, and Prisma (pgvector).

🤝 Drop-in Integrations

[LangChain] [LlamaIndex] [Prisma / pgvector] [Pinecone] [Cloudflare Vectorize] [Redis]

⚡ Results at a Glance

🧩 Feature Architecture (Edge vs Backend)

WarpVector adopts a clear architectural separation between "Edge Inference" (requiring ultra-low latency) and "Backend Training" (requiring heavy compute resources).

graph TD
    subgraph "⚡ Edge Inference Layer (Sub-ms, WASM, Zero-dep)"
        E_Core[Core Transforms<br/>Intent, Projection, Lora]
        E_ML[Neural Nets<br/>MlpAdapter, Non-linear]
        E_Opt[Optimization & Compression<br/>Whitening, Quantization]
        E_Search[Hybrid Search & VSA]
    end

    subgraph "🧠 Backend & Training Layer (Node.js/Workers)"
        B_Train[Trainers<br/>InfoNCETrainer, TripletTrainer]
        B_Auto[Auto-ML<br/>IntentMatrixFactory]
        B_Rerank[Heavy Reranking<br/>ColBERT, Scattering]
    end

    B_Train -. "Deploy Lightweight Weights" .-> E_Core
    B_Auto -. "Auto-generate Intent Matrices" .-> E_Core
    B_Train -. "Task Arithmetic Model Merging" .-> E_Core

📦 Installation

npm install warpvector
# or
pnpm add warpvector
# or
yarn add warpvector
# or
bun add warpvector

Core features operate with zero dependencies. For integrations:

# Prisma + pgvector
npm install @prisma/client sql-template-tag

# LangChain / LlamaIndex
npm install @langchain/core

🛠 Quick Start Guide

WarpVector is feature-rich, so we've grouped the basic usage by category. Refer to the documentation links below for details.

1. Basic Pipeline Configuration (WarpPipeline)

Compose complex vector transformations and DB format outputs intuitively.

import { WarpPipeline } from "warpvector";
import { MlpAdapter } from "warpvector/ml";
import { QuantizationAdapter } from "warpvector/extras";

// 1. Compose the pipeline
const pipeline = new WarpPipeline(1536)
  .addStep(new MlpAdapter(layers))
  .addIntent({ domain_x: intentWeights })
  .setFinalStage(new QuantizationAdapter({ type: "int8", dim: 1536 }));

// 2. Async init (WASM setup, etc.)
await pipeline.init();

// 3. Fast inference & output formatting
const pineconeQuery = pipeline.runAndFormat(
  rawVector,
  { format: "pinecone", topK: 10, filter: { genre: "action" } },
  { intent: "domain_x" },
);

2. Core Transforms (Intent & Dimensionality Reduction)

import { IntentAdapter, ProjectionAdapter } from 'warpvector';

// 1. IntentAdapter: Define domain-specific affine transformations
const adapter = new IntentAdapter({
  riskAnalysis: { matrix: [...], bias: [...] }
});
const warpedVector = adapter.tune(baseVector, "riskAnalysis");

// 2. ProjectionAdapter: Fast WASM dimensionality reduction (e.g., 1536D -> 512D)
const projAdapter = new ProjectionAdapter(1536, 512, { v1: { matrix: projMatrix, bias: projBias } });
const compressedVector = projAdapter.tune(baseVector, "v1");

3. Neural Nets & Space Optimization

import { MlpAdapter, WhiteningAdapter } from "warpvector/ml";
import { SoftWhiteningAdapter } from "warpvector/train";

// 1. MlpAdapter: Ultra-fast non-linear inference via WASM
const mlp = new MlpAdapter([{ matrix, bias, activation: "relu" }]);
await mlp.init();
const mlpOutput = mlp.tune(inputVector);

// 2. Whitening: Remove online spatial bias (anisotropy)
const whitener = new WhiteningAdapter(1536, {
  learningRate: 0.01,
  numComponents: 1,
});
whitener.update(rawVector); // Streaming PCA
const whitened = whitener.tune(searchVector);

// 3. Inverse Diffusion: Extract sharp intent from mixed contexts
const softWhitener = new SoftWhiteningAdapter(1536, { tau: 2.0 });
const sharpVector = softWhitener.tune(queryVector);

4. Auto-Learning & Federated Learning (Backend Layer)

import {
  IntentMatrixFactory,
  InfoNCETrainer,
  FeedbackCollector,
} from "warpvector/train";

// 1. IntentMatrixFactory: Auto-generate matrices from samples 🆕
const factory = new IntentMatrixFactory(1536);
factory.addCategory("tech", [techVec1, techVec2]);
const intents = await factory.build(); // Generated via InfoNCE loss

// 2. Feedback & Training: Generate training data from logs
const collector = new FeedbackCollector({ dwellThresholdMs: 3000 });
// ... (collect logs)
const trainer = new InfoNCETrainer(1536);
const updatedWeights = await trainer.updateOnline(
  currentWeights,
  collector.toTripletExamples()[0],
  { learningRate: 0.001 },
);

5. Advanced Search Algorithms

import {
  QuantizationAdapter,
  rrf,
  ColbertAdapter,
  VsaAdapter,
} from "warpvector";

// 1. Quantization: Int8 (1/4 size) or Binary (1/32 size)
const int8Adapter = new QuantizationAdapter({ type: "int8", dim: 1536 });
const int8Vec = int8Adapter.encode(floatVector);

// 2. Hybrid Search (RRF): Merge Dense & Sparse (BM25) results
const rrfResults = rrf([denseResults, sparseResults]);

// 3. ColBERT: WASM-accelerated MaxSim token matching
const colbert = new ColbertAdapter();
const ranks = colbert.rank(queryTokens, [doc1Tokens, doc2Tokens], 1536);

// 4. VSA (Vector Symbolic Architecture): Bundle and bind vectors
const bundled = VsaAdapter.bundle([scienceVec, technologyVec]);
const bound = VsaAdapter.bind(keyVec, valueVec);

6. Ecosystem Integrations

import { PrismaClient } from "@prisma/client";
import { withWarpVector } from "warpvector/prisma";

const prisma = new PrismaClient().$extends(
  withWarpVector({ adapter: myAdapter, vectorField: "embedding" }),
);
const results = await prisma.document.searchByVector({
  vector: rawVector,
  topK: 10,
});

import { WarpEmbeddings } from "warpvector/langchain";
const warpEmbeddings = new WarpEmbeddings({
  baseEmbeddings,
  adapter,
  intentName: "domain_x",
});

import { VectorDBAdapter } from "warpvector";
const tunedVector = await pipeline.run(queryEmbedding);
const { vector, options } = VectorDBAdapter.toVectorizeQuery(tunedVector, 10);
const results = await env.VECTORIZE_INDEX.query(vector, options);

📚 Cookbooks (Practical Examples)

See the examples/ and docs/cookbook/ directories for drop-in solutions:

1. Secure RAG Pipeline (Anomaly Detection & Safe Compression: AnomalyDetectionAdapter + SafeQuantizationAdapter)
2. MoE and Auto-Tuning
3. Cross-Encoder Training for Rerankers
4. E-commerce Search Cookbook
5. Cost-efficient RAG with Pinecone
6. Cloudflare Edge Execution

📖 Documentation

🔍 Debugging & Observability

// Debug intermediate steps
const debug = pipeline.dryRun(testVector, { intent: "tech" });

// OpenTelemetry compatible tracing
import { WarpTracer } from "warpvector";
const tracer = new WarpTracer();
const warped = tracer.trace("intent.tune", { intent: "tech" }, () =>
  adapter.tune(vector, "tech"),
);
console.log(tracer.getMetrics());

📐 Mathematical Background

Given a base embedding vector $\mathbf{x} \in \mathbb{R}^d$, WarpVector applies an affine map:

$$\mathbf{x}' = \sigma(\mathbf{W}_I \mathbf{x} + \mathbf{b}_I)$$

• $\mathbf{W}_I \in \mathbb{R}^{d \times d}$: Intent transformation matrix (rotation, scaling, shearing)
• $\mathbf{b}_I \in \mathbb{R}^d$: Intent bias vector (translation)
• $\sigma$: Non-linear activation function (ReLU, Sigmoid, Tanh)

Computational complexity is $\mathcal{O}(d^2)$ (or $\mathcal{O}(d \cdot r)$ with LoRA), optimized via WASM and Float32Array memory alignment for sub-millisecond inference on edge devices.

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

• 🐛 Bug Reports
• 💡 Feature Requests
• 📖 Documentation improvements
• 🧪 New adapters and integrations

📄 License

MIT License