Package Exports

raglite
raglite/dist/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (raglite) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

RAGLite

RAGLite is a minimal, TypeScript-first Retrieval-Augmented Generation (RAG) pipeline. It's designed for simplicity, extensibility, and performance, supporting PDF and DOCX ingestion, local embeddings, and fast vector search using SQLite/LibSQL.

Features

Simple API: to load and search for documents.
Document Ingestion: Load PDFs and DOCX files out of the box.
Local Embeddings: Uses local embedding model from Hugging Face.
Fast Vector Search: Backed by SQLite/LibSQL for efficient similarity search.
Extensible: Advanced users can use or extend individual components.

Installation

npm install raglite

Quick Start

Basic Usage

Add documents to the data store:

import { load, search } from "raglite";

// Load a document from a file path
const records = await load("path/to/document.pdf");

records; // The chunked records created from the document
records[0].content; // The content of the document chunk
records[0].metadata; // The metadata of the document chunk
records[0].vector; // The vector of the document chunk
records[0].id; // The database id of the document chunk

// Load a document from a URL
await load("https://example.com/path/to/document.docx");

// Load a document from text
await load("Hello, world!");

// Include metadata with the document
await load("Hello, world!", {
  source: "https://example.com/path/to/document.docx",
});

Search for relevant documents:

const results = await search("What is retrieval-augmented generation?");

console.log(results); // [{ content: "...", metadata: { source: "..." } }, ...]

Environment Variables

You can customize the pipeline by passing in your own components.

Variable	Default	Description
`DATABASE_URL`	`file:data/ragpipe.db`	The URL of the SQLite/LibSQL database
`TABLE_NAME`	`embeddings`	The name of the table to store the embeddings
`DIMENSIONS`	`384`	The dimensions of the embeddings
`MODEL`	`sentence-transformers/all-MiniLM-L12-v2`	The model to use for embedding
`CHUNK_SIZE`	`200`	The maximum number of words to embed per chunk
`OVERLAP`	`0`	The number of words to overlap between chunks

Advanced Usage

Customizing the Pipeline

You can customize the pipeline by passing in your own components.

import {
  Pipeline,
  UrlLoader,
  FileLoader,
  PdfLoader,
  DocxLoader,
  EmbeddingLoader,
  DataStoreLoader,
} from "raglite";

// Create a writer pipeline
const writer = new Pipeline([
  new UrlLoader({
    headers: {
      Authorization: `Bearer ${process.env.TOKEN}`,
    },
  }), // add fetch request options
  new FileLoader(),
  new PdfLoader(),
  new DocxLoader(),
  new EmbeddingLoader({
    model: "sentence-transformers/all-MiniLM-L6-v2",
    chunkSize: 200,
    overlap: 10,
  }), // customize the embedding model
  new DataStoreLoader({
    databaseUrl: "path/to/database.db",
    tableName: "documents",
    dimensions: 384,
  }), // customize the data store
]);

const records = await writer.load("path/to/document.pdf");

// Create a reader pipeline
const reader = new Pipeline([
  new EmbeddingLoader(),
  new DataStoreLoader({ search: true }),
]);

const results = await reader.search("What is retrieval-augmented generation?");

Requirements

Node.js >= 18 (recommended: >= 20)
SQLite/LibSQL database (local file or remote)

Development & Testing

Run tests: pnpm test or npm test
Build: pnpm build or npm run build

License

MIT

Contributing

Contributions, issues, and feature requests are welcome! Please open an issue or PR.