@lov3kaizen/agentsea-ingest

TypeScript-native document processing pipeline for AI/RAG applications.

Features

Multi-format Parsing: PDF, DOCX, HTML, Markdown, CSV, Excel, JSON
Intelligent Chunking: Fixed, recursive, sentence, paragraph, semantic, hierarchical
Table & Image Extraction: Automatic extraction with metadata
Text Cleaning: Normalization, deduplication, PII removal
Flexible Pipelines: Configurable processing stages
Streaming Support: Process large documents efficiently

Installation

pnpm add @lov3kaizen/agentsea-ingest

Quick Start

import { createIngester, pipelines } from '@lov3kaizen/agentsea-ingest';

// Simple ingestion
const ingester = createIngester();
const doc = await ingester.ingestFile('./document.pdf');
console.log(`Extracted ${doc.chunks.length} chunks`);

// RAG-optimized pipeline
const pipeline = pipelines.rag().build();
const result = await pipeline.process({ path: './document.md' });

Parsing Documents

Supported Formats

Format	Parser	MIME Types
PDF	PDFParser	application/pdf
DOCX	DOCXParser	application/vnd.openxmlformats-officedocument.wordprocessingml.document
HTML	HTMLParser	text/html
Markdown	MarkdownParser	text/markdown
Text	TextParser	text/plain
CSV	CSVParser	text/csv
Excel	ExcelParser	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
JSON	JSONParser	application/json

Direct Parser Usage

import {
  createPDFParser,
  createMarkdownParser,
} from '@lov3kaizen/agentsea-ingest';
import { readFileSync } from 'fs';

const pdfParser = createPDFParser();
const buffer = readFileSync('./document.pdf');
const result = await pdfParser.parse(buffer);

console.log(result.text);
console.log(result.elements);
console.log(result.tables);

Chunking Strategies

Fixed Size

import { createFixedChunker } from '@lov3kaizen/agentsea-ingest';

const chunker = createFixedChunker();
const chunks = chunker.chunk(text, {
  maxTokens: 512,
  overlap: 50,
  splitOnSentences: true,
});

Recursive

import { createRecursiveChunker } from '@lov3kaizen/agentsea-ingest';

const chunker = createRecursiveChunker();
const chunks = chunker.chunk(text, {
  maxTokens: 512,
  separators: ['\n\n', '\n', '. ', ' '],
  keepSeparator: true,
});

Semantic

import { createSemanticChunker } from '@lov3kaizen/agentsea-ingest';

const chunker = createSemanticChunker();
const chunks = await chunker.chunk(text, {
  maxTokens: 512,
  similarityThreshold: 0.5,
  embedFunction: async (text) => myEmbeddingModel(text),
});

Hierarchical

import { createHierarchicalChunker } from '@lov3kaizen/agentsea-ingest';

const chunker = createHierarchicalChunker();
const chunks = chunker.chunk(markdownText, {
  maxTokens: 512,
  headingLevels: [1, 2, 3],
  includeParentContext: true,
});

Pipeline Builder

import { createPipelineBuilder } from '@lov3kaizen/agentsea-ingest';

const pipeline = createPipelineBuilder()
  .withName('my-pipeline')
  .withStages(['load', 'parse', 'clean', 'chunk', 'embed'])
  .withChunking({
    strategy: 'semantic',
    maxTokens: 512,
    overlap: 50,
  })
  .withCleaning({
    operations: ['normalize_whitespace', 'remove_urls', 'trim'],
  })
  .withCallbacks({
    onDocumentComplete: (doc) => console.log(`Processed: ${doc.id}`),
  })
  .build();

const result = await pipeline.process({ path: './document.pdf' });

Pre-built Pipelines

import { pipelines } from '@lov3kaizen/agentsea-ingest';

// Simple text extraction
const simple = pipelines.simple().build();

// Full processing with all stages
const full = pipelines.full().build();

// RAG-optimized pipeline
const rag = pipelines.rag().build();

// Document analysis (no chunking)
const analysis = pipelines.analysis().build();

// OCR pipeline for scanned documents
const ocr = pipelines.ocr().build();

Ingester

The Ingester class provides a high-level API for document ingestion:

import { createIngester } from '@lov3kaizen/agentsea-ingest';

const ingester = createIngester({
  chunking: {
    strategy: 'recursive',
    maxTokens: 512,
  },
  concurrency: 4,
  fileSizeLimit: 10 * 1024 * 1024, // 10MB
});

// Ingest single file
const doc = await ingester.ingestFile('./document.pdf');

// Ingest from URL
const webDoc = await ingester.ingestUrl('https://example.com/page.html');

// Ingest from buffer
const bufferDoc = await ingester.ingestBuffer(buffer, 'document.pdf');

// Ingest directory
const results = await ingester.ingestDirectory('./documents', {
  recursive: true,
  include: ['*.pdf', '*.docx'],
  exclude: ['draft-*'],
});

Watch Mode

const ingester = createIngester({
  watchMode: {
    enabled: true,
    paths: ['./documents'],
    include: ['*.pdf', '*.md'],
    debounceDelay: 1000,
    processExisting: true,
  },
});

ingester.startWatching();
// Files added/modified in ./documents will be automatically processed

Events

const pipeline = createPipeline(config);
const emitter = pipeline.getEventEmitter();

emitter.on('document:loaded', (event) => {
  console.log(`Loaded: ${event.documentId}`);
});

emitter.on('document:chunked', (event) => {
  console.log(`Created ${event.chunkCount} chunks`);
});

emitter.on('document:completed', (event) => {
  console.log(`Completed: ${event.document.id}`);
});

API Reference

Types

ProcessedDocument - Processed document with chunks and metadata
Chunk - Text chunk with metadata and optional embedding
Element - Document element (paragraph, heading, list, etc.)
TableData - Extracted table data
ImageData - Extracted image data
PipelineConfig - Pipeline configuration options
ChunkingOptions - Chunking configuration options

Core Classes

Pipeline - Document processing pipeline
PipelineBuilder - Fluent pipeline builder
Ingester - High-level document ingester
ParserRegistry - Parser management
ChunkerRegistry - Chunker management

Parsers

PDFParser - PDF document parsing
DOCXParser - Word document parsing
HTMLParser - HTML document parsing
MarkdownParser - Markdown parsing
TextParser - Plain text parsing
CSVParser - CSV file parsing
ExcelParser - Excel file parsing
JSONParser - JSON file parsing

Chunkers

FixedChunker - Fixed-size chunks
RecursiveChunker - Recursive splitting
SentenceChunker - Sentence-based chunks
ParagraphChunker - Paragraph-based chunks
SemanticChunker - Semantic similarity-based
HierarchicalChunker - Heading-based hierarchy

License

MIT