Package Exports

@nlptools/splitter
@nlptools/splitter/dist/index.mjs

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (@nlptools/splitter) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

@nlptools/splitter

npm version npm license

Text splitting utilities - LangChain.js text splitters wrapper for NLPTools

This package provides convenient access to LangChain.js text splitting utilities through the NLPTools ecosystem. It includes various text splitters for chunking documents and processing large texts.

Installation

# Install with npm
npm install @nlptools/splitter

# Install with yarn
yarn add @nlptools/splitter

# Install with pnpm
pnpm add @nlptools/splitter

Usage

Basic Setup

import {
  RecursiveCharacterTextSplitter,
  CharacterTextSplitter,
  MarkdownTextSplitter,
  TokenTextSplitter,
} from "@nlptools/splitter";

Available Splitters

RecursiveCharacterTextSplitter - Splits text recursively using different separators
CharacterTextSplitter - Splits text by character count
MarkdownTextSplitter - Specialized splitter for Markdown documents
TokenTextSplitter - Splits text by token count
LatexTextSplitter - Specialized splitter for LaTeX documents

Example Usage

import { RecursiveCharacterTextSplitter } from "@nlptools/splitter";

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 200,
});

const text = "Your long text content here...";
const chunks = await splitter.splitText(text);
console.log(chunks);

Features

📝 Multiple Splitting Strategies: Character, token, and format-aware splitting
🔧 Configurable: Customizable chunk size and overlap
📦 TypeScript First: Full type safety
🚀 Based on LangChain.js: Reliable and well-tested implementations

References

This package incorporates and builds upon the following excellent open source projects:

LangChain.js Text Splitters - Core text splitting implementations via @langchain/textsplitters