JSPM

Textmate token-based language service for Visual Studio Code.

A TypeScript library for building powerful, composable parsers using the parser combinator pattern. Parsco provides small, reusable parsing tools that can be combined to create complex grammars and interpreters, all without separate tokenization or extern

Official AstroPay client for easy frontend integration to Tokenizer

Bedrock Tokenization Engine

TypeScript SDK for AddPay Cloud API - CNP payment processing

text segmentation into sentences

RWA Token SDK for the development of permissioned tokens on SVM blockchains.

A high-performance wrapper around `Intl.Segmenter` for efficient text segmentation. This class resolves memory handling issues seen with large strings and can enhance performance by 50-500x. Only ~70 loc (with comments) and no dependencies.

A Bedrock module to provide auto-rotating tokenizers

Salient is a natural language processing and sentiment analysis library

Bedrock Tokenized Cache

Bedrock VC Issuer Coordinator Storage

A parser and analyzer for advanced search queries, supporting tokenization, abstract syntax tree generation, and condition extraction.

A library for natural language processing tasks including tokenization, entity recognition, anonymization, semantic chunking, and much more.

High-performance library for tokenizing text.

Use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.

Fat Zebra payment processing library for Next.js and standalone environments

es6 version of aaditmshah/lexer

tokenizing text by rules

## How To Use

Taiwanese Hokkien Transliterator and Tokeniser

A WebAssembly binding for the charabia multilingual text tokenizer used by Meilisearch.

Here is a README generated from the code snippet:

NLP (natural language processing) for server and the browser in TypeScript. All lightweight and super-fast.

Tools for manipulating sentences

Library that allows you to make payments and tokenize debit and credit cards with Niubiz.

A simple iterative lexer written in TypeScript

Adebiet

Tokenization Service's Smart Contracts

A Bedrock module to provide auto-rotating tokenizers

A powerful and flexible text search library for JavaScript that enables you to build a simple text search engine.

Break down text into array of words.

Official SDK for MAS Hub - Enterprise blockchain platform built on MasChain

Jawi

POS tokenization of words int meaningful components usable in POS-Bayes & Elastic Search Indexes

MCP server implementation for XRP Ledger

A tokenizer and lemmatizer for canonical terms in text

An interface to the pruf network written in javascript

TokenEX Node.js library for API

CLI tool to perform NLP on selected files

Well typed, string transmutation via splicing, driven by definitions of regular expressions, priority match and recursive clauses

MCP library for AI-assisted Real World Asset tokenization on XRPL

Salient is a natural language processing and sentiment analysis library

Bedrock Tokenization Engine

Professional SDK for tokenized assets and security tokens on blockchain

Define a function to execute after completing token-defined tasks.

a lexer in javascript

Basic English tokenizer

Tools for working with human language data.

package to deploy the smart contracts that associate the tokens to the various clients proportional to their contribution

Multi-criteria Cantonese segmentation with dashes, intermediates, pipes, and spaces.

This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.

a wrapper around the LunaSec CLI enabling it to be used as an NPM package

A reverse engineered Node.js client for the Blackbox.ai API, supporting chat completions with streaming and aggregated responses.

A JSON parser implemented in TypeScript

Functions for filtering SQL-like json with conjunctive normal form filters

A simple and efficient tokenizer for natural language processing tasks.

Modular backend service for tokenization and smart contracts in real estate applications

#### Description I needed the SentenceSplitter from llamaindex but had to import the entire llamaindex package which is 1GB. I pulled it out and had GPT make a standalone version. It's not exactly the same but close.

a wrapper around the LunaSec CLI enabling it to be used as an NPM package