Package Exports

tokenizers

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (tokenizers) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

elasticsearch tokenizers

const { standardTokenizer } = require('tokenizers');
const tokens = standardTokenizer("Düsseldorf, Köln, Москва, 北京市, !@#$");

tokens ~ [ 'düsseldorf', 'köln', 'москва', '北', '京', '市' ];

const { asciiFoldingTokenizer } = require('tokenizers');
const tokens = asciiFoldingTokenizer("Düsseldorf, Köln, Москва, 北京市, !@#$");

tokens ~ [ 'dusseldorf', 'koln', 'москва', '北', '京', '市' ];