fasttext.wasm.js

Node and Browser env supported WebAssembly version of fastText: Library for efficient text classification and representation learning.

WebAssembly version of fastText with compressed lid.176.ftz model (~900KB) and a typescript wrapper. This project focuses on cross-platform, zero-dependency and out-of-the-box.

Features

• Written in TypeScript
• Supported Node, Worker, Browser and Browser extension runtime
• Integrated language identification

Usage

In Node.js, you should use this approach for binding js best performance.

import { getLIDModel } from 'fasttext.wasm.js'

const lidModel = await getLIDModel()
await lidModel.load()
const result = await lidModel.identify('Hello, world!')
console.log(result) // 'en'

In others environments, use like below:

import { getLIDModel } from 'fasttext.wasm.js/common'

const lidModel = await getLIDModel()
// Default paths:
// {
//   wasmPath: '/fastText/fasttext.common.wasm',
//   modelPath: '/fastText/models/lid.176.ftz',
// }
await lidModel.load()
const result = await lidModel.identify('Hello, world!')
console.log(result) // 'en'

Do not forget that download and place /fastText/fasttext.common.wasm and /fastText/models/lid.176.ftz in public root directory. You can override the default paths if necessary.

Benchmark

codesandbox/language-detect-benchmark

• 10x Faster and very more accurate than languagedetect
• ~3x Faster and more accurate than franc

Related

• awesome-exhibition/fasttext.wasm.js - Next.js example.
• yunsii/browser-extension-with-fasttext.wasm.js - Use fasttext.wasm.js in browser extension.

Credits

• DreamOfIce/fastText.wasm
• facebookresearch/fastText/webassembly

References

• Language identification
• Language identification resources

Build & Publish

Requirements

• emsdk
• xmake

Pay attention, add source ./emsdk_env.sh to shell profile to auto load emsdk env, and export EMSDK_QUIET=1 can be used to suppress these messages.

• npm run build
• npx changeset
• npx changeset version
• git commit
• npx changeset publish
• git push --follow-tags

changeset prerelease doc

License

MIT License © 2023 Yuns