JSPM

  • ESM via JSPM
  • ES Module Entrypoint
  • Export Map
  • Keywords
  • License
  • Repository URL
  • TypeScript Types
  • README
  • Created
  • Published
  • Downloads 4377
  • Score
    100M100P100Q126802F
  • License MIT

Package Exports

  • tiktoken-node
  • tiktoken-node/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (tiktoken-node) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

⏳ tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

const tiktoken = require('tiktoken-node')
let enc = tiktoken.getEncoding("gpt2")
console.assert(enc.decode(enc.encode("hello world")) == "hello world")

// To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encodingForModel("text-davinci-003")

The open source version of tiktoken-node can be installed from npm:

npm install tiktoken-node