tokenize-is

TypeScript tokenizer for Icelandic text. A port of Miðeind's Tokenizer.

Installation

npm install tokenize-is
# or
pnpm add tokenize-is

Usage

import { tokenize, splitIntoSentences } from "tokenize-is";

// Tokenize text
const tokens = tokenize("Kl. 14:30 komu 100 gestir.");
for (const token of tokens) {
  if (token.kind === "word") {
    console.log(token.text);
  } else if (token.kind === "number") {
    console.log(token.value); // parsed number
  }
}

// Split into sentences
const sentences = splitIntoSentences("Fyrst. Síðan.");
// → ["Fyrst.", "Síðan."]

Token Types

All tokens have a kind discriminator for TypeScript narrowing:

Options

tokenize(text, {
  replaceCompositeGlyphs: true, // Normalize Unicode (a + ́ → á)
  includeSentenceMarkers: false, // Add s_begin/s_end tokens
  includeOffsets: false, // Add span.start/end character offsets
});

Token Offsets

When includeOffsets: true, each token includes a span with character positions:

const tokens = tokenize("Halló heimur", { includeOffsets: true });
// tokens[0] = { kind: "word", text: "Halló", span: { start: 0, end: 5 } }
// tokens[1] = { kind: "word", text: "heimur", span: { start: 6, end: 12 } }

// Extract original text from spans
const text = "Halló heimur";
text.slice(tokens[0].span.start, tokens[0].span.end); // "Halló"

Port Fidelity

This is a TypeScript port of Miðeind's Tokenizer (MIT licensed).

Supported

• All 30 token types from the original
• Sentence boundary detection with abbreviation awareness
• Unicode normalization (composite glyphs)
• Icelandic number formats (1.234,56)
• Spelled-out time expressions (hálftvö → 1:30)
• ~100 Icelandic abbreviations
• 70+ SI units, 18+ currencies
• Kennitala (SSN) validation with checksum

Not Yet Implemented

• detokenize() - reconstruct text from tokens
• correct_spaces() - fix spacing between tokens
• paragraphs() / mark_paragraphs() - paragraph handling
• HTML entity unescaping (á → á)
• Full abbreviation list (300+ in original vs ~100 here)

Design Differences

• ESM-only (no CommonJS)
• Returns arrays instead of generators
• Discriminated unions instead of numeric token codes
• Zero runtime dependencies

Development

pnpm install
pnpm test        # Run tests
pnpm build       # Build with tsdown
pnpm check       # Lint + format + typecheck

License

MIT - same as the original Tokenizer.