Package Exports

strtok3
strtok3/lib/AbstractTokenizer
strtok3/lib/core
strtok3/lib/type

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (strtok3) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

strtok3

A promise based streaming tokenizer for NodeJS. This node module is a successor of strtok2.

Usage

The strtok3 contains one class, ReadStreamTokenizer, which is constructed with a a stream.Readable.

The ReadStreamTokenizer has one method readToken which takes a token as an argument and returns a Promise resolving the decoded token value.

The token is basically a description what to read form the stream. A basic set of token types can be found here: token-types.

Examples

Below is an example of parsing the the first byte from a readable stream as an unsigned-integer:

import * as strtok3 from "strtok3";
import * as Token from "token-types";
    
let readableStream; // stream.Readable;

strtok3.fromStream(readableStream).then(tokenizer => {
  return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
    console.log("My number: %s", myUint8Number);
  });
})

The same can be done from a file:

import * as strtok3 from "strtok3";
import * as Token from "token-types";
    
strtok3.fromFile("somefile.bin").then((tokenizer) => {
  return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
    console.log("My number: %s", myUint8Number);
  });
})

Read from a Buffer:

    
strtok3.fromBuffer(buffer).then((tokenizer) => {
  return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
    console.log("My number: %s", myUint8Number);
  });
})

Browser

To exclude fs based dependencies, you can use:

import * as strtok3 from 'strtok3/lib/core';

function	'strtok3'	'strtok3/lib/core'
`parseBuffer`	✓	✓
`parseStream`	✓	✓
`parseFromTokenizer`	✓	✓
`fromFile`	✓

If you plan to use fromStream you need to polyfill:

buffer: buffer
stream: web-streams-polyfill