Package Exports
- strtok3
- strtok3/lib/AbstractTokenizer
- strtok3/lib/core
- strtok3/lib/type
This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (strtok3) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.
Readme
strtok3
A promise based streaming tokenizer for NodeJS. This node module is a successor of strtok2.
Usage
The strtok3 contains one class, ReadStreamTokenizer, which is constructed with a
a stream.Readable.
The ReadStreamTokenizer has one method readToken which takes a token as an argument
and returns a Promise resolving the decoded token value.
The token is basically a description what to read form the stream. A basic set of token types can be found here: token-types.
Examples
Below is an example of parsing the the first byte from a readable stream as an unsigned-integer:
import * as strtok3 from "strtok3";
import * as Token from "token-types";
let readableStream; // stream.Readable;
strtok3.fromStream(readableStream).then(tokenizer => {
return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});
})The same can be done from a file:
strtok3 methods
All of the strtok3 methods return a tokenizer, either directly or via a promise.
Method strtok3.fromFile()
Returns, via a promise, a tokenizer which can be used to parse a file.
import * as strtok3 from "strtok3";
import * as Token from "token-types";
strtok3.fromFile("somefile.bin").then(tokenizer => {
return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});
})Method strtok3.fromStream()
Create tokenizer from a node.js readable stream.
| Parameter | Type | Description |
|---|---|---|
| stream | Readable | Stream to read from |
strtok3.fromStream(stream).then(tokenizer => {
return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});
})Returns a tokenizer, via a Promise, which can be used to parse a buffer.
Method strtok3.fromBuffer()
Returns a tokenizer which can be used to parse a buffer.
import * as strtok3 from "strtok3";
const tokenizer = strtok3.fromBuffer(buffer);
tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});Tokenizer methods
Method tokenizer.readBuffer()
Read buffer from stream.
readBuffer(buffer, offset, length, position)| Parameter | Type | Description |
|---|---|---|
| buffer | Buffer | Uint8Array | Target buffer to write the data read to |
| offset | number | The offset in the buffer to start writing at; if not provided, start at 0 |
| length | number | An integer specifying the number of bytes to read |
| position? | number | An integer specifying where to begin reading from in the file. If position is null, data will be read from the current file position. |
Return value Promise<number> Promise number of bytes read
Method tokenizer.peekBuffer()
Peek (read ahead) buffer from tokenizer
peekBuffer(buffer, offset, length, position, maybeless)| Parameter | Type | Description |
|---|---|---|
| buffer | Buffer | Uint8Array | Target buffer to write the data read `(peeked) to |
| offset | number | Is the offset in the buffer to start writing at; if not provided, start at 0 |
| length | number | Is an integer specifying the number of bytes to read |
| position? | number | Is an integer specifying where to begin reading from in the file. If position is null, data will be read from the current file position. |
| maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
Return value Promise<number>
Browser
To exclude fs based dependencies, you can use:
import * as strtok3 from 'strtok3/lib/core';| function | 'strtok3' | 'strtok3/lib/core' |
|---|---|---|
parseBuffer |
✓ | ✓ |
parseStream |
✓ | ✓ |
parseFromTokenizer |
✓ | ✓ |
fromFile |
✓ |
If you plan to use fromStream you need to polyfill:
- buffer: buffer
- stream: web-streams-polyfill