CSS Tokenizer

Implemented from : https://drafts.csswg.org/css-syntax/

API

Usage

Add CSS Tokenizer to your project:

npm install @csstools/css-tokenizer --save-dev

import { tokenize } from '@csstools/css-tokenizer';

const myCSS =  `@media only screen and (min-width: 768rem) {
    .foo {
        content: 'Some content!' !important;
    }
}
`;

const tokens = tokenize({
    css: myCSS,
});

console.log(tokens);

Or use the streaming interface:

import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const myCSS = `@media only screen and (min-width: 768rem) {
    .foo {
        content: 'Some content!' !important;
    }
}
`;

const t = tokenizer({
    css: myCSS,
});

while (true) {
    const token = t.nextToken();
    if (token[0] === TokenType.EOF) {
        break;
    }

    console.log(token);
}

Options

{
    onParseError?: (error: ParseError) => void
}

`onParseError`

The tokenizer is forgiving and won't stop when a parse error is encountered.

To receive parsing error information you can set a callback.

import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const t = tokenizer({
    css: '\\',
}, { onParseError: (err) => console.warn(err) });

while (true) {
    const token = t.nextToken();
    if (token[0] === TokenType.EOF) {
        break;
    }
}

Parser errors will try to inform you where in the tokenizer logic the error happened. This tells you what kind of error occurred.

Order of priorities

specification compliance
correctness
reliability
tokenizing and serializing must round trip losslessly
exposing useful aspects about the source code
runtime performance
package size