Package Exports

@jscpd/tokenizer
@jscpd/tokenizer/dist/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (@jscpd/tokenizer) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

`@jscpd/tokenizer`

tokenizer is package from @jscpd used for convert programming code to list of tokens

Installation

npm install @jscpd/tokenizer --save

Usage

import {IOptions,  ITokensMap} from '@jscpd/core';
import {Tokenizer} from '@jscpd/tokenizer';

const tokenizer = new Tokenizer();
const options: IOptions = {};

const maps: ITokensMap[] = tokenizer.generateMaps('source_id', 'let a = "11"', 'javascript', options);

ga tracker

@jscpd/tokenizer