restream

Regular expression detection implemented as a Transform steam; and regex-based buffer replacement stream to replace incoming data on-the-fly.

yarn add -E restream

Table of Contents

• Table of Contents
• API
• restream(regex: RegExp): Transform
• Replaceable Class
  • Rule Type
    • re*
    • replacement*
      • String Replacement
      • Function Replacer
      • Async Function Replacer
  • constructor(rule: Rule|Rules[]): Replaceable
  • Replacer Context
  • Replacer Errors
• Markers
  • makeMarkers(matchers: { [name]: RegExp }, config?: MakeMarkersConfig): { [name]: Marker }
    • MakeMarkersConfig
  • makeCutRule(marker: Marker): Rule
  • makePasteRule(marker: Marker): Rule
  • Accessing Replacements

API

The package contains the default restream and Replaceable functions, as well as functions to create markers and their cut and paste rules.

import restream, {
  Replaceable,
  makeMarkers, makeCutRule, makePasteRule,
} from 'restream'

restream(
  regex: RegExp,
): Transform

Create a Transform stream which will buffer incoming data and push regex results when matches can be made, i.e. when regex.exec returns non-null value. When the g flag is added to the regex, multiple matches will be detected.

/** yarn example/restream.js */
import restream from 'restream'
import { createReadable, createWritable } from './lib'

(async () => {
  try {
    const rs = createReadable('test-string-{12345}-{67890}')

    const stream = restream(/{(\d+)}/g) // create a transform stream
    rs.pipe(stream)

    const { data, ws } = createWritable()
    stream.pipe(ws)

    ws.once('finish', () => {
      console.log(data)
    })
  } catch (err) {
    console.error(err)
  }
})()

[ [ '{12345}',
    '12345',
    index: 12,
    input: 'test-string-{12345}-{67890}' ],
  [ '{67890}',
    '67890',
    index: 20,
    input: 'test-string-{12345}-{67890}' ] ]

Replaceable Class

A Replaceable transform stream can be used to transform data according to a single or multiple rules.

Rule Type

Replaceable uses rules to determine how to transform data. Below is the description of the Rule type.

String Replacement

Replacement as a string. Given a simple string, it will replace a match detected by the rule's regular expression, without consideration for the capturing groups.

Function Replacer

Replacement as a function. See MDN for more documentation on how the replacer function should be implemented.

The example below allows to replace strings like %NPM: documentary% and %NPM: @rqt/aqt% into a markdown badge (used in documentary).

const syncRule = {
  re: /^%NPM: ((?:[@\w\d-_]+\/)?[\w\d-_]+)%$/gm,
  replacement(match, name) {
    const n = encodeURIComponent(name)
    const svg = `https://badge.fury.io/js/${n}.svg`
    const link = `https://npmjs.org/package/${name}`
    return `[![npm version](${svg})](${link})`
  },
}

Async Function Replacer

An asynchronous function to get replacements. The stream won't push any data until the replacer's promise is resolved. Due to implementation details, the regex will have to be run against incoming chunks twice, therefore it might be not ideal for heavy-load applications with many matches.

This example will replace strings like %FORK-js: example example/Replaceable.js% into the output of a forked JavaScript program (used in documentary).

import { fork } from 'spawncommand'

const codeSurround = (m, lang = '') =>
  `\`\`\`${lang}\n${m.trim()}\n\`\`\``

const forkRule = {
  re: /%FORK(?:-(\w+))? (.+)%/mg,
  async replacement(match, lang, m) {
    const [mod, ...args] = m.split(' ')
    const { promise } = fork(mod, args, {
      execArgv: [],
      stdio: 'pipe',
    })
    const { stdout } = await promise
    return codeSurround(stdout, lang)
  },
}

constructor(
  rule: Rule|Rules[],
): Replaceable

Create a Transform stream which will make data available when an incoming chunk has been updated according to the specified rule or rules.

Matches can be replaced using a string, function or async function. When multiple rules are passed as an array, the string will be replaced multiple times if the latter rules also modify the data.

/** yarn example/Replaceable.js */
import Catchment from 'catchment'
import { Replaceable } from 'restream'
import { createReadable } from './lib'

const dateRule = {
  re: /%DATE%/g,
  replacement: new Date().toLocaleString(),
}

const emRule = {
  re: /__(.+?)__/g,
  replacement(match, p1) {
    return `<em>${p1}</em>`
  },
}

const authorRule = {
  re: /^%AUTHOR_ID: (.+?)%$/mg,
  async replacement(match, id) {
    const name = await new Promise(resolve => {
      // pretend to lookup author name from the database
      const authors = { 5: 'John' }
      resolve(authors[id])
    })
    return `Author: <strong>${name}</strong>`
  },
}

const STRING = `
Hello __Fred__, your username is __fred__.
You have __5__ stars.

%AUTHOR_ID: 5%
on __%DATE%__
`

;(async () => {
  try {
    const replaceable = new Replaceable([
      dateRule,
      emRule,
      authorRule,
    ])
    const rs = createReadable(STRING)
    rs.pipe(replaceable)

    const { promise } = new Catchment({
      rs: replaceable,
    })
    const res = await promise

    console.log(res)
  } catch (err) {
    console.error(err)
  }
})()

Output:

Hello <em>Fred</em>, your username is <em>fred</em>.
You have <em>5</em> stars.

Author: <strong>John</strong>
on <em>2018-8-10 02:25:11</em>

Replacer Context

Replacer functions will be executed with their context set to a Replaceable instance to which they belong. Both sync and async replacers can use the this keyword to access their Replaceable instance and modify its properties and/or emit events. This is done so that there's a mechanism by which replacers can share data between themselves.

For example, we might want to read and parse an external file first, but remember its data for use in following replacers.

Given an external file example/types.json:

{
  "TypeA": "A new type with certain properties.",
  "TypeB": "A type to represent the state of the world."
}

Replaceable can read it in the first typesRule rule, and reference its data in the second paramRule rule:

/** yarn example/context.js */
import Catchment from 'catchment'
import { createReadStream } from 'fs'
import { Replaceable } from 'restream'
import { createReadable } from './lib'

const typesRule = {
  re: /^%types: (.+?)%$/mg,
  async replacement(match, location) {
    const rs = createReadStream(location)
    const { promise } = new Catchment({ rs })
    const d = await promise
    const j = JSON.parse(d)

    this.types = j // remember types for access in following rules
    return match
  },
}

const paramRule = {
  re: /^ \* @typedef {(.+?)} (.+)(?: .*)?/mg,
  replacement(match, type, typeName) {
    const description = this.types[typeName]
    if (!description) return match
    return ` * @typedef {${type}} ${typeName} ${description}`
  },
}

const STRING = `
%types: example/types.json%

/**
 * @typedef {Object} TypeA
 */
`

;(async () => {
  try {
    const replaceable = new Replaceable([
      typesRule,
      paramRule,
    ])
    const rs = createReadable(STRING)
    rs.pipe(replaceable)

    const { promise } = new Catchment({
      rs: replaceable,
    })
    const res = await promise

    console.log(res)
  } catch (err) {
    console.error(err)
  }
})()

%types: example/types.json%

/**
 * @typedef {Object} TypeA A new type with certain properties.
 */

As can be seen above, the description of the type was automatically updated based on the data read from the file.

Replacer Errors

If an error happens in a sync or async replacer function, the Replaceable will emit it and close.

/** yarn example/errors.js */
import { Replaceable } from 'restream'
import { createReadable } from './lib'

const STRING = 'test string'

;(async () => {
  try {
    const rs = createReadable(STRING)

    const replaceable = new Replaceable([
      {
        re: /.*/,
        replacement() {
          throw new Error('An error occurred during a replacement.')
        },
      },
    ])

    await new Promise((resolve, reject) => {
      replaceable
        .on('close', resolve)
        .on('error', reject)

      rs.pipe(replaceable)
    })
  } catch ({ message }) {
    console.log(message)
  }
})()

An error occurred during a replacement.

Markers

Markers can be used to cut some portion of input text according to a regular expression, run necessary replacement rules on the remaining parts, and then restore the cut chunks. In this way, those chunks do not take part in transformations produced by rules, and can be re-inserted into the stream in their original form.

An example use case would be a situation when markdown code blocks need to be transformed into html, however those code blocks don't need to be processed when inside of a comment, such as:

<!--
The following line should be preseved:

**Integrity is the ability to stand by an idea.**
-->

But the next lines should be transformed into HTML:

**Civilization is the process of setting man free from men.**

**Every building is like a person. Single and unrepeatable.**

When using a naïve transformation with a replacement rule for changing ** into <strong>, both lines will be transformed.

import { Replaceable } from 'restream'
import { createReadStream } from 'fs'
import Catchment from 'catchment'

const FILE = 'example/markers/example.md'

const strongRule = {
  re: /\*\*(.+?)\*\*/g,
  replacement(match, p1) {
    return `<strong>${p1}</strong>`
  },
}

;(async () => {
  try {
    const rs = createReadStream(FILE)
    const replaceable = new Replaceable(strongRule)
    rs.pipe(replaceable)

    const { promise } = new Catchment({
      rs: replaceable,
    })
    const res = await promise

    console.log(res)
  } catch (err) {
    console.error(err)
  }
})()

<!--
The following line should be preseved:

<strong>Integrity is the ability to stand by an idea.</strong>
-->

But the next lines should be transformed into HTML:

<strong>Civilization is the process of setting man free from men.</strong>

<strong>Every building is like a person. Single and unrepeatable.</strong>

In the output above, the ** in the comment is also transformed using the rule. To prevent this, the strategy is to cut comments out first using markers, then perform the transformation using the strong rule, and finally place the comments back into the text.

const { comments } = makeMarkers({
  comments: /<!--([\s\S]+?)-->/g,
})
const cutComments = makeCutRule(comments)
const pasteComments = makePasteRule(comments)

const replaceable = new Replaceable([
  cutComments,
  strongRule,
  pasteComments,
])

<!--
The following line should be preseved:

**Integrity is the ability to stand by an idea.**
-->

But the next lines should be transformed into HTML:

<strong>Civilization is the process of setting man free from men.</strong>

<strong>Every building is like a person. Single and unrepeatable.</strong>

makeMarkers(
  matchers: { [name]: RegExp },
  config?: MakeMarkersConfig,
): { [name]: Marker }

This function will create markers from the hash of passed matchers object. The markers are then used to create cut and paste rules.

When a RegExp specified for a marker is matched, the chunk will be replaced with a string. By default, the string has the %%_RESTREAM_MARKER_NAME_REPLACEMENT_INDEX_%% format.

const { comments, strong } = makeMarkers({
  comments: /<!--([\s\S]+?)-->/g,
  strong: /\*\*(.+?)\*\*/g,
})
const [cutComments, cutStrong] =
  [comments, strong].map(makeCutRule)

const replaceable = new Replaceable([
  cutComments,
  cutStrong,
])

%%_RESTREAM_COMMENTS_REPLACEMENT_0_%%

But the next lines should be transformed into HTML:

%%_RESTREAM_STRONG_REPLACEMENT_0_%%

%%_RESTREAM_STRONG_REPLACEMENT_1_%%

This format can be modified with the additional configuration passed as the second argument by providing a function to generate replacement strings, and their respecitve regular expressions to replace them back with their original values.

MakeMarkersConfig: Additional configuration.

By default, %%_RESTREAM_${name.toUpperCase()}_REPLACEMENT_${index}_%% replacement is used with new RegExp(`%%_RESTREAM_${name.toUpperCase()}_REPLACEMENT_(\d+)_%%`, 'g') regex to detect it and restore the original value.

makeCutRule(
  marker: Marker,
): Rule

Make a rule for the Repleceable to cut out marked chunks so that they don't participate in further transformations.

makePasteRule(
  marker: Marker,
): Rule

Make a rule for the Repleceable to paste back chunks replaced earlier.

Accessing Replacements

Sometimes, it might be necessary to access the value replaced by a marker's regular expression. In the example below, all inner code blocks are cut at first to preserve them as they are, then the LINKS rule is applied to generate anchors in a text. However, it is also possible that an inner code block will form part of a link, but because it has been replaced with a marker, the link rule will not work properly.

const getName = (title) => {
  const name = title.toLowerCase()
    .replace(/\s+/g, '-')
    .replace(/[^\w-]/g, '')
  return name
}

const { code } = makeMarkers({
  code: /`(.+?)`/g,
})
const cutCode = makeCutRule(code)
const pasteCode = makePasteRule(code)

const linkRule = {
  re: /\[(.+?)\]\(#LINK\)/g,
  replacement(match, title) {
    const name = getName(title)
    return `<a name="${name}">${title}</a>`
  },
}

const replaceable = new Replaceable([
  cutCode,
  linkRule,
  pasteCode,
])

`a code block`

`[link in a code block](#LINK)`

[just link](#LINK)

[`A code block` in a link](#LINK)

`a code block`

`[link in a code block](#LINK)`

<a name="just-link">just link</a>

<a name="_restream_code_replacement_2_-in-a-link">`A code block` in a link</a>

To prevent this from happening, a check must be performed in the LINKS rule replacement function to see if matched text has any inner code blocks in it. If it does, the value can be accessed and placed back for the correct generation of the link name. This is achieved with the replace function.

const getName = (title) => {
  const name = title.toLowerCase()
    .replace(/\s+/g, '-')
    .replace(/[^\w-]/g, '')
  return name
}

const { code } = makeMarkers({
  code: /`(.+?)`/g,
})
const cutCode = makeCutRule(code)
const pasteCode = makePasteRule(code)

const linkRule = {
  re: /\[(.+?)\]\(#LINK\)/g,
  replacement(match, title) {
    const realTitle = title.replace(code.regExp, (m, i) => {
      const val = code.map[i]
      return val
    })
    const name = getName(realTitle)
    return `<a name="${name}">${title}</a>`
  },
}

const replaceable = new Replaceable([
  cutCode,
  linkRule,
  pasteCode,
])

`a code block`

`[link in a code block](#LINK)`

<a name="just-link">just link</a>

<a name="a-code-block-in-a-link">`A code block` in a link</a>

Now, the link is generated correctly using the title with the text inside of the code block, and not its replaced marker. Also, because the code marker's regex is used with .replace, its lastIndex property won't change so there's no side effects (compared to using .exec method of a regular expression). This simple example shows how some markers can gain access to replacements made by other markers, which can have more compres applications.

(c) Art Deco 2018