stream-chopper

Chop a single stream of data into a series of readable streams.

Stream Chopper is useful in situations where you have a stream of data you want to chop up into smaller pieces, either based on time or size. Each piece will be emitted as a readable stream (called output streams).

Possible use-cases include log rotation, splitting up large data sets, or chopping up a live stream of data into finite chunks that can then be stored.

Control how data is split

Sometimes it's important to ensure that a chunk written to the input stream isn't split up and devided over two output streams. Stream Chopper allows you to specify the chopping algorithm used (via the type option) when a chunk is too large to fit into the current output stream.

By default a chunk too large to fit in the current output stream is split between it and the next. Alternatively you can decide to either allow the chunk to "overflow" the size limit, in which case it will be written to the current output stream, or to "underflow" the size limit, in which case the current output stream will be ended and the chunk written to the next output stream.

Installation

npm install stream-chopper --save

Usage

Example app:

const StreamChopper = require('stream-chopper')

const chopper = new StreamChopper({
  size: 30,                    // chop stream when it reaches 30 bytes,
  time: 10000,                 // or when it's been open for 10s,
  type: StreamChopper.overflow // but allow stream to exceed size slightly
})

chopper.on('stream', function (stream, next) {
  console.log('>> Got a new stream! <<')
  stream.pipe(process.stdout)
  stream.on('end', next) // call next when you're ready to receive a new stream
})

chopper.write('This write contains more than 30 bytes\n')
chopper.write('This write contains less\n')
chopper.write('This is the last write\n')

Output:

>> Got a new stream! <<
This write contains more than 30 bytes
>> Got a new stream! <<
This write contains less
This is the last write

API

chopper = new StreamChopper([options])

Instantiate a StreamChopper instance. StreamChopper is a writable stream.

Takes an optional options object which, besides the normal options accepted by the Writable class, accepts the following config options:

• size - The maximum number of bytes that can be written to the chopper stream before a new output stream is emitted (default: Infinity)
• time - The maximum number of milliseconds that an output stream can be in use before a new output stream is emitted (default: -1 which means no limit)
• type - Change the algoritm used to determine how a written chunk that cannot fit into the current output stream should be handled. The following values are possible:
  • StreamChopper.split - Fit as much data from the chunk as possible into the current stream and write the remainder to the next stream (default)
  • StreamChopper.overflow - Allow the entire chunk to be written to the current stream. After writing, the stream is ended
  • StreamChopper.underflow - End the current output stream and write the entire chunk to the next stream

If type is StreamChopper.underflow and the size of the chunk to be written is larger than size an error is emitted.

Event: stream

Emitted every time a new output stream is ready. You must listen for this event.

The listener function is called with two arguments:

• stream - A readable output stream
• next - A function you must call when you're ready to receive a new output stream. If called with an error, the chopper stream is destroyed

License

MIT