Package Exports

benny

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (benny) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Benny - a dead simple benchmarking framework

npm CircleCI David Codecov GitHub

Example

Overview
Installation
Quick example
API
Working with many suites
Working with async code
Tweaking benchmarks
Snippets
Additional examples
License

Overview

Under the hood, Benny uses an excellent (but complex) benchmark package.

Benny provides an improved API that allows you to:

easily prepare benchmarks for synchronous, as well as async code,
prepare local setup (sync or async) for each case
skip or run only selected cases
save essential results to a file in a JSON format
pretty-print results without additional setup
use suite results as Promises

Additionally, it provides sound defaults suitable for most use cases and excellent IDE support with built-in type definitions.

Installation

Using NPM:

npm i benny -D

Using Yarn:

yarn add benny -D

Quick example

/* benchmark.js */
const b = require('benny')

b.suite(
  'Example',

  b.add('Reduce two elements', () => {
    ;[1, 2].reduce((a, b) => a + b)
  }),

  b.add('Reduce five elements', () => {
    ;[1, 2, 3, 4, 5].reduce((a, b) => a + b)
  }),

  b.cycle(),
  b.complete(),
  b.save({ file: 'reduce', version: '1.0.0' }),
)

Execute:

node benchmark.js

Output:

Running "Example" suite...

  Reduce two elements:
    147 663 243 ops/s, ±0.78%   | fastest

  Reduce five elements:
    118 640 209 ops/s, ±0.93%   | slowest, 19.65% slower

Finished 2 cases!
  Fastest: Reduce two elements
  Slowest: Reduce five elements

Saved to: benchmark/results/reduce.json

File content:

{
  "name": "Example",
  "date": "2019-10-01T21:45:13.058Z",
  "version": "1.0.0",
  "results": [
    {
      "name": "Reduce two elements",
      "ops": 147663243,
      "margin": 0.78,
      "percentSlower": 0
    },
    {
      "name": "Reduce five elements",
      "ops": 118640209,
      "margin": 0.93,
      "percentSlower": 19.65
    }
  ],
  "fastest": {
    "name": "Reduce two elements",
    "index": 0
  },
  "slowest": {
    "name": "Reduce five elements",
    "index": 1
  }
}

API

// You can also use ES Modules syntax and default imports
const { add, complete, cycle, save, suite } = require('benny')

suite(
  /**
   * Name of the suite - required
   */
  "My suite",

  /**
   * If the code that you want to benchmark has no setup,
   * you can run it directly:
   */
  add('My first case', () => {
    myFunction()
  }),

  /**
   * If the code that you want to benchmark requires setup,
   * you should return it wrapped in a function:
   */
  add('My second case', () => {
    // Some setup:
    const testArr = Array.from({ length: 1000 }, (_, index) => index)

    // Benchmarked code wrapped in a function:
    return () => myOtherFunction(testArr)
  }),

  /**
   * This benchmark will be skipped:
   */
  add.skip('My third case', () => {
    1 + 1
  }),

  /**
   * This benchmark will be the only one that runs
   * (unless there are other cases marked by .only)
   */
  add.only('My fourth case', () => {
    Math.max(1, 2, 3, 4, 5)
  }),

  /**
   * This will run after each benchmark in the suite.
   *
   * You can pass a function that takes:
   *   - as a first argument: an object with the current result
   *   - as a first argument: an object with all results
   * If you return a value, it will be logged,
   * replacing in-place the previous cycle output.
   *
   * By default, it pretty-prints case results
   */
  cycle(),

  /**
   * This will run after all benchmarks in the suite.
   * You can pass a function that takes an object with all results.
   *
   * By default, it pretty-prints a simple summary.
   */
  complete(),

  /**
   * This will save the results to a file.
   * You can pass an options object.
   *
   * By default saves to benchmark/results/<ISO-DATE-TIME>.json
   */
  save({
    /**
     * String or function that produces a string,
     * if function, then results object will be passed as argument:
     */
    file: 'myFileNameWithoutExtension'
    /**
     * Destination folder (can be nested), will be created if not exists:
     */
    folder: 'myFolder',
    /**
     * Version string - if provided will be included in the file content
     */
    version: require('package.json').version,
  }),
)

All methods are optional - use the ones you need.

Additionally, each suite returns a Promise that resolves with results object (the same as passed to the complete method).

Working with many suites

You can create as many suites as you want. It is a good practice to define each suite in a separate file, so you can run them independently if you need. To run multiple suites, create the main file, where you import all your suites.

Example:

/* suites/suite-one.js */

const b = require('benny')

module.exports = b.suite(
  'Suite one',

  b.add('Reduce two elements', () => {
    ;[1, 2].reduce((a, b) => a + b)
  }),

  b.add('Reduce five elements', () => {
    ;[1, 2, 3, 4, 5].reduce((a, b) => a + b)
  }),

  b.cycle(),
  b.complete(),
  b.save({ file: 'reduce' }),
)

/* suites/suite-two.js */

const b = require('benny')

module.exports = b.suite(
  'Suite two',

  b.add('Multiple two numbers', () => {
    2 * 2
  }),

  b.add('Multiply three numbers', () => {
    2 * 2 * 2
  }),

  b.cycle(),
  b.complete(),
  b.save({ file: 'add' }),
)

/* benchmark.js */

require('./suites/suite-one')
require('./suites/suite-two')

Run:

node benchmark.js

Working with async code

Benny handles Promises out of the box. You can have async benchmarks, async setup, or both.

To demonstrate how this work, I will use the delay function that simulates a long-pending promise:

const delay = (seconds) =>
  new Promise((resolve) => setTimeout(resolve, seconds * 1000))

Async benchmark without setup

add('Async benchmark without setup', async () => {
  // You can use await or return - works the same,
  // (async function always returns a Promise)
  await delay(0.5) // Resulting in 2 ops/s
})

If a benchmark has many async operations you should await every statement that you want to be completed before the next iteration:

add('Async benchmark without setup - many async operations', async () => {
  await delay(0.5)
  await delay(0.5)
  // Resulting in 1 ops/s
})

Async benchmark with setup - return a promise wrapped in a function

add('Async benchmark with some setup', async () => {
  await delay(2) // Setup can be async, it will not affect the results

  return async () => {
    await delay(0.5) // Still 2 ops/s
  }
})

Synchronous benchmark with async setup

add('Sync benchmark with some async setup', async () => {
  await delay(2) // Setup can be async, it will not affect the results

  return () => {
    1 + 1 // High ops, not affected by slow, async setup
  }
})

If we add these cases to a suite and execute it, we will get results that would look similar to this:

Running "Async madness" suite...

  Async benchmark without setup:
    2 ops/s, ±0.02%             | 100% slower

  Async benchmark without setup - many async operations:
    1 ops/s, ±0.05%             | slowest, 100% slower

  Async benchmark with some setup:
    2 ops/s, ±0.11%             | 100% slower

  Sync benchmark with some async setup:
    674 553 637 ops/s, ±2.13%   | fastest

Finished 4 cases!
  Fastest: Sync benchmark with some async setup
  Slowest: Async benchmark without setup - many async operations

Saved to: benchmark/results/async-madness.json

Note: If you look closely, because of the async keyword, the last two examples return not a function, but a Promise, that resolves to a function, that returns either another Promise or other value (undefined in the last case). Benny is smart enough to get your intent and build proper async or sync benchmark for you.

Tweaking benchmarks

If the default results are not optimal (high error margin, etc.), you can change parameters for each case by providing an options object as a third parameter to the add function.

Available options:

/**
 * The delay between test cycles (secs).
 *
 * @default 0.005
 */
delay: number

/**
 * The default number of times to execute a test on a benchmark's first cycle.
 *
 * @default 1
 */
initCount: number

/**
 * The maximum time a benchmark is allowed to run before finishing (secs).
 *
 * Note: Cycle delays aren't counted toward the maximum time.
 *
 * @default 5
 */
maxTime: number

/**
 * The minimum sample size required to perform statistical analysis.
 *
 * @default 5
 */
minSamples: number

/**
 * The time needed to reduce the percent uncertainty of measurement to 1% (secs).
 *
 * @default 0
 */
minTime: number

Example usage:

const b = require('benny')

const options = {
  minSamples: 10,
  maxTime: 2,
}

b.suite(
  'My suite',

  b.add(
    'Reduce two elements',
    () => {
      ;[1, 2].reduce((a, b) => a + b)
    },
    options,
  ),
  // ...other methods
)

Snippets

If you are using Visual Studio Code or VSCodium, you can use following code snippets -> click

To add them, open File -> Preferences -> User Snippets, chose a language (JS, TS or both) and paste additional keys from the snippets file.

You can see how they work in the demo GIF.

Additional examples

For more examples check out the /examples folder.

You can run all the examples locally if you want. Just remember to run npm i or yarn in the examples folder first.

License

Project is under open, non-restrictive ISC license.

benny