Web Test Runner

This project is in beta. We are looking for people to test it out, and let us know about issues and what they think about it.

A test runner for web applications.

👉   Headless browsers with puppeteer, playwright, or selenium.
🚧   Reports logs, 404s, and errors from the browser.
📦   Supports native es modules.
🔧   Runs tests in parallel and isolation.
👀   Interactive watch mode.
🏃   Reruns only changed tests.
🚀   Powered by esbuild and rollup plugins

Getting started

Install the test runner:

npm i -D @web/test-runner

Do a single test run:

web-test-runner test/**/*.test.js --node-resolve
wtr test/**/*.test.js --node-resolve

Run in watch mode, reloading on file changes:

web-test-runner test/**/*.test.js --node-resolve --watch
wtr test/**/*.test.js --node-resolve --watch

Run with test coverag (this is slower):

web-test-runner test/**/*.test.js --node-resolve --coverage
wtr test/**/*.test.js --node-resolve --coverage

Example projects

Check out these example projects for a full setup.

Writing tests

Web test runner can support different test frameworks. By default we use @web/test-runner-mocha, check out the docs to learn more about writing tests with mocha and configuring mocha options.

JS tests

When you point the test runner to a JS file, it will hand the file to the configured test framework to load and run it.

HTML tests

When you point the test runner at an HTML file you can take full control over the test environment. There is no automatic bootstrapping of a test framework, you need to make sure things are set up and results are communicated back to the test runner. @web/test-runner-mocha can be used as a library for HTML tests, taking care of most of the heavy lifting.

Browsers

By default, tests are run with the locally installed instance of Chrome, controlled via puppeteer-core.

Puppeteer

You can run tests with puppeteer, which will download its a local instance of Chromium instead of relying on a globally installed version of Chrome.

# add the package
npm i -D @web/test-runner-puppeteer

# add the flag
wtr test/**/*.test.js --node-resolve --puppeteer

Playwright

You can run tests with playwright, which like puppeteer downloads it's own browsers. Playwright allows testing on chromium, firefox, and WebKit.

# add the package
npm i -D @web/test-runner-playwright

# add the flag
wtr test/**/*.test.js --node-resolve --playwright --browsers chromium firefox webkit

Commands

Configuration

Web test runner looks for a configuration file in the current working directory called web-test-runner.config.

The file extension can be .js, .cjs or .mjs. A .js file will be loaded as an es module or common js module based on your version of node, and the package type of your project.

module.exports = {
  concurrency: 10,
  nodeResolve: true,
  watch: true,
  // in a monorepo you need to set set the root dir to resolve modules
  rootDir: '../../',
};

A configuration file accepts most of the command line args camel cased. Check the type definition for all options:

import { Plugin, Middleware } from '@web/dev-server';

interface TestFramework {
  path: string;
  config?: unknown;
}

interface CoverageThresholdConfig {
  statements: number;
  branches: number;
  functions: number;
  lines: number;
}

interface CoverageConfig {
  include?: string[];
  exclude?: string[];
  threshold?: CoverageThresholdConfig;
  report: boolean;
  reportDir: string;
}

type LogLevel = 'log' | 'warn' | 'error' | 'debug';

interface TestRunnerConfig {
  // globs of files to test
  files: string | string[];
  // amount of test files to run in parallel
  concurrency?: number;
  // run in watch mode, reloading when files change
  watch?: boolean;
  // resolve bare imports imports
  nodeResolve?: boolean;
  // preserve symlinks when resolve imports, instead of following
  // symlinks to their original files
  preserveSymlinks?: boolean;
  // the root directory to serve files from. this is useful in a monorepo
  // when executing commands from a package
  rootDir?: string;

  // the test framework to run tests in the browser
  testFramework?: TestFramework;
  // browsers to run tests in
  browsers?: BrowserLauncher | BrowserLauncher[];
  // server which serves files and responds to browser requests
  server?: Server;
  // reporters for posting results and progress to the terminal and/or file system
  reporters?: Reporter[];

  // middleware used by the server to modify requests/responses, for example to proxy
  // requests or rewrite urls
  middleware?: Middleware[];
  // plugins used by the server to serve or transform files
  plugins?: Plugin[];

  // configuration for the server
  protocol?: string;
  hostname?: string;
  port?: number;

  // html page used to run tests
  testRunnerHtml?: (testRunnerImport: string, config: TestRunnerCoreConfig) => string;

  // whether to log browsers logs to the terminal, or which log levels to display
  logBrowserLogs?: boolean | LogLevel[];
  // whether to log uncaught errors to the terminal
  logUncaughtErrors?: boolean;
  // run test coverage
  coverage?: boolean;
  // configuration for test coverage
  coverageConfig?: CoverageConfig;

  // how long a browser can take to start up before failing. defaults to 30000
  browserStartTimeout?: number;
  // how long a test file can take to load. defaults to 10000
  testsStartTimeout?: number;
  // how long a test file can take to finish. defaults to 20000
  testsFinishTimeout?: number;
}

Test coverage

You can run tests with test coverage using the --coverage flag:

wtr test/**/*.test.js --coverage

In the config you can define test coverage thresholds, the test run fails if you drop below this level. You can also configure where and if the detailed test report is written to disk.

module.exports = {
  coverageConfig: {
    report: true,
    reportDir: 'test-coverage',
    threshold: {
      statements: 70,
      branches: 70,
      functions: 70,
      lines: 70,
    },
  },
};

Viewport

You can change the viewport during tests using the @web/test-runner-helpers library.

Authoring es modules

We recommend writing tests as native es modules. Check out these es modules to learn more about authoring and using es modules.

Server and code transformation

The test runner server is based on @web/dev-server-core, which has a rich plugin and middleware system. It also supports reusing rollup plugins with @web/dev-server-rollup.

Plugins can be installed from the configuration file, for example to use typescript:

const { esbuildPlugin } = require('@web/dev-server-esbuild');

module.exports = {
  plugins: [esbuildPlugin({ ts: true })],
};

Check out the full documentation for more information and examples:

• Common code transformations
• Server Middleware
• Server plugins

Customizing test runner HTML

When running javascript tests, the test runner runs the test in a standard minimal HTML page. You can provide a custom HTML container to run the tests in with the testRunnerHtml function. This function receives the module import for the test runner and the test runner config.

You can use this to set up the testing environment, for example, to set global variables or load test bootstrap code.

module.exports = {
  testRunnerHtml: (testRunnerImport, config) => `
    <html>
      <body>
        <script type="module">
          window.someGlobal = 'foo';
        </script>

        <script type="module">
          import '${testRunnerImport}';
        </script>
      </body>
    </html>
  `,
};

Customizing browser launch options

You can customize the configuration to launch browsers by setting the browser launchers in the config.

// import the browser launcher you want to use, chromeLauncher is the default
const { chromeLauncher } = require('@web/test-runner');
// const { playwrightLauncher } = require('@web/test-runner-playwright');
// const { puppeteerLauncher } = require('@web/test-runner-puppeteer');

module.exports = {
  browsers: [chromeLauncher({ launchOptions: { args: ['--no-sandbox'] } })],
};

The browser launchers that are currently available:

• @web/test-runner-chrome
• @web/test-runner-puppeteer
• @web/test-runner-playwright
• @web/test-runner-selenium
• @web/test-runner-browserstack

Customizing reporters

You can customize the test reporters using the reporters option.

// import the browser launcher you want to use
const { defaultReporter } = require('@web/test-runner');
const { myReporter } = require('my-reporter');

module.exports = {
  reporters: [
    // use the default reporter only for reporting test progress
    defaultReporter({ reportTestResults: false, reportTestProgress: true }),
    // use another reporter to report test results
    myReporter(),
  ],
};

We currently don't have any reporters other than the default. If you're interested in creating a reporter, let us know and we can help you get started.

Advanced customization

This package is the opinionated default implementation of the test runner so that people can get started without much configuration.

For more advanced configuration you can use the individual packages this project is based on. See @web/test-runner-core and @web/test-runner-cli to learn more about that.