ai-experiments

A minimalistic experiment runner for AI tasks.

Installation

npm install ai-experiments
# or
yarn add ai-experiments
# or
pnpm add ai-experiments

Usage

Basic Example

import { Experiment } from 'ai-experiments';

const result = await Experiment('simple-test', {
  models: ['gpt-4o'],
  temperature: 0.7,
  prompt: 'What is the capital of France?',
});

console.log(result);

Using Parameter Combinations

import { Experiment } from 'ai-experiments';

const result = await Experiment('temperature-comparison', {
  models: ['gpt-4o', 'gpt-4o-mini'],
  temperature: [0, 0.3, 0.7, 1.0],
  prompt: 'Generate a creative story about a robot.',
});

// This will run 8 combinations (2 models × 4 temperatures)
console.log(result);

Using the Cartesian Function Directly

import { cartesian } from 'ai-experiments';

const combinations = cartesian({
  model: ['gpt-4o', 'gpt-4o-mini'],
  temperature: [0, 0.7],
  maxTokens: [100, 500]
});

// Returns:
// [
//   { model: 'gpt-4o', temperature: 0, maxTokens: 100 },
//   { model: 'gpt-4o', temperature: 0, maxTokens: 500 },
//   { model: 'gpt-4o', temperature: 0.7, maxTokens: 100 },
//   { model: 'gpt-4o', temperature: 0.7, maxTokens: 500 },
//   { model: 'gpt-4o-mini', temperature: 0, maxTokens: 100 },
//   { model: 'gpt-4o-mini', temperature: 0, maxTokens: 500 },
//   { model: 'gpt-4o-mini', temperature: 0.7, maxTokens: 100 },
//   { model: 'gpt-4o-mini', temperature: 0.7, maxTokens: 500 }
// ]

Using the Runner

// vitest.config.ts
import { defineConfig } from 'vitest/config';
import { createRunner } from 'ai-experiments';

export default createRunner({
  outputDir: '.ai/experiments',
  testMatch: ['**/*experiment*.(js|ts|mjs|cjs)'],
  watch: false,
});

API Reference

Experiment

function Experiment<T = any, E = any>(
  name: string,
  config: ExperimentConfig<T, E>
): Promise<ExperimentResult>

Parameters

name: Name of the experiment
config: Configuration object with the following properties:
- models: Array of model names to use
- temperature: Number or array of temperature values
- seed (optional): Number or array of seed values
- prompt (optional): String or function that generates prompts
- inputs (optional): Array or function that returns input values
- expected (optional): Expected output for validation
- schema (optional): Schema for structured output

Returns

Promise that resolves to an ExperimentResult object with:

name: Name of the experiment
results: Array of results for each parameter combination
totalTime: Total time taken for the experiment
timestamp: ISO string of when the experiment was run

cartesian

function cartesian<T extends Record<string, readonly any[]>>(
  spec: T
): Array<{ [K in keyof T]: T[K][number] }>

Parameters

spec: Object with keys mapping to arrays of values

Returns

Array of objects representing all possible combinations of the input values.

createRunner

function createRunner(config?: RunnerConfig): VitestConfig

Parameters

config (optional): Configuration object with the following properties:
- outputDir (optional): Directory where experiment results will be saved
- testMatch (optional): Custom test matcher pattern
- watch (optional): Whether to watch for file changes

Returns

A Vitest configuration function that can be used in vitest.config.ts.