Hammurabi

Spec → rubric → fixtures → runner. A small, opinionated framework for authoring LLM evaluations you'll actually trust.

Status: alpha (v0.0.4). On-disk loaders + CLI runner + multi-judge panel + baseline regression detection are all shipped. Schemas use Zod v4.

Why

Teams ship faster when they can trust outputs without re-reading every diff. Hammurabi is the connective tissue: a single schema for specs, rubrics, and fixtures that any runner (or downstream tool like Knack) can consume.

Install

npm install @collabb/hammurabi

Or run the CLI directly without installing:

npx @collabb/hammurabi hammurabi-run path/to/foo.spec.md

CLI

Hammurabi ships a hammurabi-run bin (available after install via npm bin):

hammurabi-run path/to/foo.spec.md \
  --judges claude-haiku-4-5,claude-sonnet-4-6 \
  --aggregator min \
  --baseline path/to/baseline.report.json

Writes foo.report.json and foo.report.md alongside the spec (or in --out <dir>). Exit codes are CI-meaningful:

0 — all fixtures passed, no regressions
1 — any failure or regression
2 — could not run (bad args, malformed bundle, runner error)

Run hammurabi-run --help for the full flag list.

Slash commands

Copy them into any project's .claude/commands/:

cp node_modules/@collabb/hammurabi/commands/*.md .claude/commands/

/hammurabi <path-to-spec.md> — 4-step authoring flow: refine spec via Q&A → generate rubric → generate fixtures
/hammurabi-run <path-to-spec.md> — invoke the CLI, then summarize the report in chat (failures, regressions, errored fixtures, judge votes)

Schema

Three artifact types, all language-neutral on disk:

Artifact	Format	TS type
Spec	`*.spec.md` (frontmatter + body)	`Spec`
Rubric	`*.rubric.json`	`Rubric`
Fixtures	`*.fixtures.jsonl`	`FixtureSet`

Reports are JSON: Report type.

import type { Spec, Rubric, FixtureSet, Report } from "@collabb/hammurabi/schema";
import { run } from "@collabb/hammurabi/runner";

Runner

import { run } from "@collabb/hammurabi";

const report = await run({
  spec,
  rubric,
  fixtures,
  judges: [{ model: "claude-haiku-4-5" }, { model: "claude-sonnet-4-6" }],
  aggregator: "min",
  baseline: previousReport,
});

Judge panel

The runner judges each fixture's output with a configurable panel of LLMs. Default is a single Haiku 4.5 judge. Teams running risk-sensitive workloads (trading strategies, backtests, anything touching real money) should grow the panel and mix model families to match their risk tolerance.

Option	Default	Notes
`judges`	`[{ model: "claude-haiku-4-5" }]`	Array of judge configs. Panel calls run in parallel per fixture.
`aggregator`	`"mean"`	`"mean" \| "median" \| "min" \| "max"` or a custom `(scores: number[]) => number`. Risk-sensitive callers should use `"min"`.
`regressionThreshold`	`0.05`	Per-fixture weighted-score delta below which a regression is flagged vs the baseline.
`execute`	—	Required for `target.kind === "free-form"`. Custom executor that returns the output for a given input.

Cost note: panel size multiplies cost linearly. A 2-judge panel costs 2× a 1-judge panel. Wall-clock latency is roughly constant (panel calls run in parallel via Promise.all).

Audit trail

Each CriterionScore in the report preserves the full panel's votes:

{
  criterionId: "is-uppercase",
  score: 0.5,                     // aggregated
  reasoning: "[claude-haiku-4-5] ...\n\n[claude-sonnet-4-6] ...",
  judgeVotes: [
    { model: "claude-haiku-4-5", score: 1, reasoning: "..." },
    { model: "claude-sonnet-4-6", score: 0, reasoning: "..." },
  ],
}

If a judge errors, its synthetic vote (score: 0, reasoning: "judge errored: ...") still lands in judgeVotes so the run continues and the cause is auditable.

Target kinds

Spec.frontmatter.target is a tagged union:

`kind`	Required fields	How the runner executes
`cli`	`command`	Shell-exec; fixture input is piped to stdin as JSON; stdout parsed as JSON (falls back to plain text).
`function`	`module`, `export`	Dynamic `import(module)`, calls `module[export](input)`. Module path should be importable from the caller's resolution context.
`http`	`url`, optional `method` (default `POST`)	`fetch` with `application/json` body.
`free-form`	`description`	Calls `options.execute(input)`. Required when `kind` is `free-form`.

Smoke test

npm install
npm run smoke

Requires ANTHROPIC_API_KEY (and optionally AI_GATEWAY_URL for Cloudflare AI Gateway routing).

Panel smoke:

HAMMURABI_SMOKE_JUDGES="claude-haiku-4-5,claude-sonnet-4-6" HAMMURABI_SMOKE_AGGREGATOR=min npm run smoke

Expected cost: ~$0.02 for a single-judge run, ~$0.10 for a 2-judge panel including Sonnet.

Layout

hammurabi/
├── src/
│   ├── schema/       # Spec, Rubric, Fixture, Report types
│   ├── loaders/      # On-disk parsers (*.spec.md, *.rubric.json, *.fixtures.jsonl)
│   ├── runner/       # Execute fixtures + judge panel + score
│   ├── cli/          # hammurabi-run CLI + Report → markdown renderer
│   └── index.ts
├── commands/         # /hammurabi, /hammurabi-run
├── examples/smoke/   # End-to-end smoke (inline + disk variants)
├── tests/            # Unit tests (tsx --test)
└── plans/            # Approved implementation plans