Harness Evolver

LangSmith-native autonomous agent optimization. Point at any LLM agent codebase, and Harness Evolver will evolve it — prompts, routing, tools, architecture — using multi-agent evolution with LangSmith as the evaluation backend.

Inspired by Meta-Harness (Lee et al., 2026). The scaffolding around your LLM produces a 6x performance gap on the same benchmark. This plugin automates the search for better scaffolding.

Install

Claude Code Plugin (recommended)

/plugin marketplace add raphaelchristi/harness-evolver-marketplace
/plugin install harness-evolver

Updates are automatic. Python dependencies (langsmith, langsmith-cli) are installed on first session start via hook.

npx (first-time setup or non-Claude Code runtimes)

npx harness-evolver@latest

Interactive installer that configures LangSmith API key, creates Python venv, and installs all dependencies. Works with Claude Code, Cursor, Codex, and Windsurf.

Both install paths work together. Use npx for initial setup (API key, venv), then the plugin marketplace handles updates automatically.

Quick Start

cd my-llm-project
export LANGSMITH_API_KEY="lsv2_pt_..."
claude

/evolver:setup      # explores project, configures LangSmith
/evolver:health     # check dataset quality (auto-corrects issues)
/evolver:evolve     # runs the optimization loop
/evolver:status     # check progress (rich ASCII chart)
/evolver:deploy     # tag, push, finalize

How It Works

Commands

Agents

Evolution Loop

/evolver:evolve
  |
  +- 0.5  Validate state (check .evolver.json vs LangSmith)
  +- 0.6  /evolver:health — dataset quality + secret scan + auto-correct
  +- 0.7  Baseline LLM-judge — re-score baseline with correctness if only has_output exists
  +- 1.   Read state (.evolver.json + LangSmith experiments)
  +- 1.5  Gather trace insights + judge feedback (cluster errors, tokens, latency)
  +- 1.8  Analyze per-task failures with judge comments (train split only)
  +- 1.8a Claude generates strategy.md + lenses.json (incl. archive_branch lens)
  +- 1.9  Prepare shared proposer context (KV cache-optimized prefix)
  +- 2.   Wave 1: spawn critical/high proposers in parallel worktrees
  +- 2.5  Wave 2: medium/open proposers see wave 1 results before starting
  +- 3.   Copy config to worktrees, run canary, evaluate candidates
  +- 3.5  Spawn evaluator agent (rubric-aware, few-shot calibrated LLM-as-judge)
  +- 4.   Compare on held-out split -> winner + Pareto front + pairwise if close
  +- 4.5  Constraint gate — reject candidates that break size/tests/entry-point
  +- 5.   Merge winning worktree into main branch
  +- 5.5  Archive ALL candidates (winners + losers) to evolution_archive/
  +- 5.6  Regression tracking + auto-guard failures
  +- 6.   Report results + evolution chart
  +- 6.2  Consolidator agent updates evolution memory (runs in background)
  +- 6.5  Auto-trigger Active Critic (detect + fix evaluator gaming)
  +- 7.   Auto-trigger ULTRAPLAN Architect (opus model, deep analysis)
  +- 8.   Claude assesses gate conditions (plateau, target, diminishing returns)

Architecture

Plugin hook (SessionStart)
  └→ Creates venv, installs langsmith + langsmith-cli, exports env vars

Skills (markdown)
  ├── /evolver:setup    → explores project, smart defaults, runs setup.py
  ├── /evolver:health   → dataset quality + secret scan + auto-correct
  ├── /evolver:evolve   → orchestrates the evolution loop
  ├── /evolver:status   → rich ASCII evolution chart + stagnation detection
  └── /evolver:deploy   → tags and pushes

Agents (markdown)
  ├── Proposer (xN)     → self-organizing, lens-driven, isolated git worktrees
  ├── Evaluator          → rubric-aware LLM-as-judge via langsmith-cli
  ├── Critic             → detects gaming + implements stricter evaluators
  ├── Architect          → ULTRAPLAN deep analysis (opus model)
  ├── Consolidator       → cross-iteration memory (autoDream-inspired)
  └── TestGen            → generates test inputs with rubrics + adversarial injection

Tools (Python)
  ├── setup.py              → creates datasets, configures evaluators + weights
  ├── run_eval.py           → runs target against dataset (canary preflight, {input_text})
  ├── read_results.py       → weighted scoring, Pareto front, judge feedback
  ├── trace_insights.py     → clusters errors from traces
  ├── seed_from_traces.py   → imports production traces (secret-filtered)
  ├── evolution_chart.py    → rich ASCII chart (stdlib-only)
  ├── constraint_check.py   → validates proposals (growth, syntax, tests) (stdlib-only)
  ├── secret_filter.py      → detects 15+ secret patterns (stdlib-only)
  ├── mine_sessions.py      → extracts eval data from Claude Code history (stdlib-only)
  ├── dataset_health.py     → dataset quality diagnostic + secret scanning
  ├── validate_state.py     → validates config vs LangSmith state
  ├── regression_tracker.py → tracks regressions, auto-adds failure guards
  ├── archive.py            → persistent candidate history (diffs, proposals, scores)
  ├── add_evaluator.py      → programmatically adds evaluators
  └── adversarial_inject.py → detects memorization, injects adversarial tests

Entry Point Placeholders

When configuring your agent's entry point during setup, use the placeholder that matches how your agent takes input:

Example:

# Agent that takes a query as text:
python agent.py --query {input_text}

# Agent that reads a JSON file:
python agent.py {input}

Requirements

• LangSmith account + LANGSMITH_API_KEY
• Python 3.10+
• Git (for worktree-based isolation)
• Claude Code (or Cursor/Codex/Windsurf)

Dependencies (langsmith, langsmith-cli) are installed automatically by the plugin hook or the npx installer.

Framework Support

LangSmith traces any AI framework. The evolver works with all of them:

References

• Meta-Harness: End-to-End Optimization of Model Harnesses — Lee et al., 2026
• Drop the Hierarchy and Roles: How Self-Organizing LLM Agents Outperform Designed Structures — Dochkina, 2026
• Hermes Agent Self-Evolution — NousResearch (rubric-based eval, constraint gates)
• Agent Skills for Context Engineering — Koylan (justification-before-score, observation masking, anchored summarization)
• Darwin Godel Machine — Sakana AI
• AlphaEvolve — DeepMind
• LangSmith Evaluation — LangChain
• Harnessing Claude's Intelligence — Martin, Anthropic, 2026
• Traces Start the Agent Improvement Loop — LangChain

License

MIT