selftune — Skill Observability & Continuous Improvement CLI

Observe real sessions, detect missed triggers, grade execution quality, and automatically evolve skill descriptions toward the language real users actually use.

Works with Claude Code, Codex, and OpenCode.

Observe → Detect → Diagnose → Propose → Validate → Deploy → Watch → Repeat

Install

npx selftune@latest doctor

Or install globally:

npm install -g selftune
selftune doctor

Requires Bun or Node.js 18+ with tsx.

Why

Agent skills are static, but users are not. When a skill undertriggers — when someone says "make me a slide deck" and the pptx skill doesn't fire — that failure is invisible. The user concludes "AI doesn't follow directions" rather than recognizing the skill description doesn't match how real people talk.

selftune closes this feedback loop.

What It Does

Quick Start

1. Initialize

npx selftune@latest init

The init command auto-detects your agent environment (Claude Code, Codex, or OpenCode), resolves the CLI path, determines the LLM mode, and writes config to ~/.selftune/config.json. All subsequent commands read from this config.

Use --agent claude_code|codex|opencode to override detection, --llm-mode agent|api to override LLM mode, or --force to reinitialize.

4. Install hooks (Claude Code)

If init reports hooks are not installed, merge the entries from skill/settings_snippet.json into ~/.claude/settings.json. Replace /PATH/TO/ with the absolute path to this repository.

5. Verify setup

selftune doctor

Doctor checks log file health, hook installation, schema validity, and config status.

Platform-Specific Notes

Claude Code — Hooks capture telemetry automatically after installation. Zero configuration once hooks are in settings.json.

Codex — Use the wrapper for real-time capture or the batch ingestor for historical logs:

selftune wrap-codex -- <your codex args>
selftune ingest-codex

OpenCode — Backfill historical sessions from SQLite:

selftune ingest-opencode

All platforms write to the same shared JSONL log schema at ~/.claude/.

Commands

selftune <command> [options]

No separate API key required — grading and evolution use whatever agent CLI you already have installed. Set ANTHROPIC_API_KEY to use the API directly instead.

See skill/Workflows/ for detailed step-by-step guides for each command.

How It Works

Telemetry Capture

Claude Code (hooks):                 OpenCode (hooks):
  UserPromptSubmit → prompt-log.ts     message.*        → opencode-prompt-log.ts
  PostToolUse      → skill-eval.ts     tool.execute.after → opencode-skill-eval.ts
  Stop             → session-stop.ts   session.idle     → opencode-session-stop.ts
          │                                    │
          └──────────┬─────────────────────────┘
                     ▼
          Shared JSONL Log Schema (~/.claude/)
            ├── all_queries_log.jsonl
            ├── skill_usage_log.jsonl
            └── session_telemetry_log.jsonl

Codex (wrapper/ingestor — hooks not yet available):
  codex-wrapper.ts  (real-time tee of JSONL stream)
  codex-rollout.ts  (batch ingest from rollout logs)
          │
          └──→ Same shared JSONL schema

Eval & Grading

selftune evals cross-references the two query logs:
  Positives  = skill_usage_log entries for target skill
  Negatives  = all_queries_log entries NOT in positives

selftune grade reads:
  session_telemetry_log → process metrics (tool calls, errors, turns)
  transcript JSONL       → what actually happened
  expectations           → what should have happened

Evolution Loop

selftune evolve:
  1. Load eval set (or generate from logs)
  2. Extract failure patterns (missed queries grouped by invocation type)
  3. Generate improved description via LLM
  4. Validate against eval set (must improve, <5% regression)
  5. Deploy updated SKILL.md + PR + audit trail

selftune watch:
  Monitor pass rate over sliding window of recent sessions
  Alert (or auto-rollback) on regression > threshold

Architecture

cli/selftune/
├── index.ts                     CLI entry point (command router)
├── init.ts                      Agent detection, config bootstrap
├── types.ts, constants.ts       Shared interfaces and constants
├── observability.ts             Health checks (doctor command)
├── utils/                       JSONL, transcript parsing, LLM calls, schema validation
├── hooks/                       Claude Code + OpenCode telemetry capture
├── ingestors/                   Codex adapters + OpenCode backfill
├── eval/                        False negative detection, eval set generation
├── grading/                     3-tier session grading (agent or API mode)
├── evolution/                   Failure extraction, proposal, validation, deploy, rollback
└── monitoring/                  Post-deploy regression detection

skill/
├── SKILL.md                     Routing table (~120 lines)
├── settings_snippet.json        Claude Code hook config template
├── references/                  Domain knowledge (logs, grading methodology, taxonomy)
└── Workflows/                   Step-by-step guides (1 per command)

Dependencies flow forward only: shared → hooks/ingestors → eval → grading → evolution → monitoring. Enforced by lint-architecture.ts.

Config persists at ~/.selftune/config.json (written by init, read by all commands via skill workflows).

See ARCHITECTURE.md for the full domain map and module rules.

Log Schema

Three append-only JSONL files at ~/.claude/:

The source field identifies the platform: claude_code, codex, or opencode.

Development

make check    # lint + architecture lint + all tests
make lint     # biome check + architecture lint
make test     # bun test

Zero runtime dependencies. Uses Bun built-ins only.

Tips

• Run selftune init first — everything else reads from the config it writes.
• Let logs accumulate over several days before running evals — more diverse real queries = more reliable signal.
• All hooks are silent (exit 0) and take <50ms. Negligible overhead.
• Logs are append-only JSONL. Safe to delete to start fresh, or archive old files.
• Use --max 75 to increase eval set size once you have enough data.
• Use --seed 123 for a different random sample of negatives.
• Use --dry-run with evolve to preview proposals without deploying.
• The doctor command checks log health, hook presence, config status, and schema validity.

Milestones