tokenjuice 🧃

lean output compaction for terminal-heavy agent workflows.

what it does

an agent or harness calls noisy tools like git status, pnpm test, pnpm --help, docker build, or rg.

tokenjuice sits in front of that tool call, runs it, trims the fat from the output, and passes back a much cleaner result to the harness.

the important bit is the boundary:

• the original command still runs
• tokenjuice compacts the observed output after execution
• --raw / --full gives you an explicit unaltered escape hatch when you need it
• raw output can be stored locally when you explicitly ask for it
• the harness gets a smaller, more useful payload instead of a wall of terminal junk

install

npm install -g tokenjuice
# or
pnpm add -g tokenjuice
# or
yarn global add tokenjuice
# or
brew tap vincentkoc/tap
brew install tokenjuice

then:

tokenjuice --help
tokenjuice --version
tokenjuice install codex

why

tool output wastes absurd amounts of context. your llm needs a diet.

tokenjuice compacts observed output after execution, gives hosts a boring, deterministic summary by default, and only stores raw output when you explicitly ask for it.

goals

• library first, not framework-locked
• JSON rules for parseability and inspection
• explicit reduce and wrap modes
• file-backed artifacts that are easy to debug
• no silent command rewrite
• speed and reliability over gimmicks

commands

tokenjuice --help
tokenjuice --version
tokenjuice reduce [file]
tokenjuice reduce-json [file]
tokenjuice wrap -- <command> [args...]
tokenjuice wrap --raw -- <command> [args...]
tokenjuice wrap --store -- <command> [args...]
tokenjuice install codex
tokenjuice ls
tokenjuice cat <artifact-id>
tokenjuice verify
tokenjuice discover
tokenjuice doctor
tokenjuice doctor codex
tokenjuice stats

adapter JSON

reduce-json is the machine-facing adapter command. it reads JSON from stdin or a file and always writes JSON to stdout.

direct payload:

{
  "toolName": "exec",
  "command": "pnpm test",
  "argv": ["pnpm", "test"],
  "combinedText": "RUN  v3.2.4 /repo\n...",
  "exitCode": 1
}

codex

for codex, the clean path is a home hook:

tokenjuice install codex
tokenjuice doctor codex

that writes a PostToolUse hook into ~/.codex/hooks.json so codex can compact noisy Bash output after the command runs.

important detail:

• the original shell command still runs untouched
• tokenjuice only rewrites the output that goes back through the hook
• raw command execution logs are still raw
• tokenjuice doctor codex checks whether the hook command is missing or pinned to a stale Homebrew Cellar path

library-side adapters can also use runReduceJsonCli(...) to call the CLI without rebuilding the child-process + JSON plumbing themselves.

when a reducer gets it wrong or the engine needs the untouched output, use the explicit bypass:

tokenjuice wrap --raw -- pnpm --help
tokenjuice wrap --full -- git status

if the hook itself goes stale after a package upgrade, repair it with:

tokenjuice doctor codex
tokenjuice install codex

for machine callers, set:

{
  "options": {
    "raw": true
  }
}

envelope payload:

{
  "input": {
    "toolName": "exec",
    "command": "pnpm test",
    "combinedText": "RUN  v3.2.4 /repo\n...",
    "exitCode": 1
  },
  "options": {
    "classifier": "tests/pnpm-test",
    "store": true,
    "maxInlineChars": 1200
  }
}

rule system

• built-in JSON rules live in src/rules
• user overrides live in ~/.config/tokenjuice/rules
• project overrides live in .tokenjuice/rules
• later layers override earlier ones by rule id

docs

• spec: docs/spec.md
• rules: docs/rules.md
• security: SECURITY.md

status

usable foundation for token reduction with diagnostics and a growing reducer set, now focused on deeper coverage and tuning.

💙 built by Vincent Koc.