tokenlean

Lean CLI tools for AI agents and developers. Reduce context usage, save tokens, improve workflow efficiency.

The Problem

AI coding assistants are powerful, but they have a fundamental constraint: context windows. Every file you read, every search result, every piece of context consumes tokens. This matters because:

• Cost: More tokens = higher API costs
• Quality: Overstuffed context leads to worse responses
• Speed: Larger contexts take longer to process
• Limits: Hit the ceiling and the AI loses important information

Most developers (and AI agents) approach codebases inefficiently - reading entire files when they only need function signatures, grepping repeatedly when a structured search would do, or diving into implementation before understanding the API surface.

The Solution

tokenlean provides 27 specialized CLI tools that give you (or your AI agent) exactly the information needed - no more, no less. Each tool is designed to answer a specific question about your codebase with minimal token overhead.

Instead of reading a 500-line file to understand its exports, run tl-exports (50 tokens). Instead of reading all your types to understand data shapes, run tl-types (100 tokens). Instead of guessing which files might be affected by a change, run tl-impact and know for sure.

Installation

npm install -g tokenlean

Or link locally for development:

git clone https://github.com/edimuj/tokenlean.git
cd tokenlean
npm link

Quick Start

# Get project overview
tl-structure

# Before reading a file - check its size
tl-context src/api/

# Get function signatures without bodies
tl-symbols src/utils.ts

# Understand what a module exports
tl-exports src/lib/

# Check what would break if you change a file
tl-impact src/core/auth.ts

AI Agent Integration

Add tokenlean instructions to your AI tool's config:

The --minimal flag produces a compact version that uses fewer tokens.

Tools Reference

Before Reading Files

These tools help you understand code structure without reading full implementations.

Before Modifying Files

Understand dependencies and impact before making changes.

Understanding History

Track changes and authorship efficiently.

Finding Things

Search and discover code patterns.

Configuration & Utilities

Common Options

All tools support these flags:

-l N, --max-lines N    Limit output to N lines
-t N, --max-tokens N   Limit output to ~N tokens
-j, --json             Output as JSON (for piping)
-q, --quiet            Minimal output (no headers/stats)
-h, --help             Show help

Configuration

tokenlean uses .tokenleanrc.json for configuration:

• Project config: .tokenleanrc.json in your project root
• Global config: ~/.tokenleanrc.json in your home directory

Project config overrides global config. Both are optional.

{
  "output": {
    "maxLines": 100,
    "maxTokens": null
  },
  "skipDirs": [
    "generated",
    "vendor"
  ],
  "skipExtensions": [
    ".gen.ts"
  ],
  "importantDirs": [
    "domain",
    "core"
  ],
  "importantFiles": [
    "ARCHITECTURE.md"
  ],
  "searchPatterns": {
    "hooks": {
      "description": "Find React hooks",
      "pattern": "use[A-Z]\\w+",
      "glob": "**/*.{ts,tsx}"
    }
  },
  "hotspots": {
    "days": 90
  },
  "structure": {
    "depth": 3
  }
}

Config values extend built-in defaults (they don't replace them).

Caching

tokenlean caches expensive ripgrep operations to speed up repeated searches. The cache uses git-based invalidation - it automatically invalidates when you make commits or modify files.

tl-cache stats      # View cache statistics
tl-cache clear      # Clear cache for current project
tl-cache clear-all  # Clear all cached data

Cache is stored in ~/.tokenlean/cache/ and is enabled by default.

Cache Configuration

{
  "cache": {
    "enabled": true,
    "ttl": 300,
    "maxSize": "100MB",
    "location": null
  }
}

Disable Caching

# Disable for a single command
TOKENLEAN_CACHE=0 tl-search hooks

# Disable in config
echo '{"cache":{"enabled":false}}' > .tokenleanrc.json

Example Workflows

Starting work on an unfamiliar codebase

tl-structure                    # Get the lay of the land
tl-entry                        # Find entry points
tl-exports src/lib/             # Understand the public API
tl-types src/types/             # Understand data shapes

Before refactoring a file

tl-impact src/core/auth.ts      # What would break?
tl-deps src/core/auth.ts        # What does it depend on?
tl-related src/core/auth.ts     # Find the tests
tl-coverage src/core/auth.ts    # Is it well tested?
tl-complexity src/core/auth.ts  # How complex is it?

Understanding a component

tl-component src/Button.tsx     # Props, hooks, dependencies
tl-symbols src/Button.tsx       # Function signatures
tl-history src/Button.tsx       # Recent changes
tl-blame src/Button.tsx         # Who wrote what

Finding technical debt

tl-complexity src/ --threshold 15  # Complex functions
tl-unused src/                     # Dead code
tl-todo                            # Outstanding TODOs
tl-hotspots                        # Frequently changed (unstable?)

Starting a new project

tl-name coolproject awesomelib     # Check npm, GitHub, domains
tl-name myapp -s                   # Suggest variations if taken
tl-name myapp --tld io             # Check .io domain

Dependencies

• ripgrep (rg): Required for search-based tools
• git: Required for history/blame/diff tools

Design Principles

1. Single purpose: Each tool does one thing well
2. Minimal output: Show only what's needed, nothing more
3. Token-conscious: Every tool is designed to save context tokens
4. Composable: Tools work together and support JSON output for piping
5. Fast: No heavy parsing or external services - just regex and file operations
6. Universal: Works with any JavaScript/TypeScript project, most tools support Python/Go too

License

MIT