AgRe CLI

Make your site crawlable by AI agents in minutes.

npx agre-cli

That's it. No install, no config, no account. Works on any machine with Node.js 20+.

Quick Start

npx agre-cli --site-url https://example.com
npx agre-cli generate all --site-url https://example.com

Prefer npx for one-off usage, or install globally:

npm i -g agre-cli
agre-cli --help

What it does

Run npx agre-cli in any web project root and it:

1. Detects your framework — Next.js, Nuxt, Astro, SvelteKit, Remix, Vite, plain HTML
2. Scans all routes and extracts metadata (title, description, content type)
3. Prints a scored checklist of AI-crawlability issues (0–100, A–F grade)
4. Generates llms.txt and llms-full.txt in the correct public directory
5. Analyzes robots.txt and patches it to allow all known AI bots
6. Detects CSR-only pages invisible to AI crawlers
7. Generates Schema.org JSON-LD snippets for key page types
8. Shows an agent preview — what an AI bot actually sees from a route file

Example output

┌─────────────────────────────────────────────────┐
│                                                 │
│   agre-cli v1.0.0                             │
│   Make your site crawlable by AI agents         │
│                                                 │
└─────────────────────────────────────────────────┘

◆ Detected: Next.js (App Router) — next.config.ts
◆ Found 23 routes

  CRITICAL ──────────────────────────────────────────
  ✗  llms.txt missing           Not found in public/
  ✗  llms-full.txt missing      Not found in public/

  HIGH ──────────────────────────────────────────────
  ✗  ClaudeBot allowed          ClaudeBot: Disallow: /
  ✓  GPTBot allowed             GPTBot: Allow: /

  MEDIUM ────────────────────────────────────────────
  ◐  Schema.org — 0/23 pages    No JSON-LD found
  ✓  sitemap.xml exists
  ✓  Meta descriptions — 19/23 pages

  Score: 44/100  Grade: D

Commands

npx agre-cli                          # full interactive scan (default)
npx agre-cli scan                     # scan only, print checklist, no writes
npx agre-cli generate llms            # generate llms.txt
npx agre-cli generate llms-full       # generate llms-full.txt
npx agre-cli generate robots          # patch robots.txt to allow AI bots
npx agre-cli generate schema          # generate Schema.org JSON-LD snippets
npx agre-cli generate all             # generate everything
npx agre-cli preview [file]           # show what an AI bot sees from a route
npx agre-cli init                     # create agre-cli.config.ts

Flags

--cwd <path>          Working directory (default: cwd)
--site-url <url>      Production URL (e.g. https://myapp.com)
--out <dir>           Output directory override
--yes, -y             Skip confirmation prompts
--dry-run             Show output without writing files
--json                Output as JSON (for piping/scripting)
--ci                  CI mode: non-interactive, exit 1 on critical failures
--quiet, -q           Suppress non-error output
--verbose             Debug-level output
--config <path>       Path to agre-cli.config.ts

Configuration

Create an agre-cli.config.ts in your project root (or run agre-cli init):

import type { agreConfig } from 'agre-cli';

const config: agreConfig = {
  siteUrl: 'https://myapp.com',
  siteName: 'MyApp',
  siteDescription: 'The fastest way to ship AI-ready web apps.',
  excludeRoutes: ['/internal/**', '/preview/**'],
  includeApiRoutes: false,
  includeDynamicRoutes: false,
  tokenBudget: 80000,
};

export default config;

Supported Frameworks

CI / GitHub Actions

Quick CI check

- name: Check AI crawlability
  run: npx agre-cli scan --ci --site-url https://myapp.com

Exit code 1 if any critical checklist items fail.

GitHub Action

# If this project is published as a GitHub Action:
- uses: <owner>/<repo>@v1
  with:
    site-url: https://myapp.com
    generate: all
    fail-on-critical: true

JSON output for scripting

npx agre-cli scan --json | jq '.checklist.score'

Checklist Items

Programmatic API

import {
  detect,
  scanRoutes,
  runChecklist,
  generateLlmsTxt,
  analyzeRobots,
  generateSchema,
  previewAgent,
} from 'agre-cli';

const framework = await detect(process.cwd());
const routes = await scanRoutes(process.cwd(), framework);
const result = await runChecklist(process.cwd(), framework, routes);

console.log(`Score: ${result.score}/100 (${result.grade})`);

Development

pnpm install
pnpm lint            # Run ESLint
pnpm build           # Build with tsup
pnpm test            # Run vitest
pnpm test:coverage   # Run tests with coverage
pnpm test:watch      # Watch mode
pnpm typecheck       # TypeScript type checking
pnpm dev             # Build in watch mode

How scoring works

Each checklist item has a severity weight:

• Critical: 35 points
• High: 25 points
• Medium: 15 points
• Low: 5 points

Pass = full points, Warn = half, Fail = 0, Skip = excluded.

Non-goals (v1)

• Does not deploy or host anything
• Does not make HTTP requests (fully offline static analysis)
• Does not run a headless browser
• Does not require an account or API key
• Does not modify source code — generates new files and snippets only

License

MIT © agre-cli