JSPM

The web's scribe — writes any HTTP documentation into your AI agent's context.

What is WebScribe?

WebScribe is an MCP server that crawls any public documentation website and makes it instantly searchable by AI agents. Think of it as a scribe sitting between your copilot and the internet — reading docs, understanding their structure, and delivering precise, context-rich answers.

Agent asks: "How do I add a loading spinner to an MUI Button?"
                    ↓
WebScribe searches its locally indexed chunks
                    ↓
Returns: {
  heading_path: ["Button", "Loading", "API"],
  content: "Use the `loading` prop on LoadingButton from @mui/lab.
            <LoadingButton loading variant='contained'>Submit</LoadingButton>",
  score: 4.1
}

No more hallucinations from outdated training data. No more full-page dumps. Just the right chunk of the right doc, every time.

✨ Features

🚀 Quick Start

Use via npx (recommended)

Add to your MCP client configuration (Claude Desktop, Cursor, VS Code, etc.):

{
  "mcpServers": {
    "webscribe": {
      "command": "npx",
      "args": ["-y", "webscribe-mcp@latest"]
    }
  }
}

Run locally

git clone https://github.com/yourname/http-crawl-mcp
cd http-crawl-mcp
npm install
npm run dev

Requirements: Node.js ≥ 18, macOS / Linux / Windows (WSL recommended)

🛠️ MCP Tools

read_and_extract_page

Visits a single URL, strips navigation/sidebars, and returns clean Markdown with language-tagged code blocks.

// Input
{ "url": "https://cube.dev/docs/product/data-modeling/overview" }

crawl_documentation_site

BFS crawl from a starting URL. Discovers pages via sitemap.xml first, then link-following.
Respects robots.txt. Skips unchanged pages. Expands sub-tabs automatically.

// Input
{
  "start_url": "https://nextjs.org/docs",
  "max_pages": 50,
  "version": "v15",
  "url_glob": "**/docs/**",
  "expand_tabs": true
}

// Output
{ "crawled_count": 47, "urls": ["https://nextjs.org/docs/...", "..."] }

search_crawled_docs

BM25-ranked full-text search over all indexed chunks. Returns structured JSON with heading breadcrumbs.

// Input
{ "query": "button aria label accessibility" }

// Output
{
  "results": [
    {
      "url": "https://saltdesignsystem.com/salt/components/button/accessibility",
      "title": "Button — Salt Design System",
      "heading_path": ["Button", "Accessibility", "Keyboard Navigation"],
      "content": "The button component supports Space and Enter key activation...",
      "score": 3.82,
      "version": "latest"
    }
  ]
}

heading_path is the key differentiator — agents get full breadcrumb context so they know exactly where in the docs the answer sits.

crawl_component_docs

Two-phase crawler for component library index pages.
Phase 1 discovers component links. Phase 2 crawls each component + sub-tabs (/usage, /examples, /api, /accessibility) concurrently.

{ "index_url": "https://saltdesignsystem.com/salt/components/", "max_pages": 200 }

Works with Salt DS, MUI, Ant Design, Chakra UI, Radix, and any similar design system.

get_document

Retrieves the full cached Markdown for a specific URL. Use when a search chunk needs more surrounding context.

{ "url": "https://nextjs.org/docs/routing/introduction", "version": "v15" }

get_index_stats

Returns the total pages and semantic chunks in the local index.

// Output
{ "total_pages": 142, "total_chunks": 891 }

⚙️ Architecture

┌─────────────────────────────────────────────────────────┐
│         MCP Client (Cursor · Claude Desktop · Copilot)  │
└────────────────────────┬────────────────────────────────┘
                         │  JSON-RPC / stdio
┌────────────────────────▼────────────────────────────────┐
│               WebScribe  MCP Server                     │
│                   src/index.ts                          │
└──────┬──────────────────────────┬───────────────────────┘
       │                          │
┌──────▼──────────┐    ┌──────────▼──────────────────────┐
│  crawler.ts     │    │  db.ts  (SQLite WAL)             │
│                 │    │                                  │
│  sitemap.ts     │    │  documents_v2  url·version·etag  │
│  robots.ts      │    │  chunks        heading_path·text │
│  profiles.ts    │    │  chunks_fts    FTS5 BM25 index   │
│  chunker.ts     │    └──────────────────────────────────┘
│  utils.ts       │
└─────────────────┘

Crawl pipeline

🎯 Supported Frameworks

💾 Storage

Data is stored in ~/.universal-docs-mcp/documents.db — survives npx package updates.

~/.universal-docs-mcp/documents.db
├── documents_v2   — full page markdown per (url, version), with etag cache headers
├── chunks         — heading-based sections with JSON heading_path breadcrumb
└── chunks_fts     — FTS5 virtual table powering BM25 search  

Multiple doc versions coexist: crawl with version: "v14" and version: "v15" — they live side by side.

🧪 Tests

# Unit tests — no browser, ~5 seconds
npm run test:unit

# Live integration test — Playwright crawl of cube.dev
npm run test:integration

# Full suite
npm test

Smoke tests (manual/interactive):

npm run smoke:extraction    # Single page extract on example.com
npm run smoke:crawler       # BFS crawler on turndown.js.org
npm run smoke:card          # Salt DS Card component extraction
npm run smoke:mcp-client    # All 6 MCP tools via stdio
npm run smoke:mcp-crawl     # Full crawl + search flow via MCP

📦 Dependencies

All runtime dependencies are MIT licensed — no GPL, LGPL, or copyleft.

¹ Apache-2.0 is permissive (not copyleft). The Chromium binary uses the BSD-style Chromium license.

📄 License

MIT © Rohit