JSPM

Reliable web access for AI agents.
Fetch any page · Extract structured data · Crawl entire sites · Deep research — one tool, three interfaces.

Website · Docs · Playground · Dashboard · Discussions

What is WebPeel?

WebPeel gives your AI agent reliable access to the web. Fetch any page, extract structured data, crawl entire sites, and research topics — all through a single CLI, API, or MCP server.

It automatically handles the hard parts: JavaScript rendering, bot detection, Cloudflare challenges, infinite scroll, pagination, and content noise. Your agent gets clean markdown. You don't think about the plumbing.

🚀 Quick Start

Three paths in, all free to try:

CLI

npx webpeel "https://news.ycombinator.com"

No install needed. First 25 fetches work without signup. Get 500/week free →

MCP Server (for Claude, Cursor, VS Code, Windsurf)

Add to your MCP config (~/Library/Application Support/Claude/claude_desktop_config.json on Mac):

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["-y", "webpeel", "mcp"]
    }
  }
}

REST API

curl "https://api.webpeel.dev/v1/fetch?url=https://example.com" \
  -H "Authorization: Bearer wp_YOUR_KEY"

✨ Features

Core

AI Agent

Extraction

Anti-Bot

Advanced

🤖 MCP Integration

WebPeel exposes 13 tools to your AI coding assistant:

{
  "mcpServers": {
    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
  }
}

{
  "mcpServers": {
    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
  }
}

{
  "servers": {
    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
  }
}

{
  "mcpServers": {
    "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"] }
  }
}

{
  "mcpServers": {
    "webpeel": { "command": "docker", "args": ["run", "-i", "--rm", "webpeel/mcp"] }
  }
}

🔬 Deep Research

Multi-hop research that thinks like a researcher, not a search engine:

# Sources only — no API key needed
npx webpeel research "best practices for rate limiting APIs" --max-sources 8

# Full synthesis with LLM (BYOK)
npx webpeel research "compare Firecrawl vs Crawl4AI vs WebPeel" --llm-key sk-...

How it works: Search → fetch top results → extract key passages (BM25) → follow the most relevant links → synthesize across sources. No circular references, no duplicate content.

📦 Extraction

Three ways to get structured data out of any page:

CSS Schema (zero config, auto-detected)

# Auto-detects Amazon and applies the built-in schema
npx webpeel "https://www.amazon.com/s?k=mechanical+keyboard" --json

# Force a specific schema
npx webpeel "https://www.booking.com/searchresults.html?city=Paris" --schema booking --json

# List all built-in schemas
npx webpeel --list-schemas

Built-in schemas: amazon · booking · ebay · expedia · hackernews · walmart · yelp

JSON Schema (type-safe structured extraction)

npx webpeel "https://example.com/product" \
  --extract-schema '{"type":"object","properties":{"title":{"type":"string"},"price":{"type":"number"}}}' \
  --llm-key sk-...

LLM Extraction (natural language, BYOK)

npx webpeel "https://hn.algolia.com" \
  --llm-extract "top 10 posts with title, score, and comment count" \
  --llm-key $OPENAI_API_KEY \
  --json

import { peel } from 'webpeel';

// CSS selector extraction
const result = await peel('https://news.ycombinator.com', {
  extract: {
    selectors: {
      titles: '.titleline > a',
      scores: '.score',
    }
  }
});
console.log(result.extracted); // { titles: [...], scores: [...] }

// LLM extraction with JSON Schema
const product = await peel('https://example.com/product', {
  llmExtract: 'title, price, rating, availability',
  llmKey: process.env.OPENAI_API_KEY,
});

🛡️ Stealth & Anti-Bot

WebPeel detects 7 bot-protection vendors and handles them automatically:

• Cloudflare (JS challenge, Turnstile, Bot Management)
• PerimeterX / HUMAN (behavioral analysis)
• DataDome (ML-based bot detection)
• Akamai Bot Manager
• Distil Networks
• reCAPTCHA / hCaptcha (page-level detection)
• Generic challenge pages

28 high-protection domains (Amazon, LinkedIn, Glassdoor, Zillow, Ticketmaster, and more) automatically route through stealth mode — no flags needed.

# Explicitly enable stealth
npx webpeel "https://glassdoor.com/jobs" --stealth

# Auto-escalation (stealth triggers automatically on challenge detection)
npx webpeel "https://amazon.com/dp/ASIN"

⚡ Benchmark

Evaluated on 30 real-world URLs across 6 categories (static, dynamic, SPA, protected, documents, international):

WebPeel is the only tool that extracted content from all 30 test URLs. Full methodology →

🆚 Comparison

💳 Pricing

All features on all plans. Pro/Max add pay-as-you-go extra usage (fetch $0.002, search $0.001, stealth $0.01). Quota resets every Monday.

Sign up free → · Compare with Firecrawl →

🐍 Python SDK

pip install webpeel

from webpeel import WebPeel

client = WebPeel(api_key="wp_...")  # or use WEBPEEL_API_KEY env var

# Fetch a page
result = client.scrape("https://example.com")
print(result.content)    # Clean markdown
print(result.metadata)   # title, description, author, ...

# Search the web
results = client.search("latest AI research papers")

# Crawl a site
job = client.crawl("https://docs.example.com", limit=100)

# With browser + stealth
result = client.scrape("https://protected-site.com", render=True, stealth=True)

Sync and async clients. Pure Python 3.8+, zero dependencies. Full SDK docs →

🐳 Self-Hosting

git clone https://github.com/webpeel/webpeel.git
cd webpeel && docker compose up

Full REST API available at http://localhost:3000. AGPL-3.0 licensed. Self-hosting guide →

Just the MCP server:

docker run -i webpeel/mcp

Just the API server:

docker run -p 3000:3000 webpeel/api

📖 API Reference

Full OpenAPI spec at openapi.yaml and api.webpeel.dev.

# Fetch
GET  /v1/fetch?url=<url>

# Search
GET  /v1/search?q=<query>

# Crawl
POST /v1/crawl  { "url": "...", "limit": 100 }

# Map
GET  /v1/map?url=<url>

# Extract
POST /v1/extract  { "url": "...", "schema": { ... } }

Full API reference →

🤝 Contributing

git clone https://github.com/webpeel/webpeel.git
cd webpeel
npm install && npm run build
npm test

• Bug reports: Open an issue
• Feature requests: Start a discussion
• Code: See CONTRIBUTING.md for guidelines

The project has a comprehensive test suite. Please add tests for new features.

Star History

License

AGPL-3.0 — free to use, modify, and distribute. If you run a modified version as a network service, you must release your source under AGPL-3.0.

Need a commercial license? support@webpeel.dev

Versions 0.7.1 and earlier were released under MIT and remain MIT-licensed.

If WebPeel saves you time, ⭐ star the repo — it helps others find it.

© WebPeel