@purepageio/fetch-engines

Fetch websites with confidence. @purepageio/fetch-engines gives teams an HTTP-first workflow that automatically promotes tricky pages to a managed Playwright browser and can even hand structured results back through OpenAI.

Table of contents

• Why fetch-engines?
• Installation
• Quick start
• Usage patterns
  • Pick an engine
  • Structured extraction
• Configuration
  • Essentials
  • Complete reference
• Error handling
• Tooling and examples
• Contributing
• License

Why fetch-engines?

• One API for multiple strategies – Call fetchHTML for rendered pages or fetchContent for raw responses. The library handles HTTP shortcuts and Playwright fallbacks automatically.
• Production-minded defaults – Retries, caching, and consistent telemetry are ready out of the box.
• Drop-in AI enrichment – Provide a Zod schema and let OpenAI (or any OpenAI-compatible API) convert full pages into structured data.
• Typed and tested – Built in TypeScript with examples that mirror real-world scraping pipelines.

Installation

pnpm add @purepageio/fetch-engines
# install Playwright browsers once if you plan to use the Hybrid or Playwright engines
pnpm exec playwright install

Quick start

import { HybridEngine } from "@purepageio/fetch-engines";

const engine = new HybridEngine();

const page = await engine.fetchHTML("https://example.com");
console.log(page.title);

await engine.cleanup();

Usage patterns

Pick an engine

Structured extraction

import { fetchStructuredContent } from "@purepageio/fetch-engines";
import { z } from "zod";

// IMPORTANT: All schema fields must have .describe() calls to guide the AI model
const schema = z.object({
  title: z.string().describe("The title of the article"),
  summary: z.string().describe("A brief summary of the article content"),
});

// model is required - use any model supported by your API provider
const result = await fetchStructuredContent("https://example.com/article", schema, { model: "gpt-4.1-mini" });

console.log(result.data.summary);

Set OPENAI_API_KEY (or OPENROUTER_API_KEY) before running structured helpers, or use apiConfig to connect to OpenAI-compatible APIs like OpenRouter. The engine automatically adds the Authorization header when you provide an API key:

const result = await fetchStructuredContent("https://example.com/article", schema, {
  model: "anthropic/claude-3.5-sonnet",
  apiConfig: {
    apiKey: process.env.OPENROUTER_API_KEY,
    baseURL: "https://openrouter.ai/api/v1",
    headers: {
      "HTTP-Referer": "https://your-app.com",
      "X-Title": "Your App Name",
    },
  },
});

When you supply a custom baseURL, the engine automatically switches to the Vercel AI SDK's createOpenAICompatible provider (instead of createOpenAI) so OpenAI-compatible services like OpenRouter receive the correct API-key auth flow.

Configuration

Essentials

All engines accept familiar fetch options such as custom headers. Additional Hybrid/Playwright options you are likely to tweak:

• markdown – return Markdown instead of HTML.
• spaMode & spaRenderDelayMs – allow single-page apps to render before extraction.
• cacheTTL, maxRetries, and browser pool sizes – control resilience and throughput.

Check the inline TypeScript docs or the /examples directory for end-to-end flows.

Complete reference

Every option from PlaywrightEngineConfig (consumed by HybridEngine) with defaults:

Per-request overrides: fetchHTML accepts fastMode, markdown, spaMode, and headers, while fetchContent supports fastMode and headers.

Error handling

Failures raise a typed FetchError exposing code, statusCode, and the underlying error. Log these fields to diagnose issues quickly and tune your retry policy.

Tooling and examples

• Explore the examples directory for scripts you can run end-to-end.
• Ready-to-use TypeScript types ship with the package.
• pnpm test runs the automated suite when you are ready to contribute.

Contributing

Issues and pull requests are welcome! Please follow the existing linting/test commands before sending a change.

License

Distributed under the MIT license.