🕷️ WebsiteCrawler SDK

A Node.js SDK for interacting with the WebsiteCrawler.org API. This SDK allows developers to submit URLs for crawling, monitor progress in real time, and retrieve structured metadata in JSON format—perfect for LLM ingestion, chatbot training, or website audits.

📦 Installation

npm install website-crawler-sdk

Usage Example

const { WebsiteCrawlerConfig, WebsiteCrawlerClient } = require('website-crawler-sdk');

const config = new WebsiteCrawlerConfig('YOUR_API_KEY');
const client = new WebsiteCrawlerClient(config);

client.submitUrlToWebsiteCrawler('YOUR_URL', 'YOUR_LIMIT');

const intervalId = setInterval(() => {
  const status = client.getCrawlStatus();
  console.log('Status:', status);
  console.log('Current URL:', client.getCurrentURL());

  if (status === 'Completed!') {
    console.log('Crawl Data:', client.getCrawlData());
    console.log('Job completed...');
    clearInterval(intervalId);
  }
}, 3000);

Authentication

To use the API, you'll need an API key:

Visit WebsiteCrawler.org
Create a free account or log in
Navigate to the Settings page
Generate your API key