Package Exports

web-metadata-scraper
web-metadata-scraper/dist/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (web-metadata-scraper) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Web Metadata Scraper

A TypeScript-based tool for scraping and analyzing metadata from web pages. It can extract various types of metadata including basic meta tags, Open Graph tags, Twitter Card data, and structured data.

Features

Extract comprehensive metadata from web pages
Support for custom HTTP headers
HTML report generation
Website comparison functionality
Configurable through command line or config files

Installation

npm install web-metadata-scraper

Usage

Basic Usage

Scrape metadata and output to console:

npm start -- https://example.com

Use custom headers:

npm start -- https://example.com '{"Authorization": "Bearer token"}'

Generate HTML report:

npm start -- https://example.com --html ./reports/report.html --open

Programmatic Usage

import { scrapeMetadata } from "web-metadata-scraper";

// Basic usage
const metadata = await scrapeMetadata("https://example.com");

// With custom headers
const metadata = await scrapeMetadata("https://example.com", {
  Authorization: "Bearer token",
  "User-Agent": "Custom Bot 1.0",
});

// With custom options
const metadata = await scrapeMetadata(
  "https://example.com",
  {},
  {
    timeout: 5000,
    followRedirect: true,
  },
);

// Access specific metadata
console.log(metadata.basic.title); // Page title
console.log(metadata.opengraph.image); // OG image URL
console.log(metadata.scripts.gtm.ids); // Google Tag Manager IDs
console.log(metadata.structured); // JSON-LD data

Comparison Mode

Compare metadata between two websites:

Using command line:

npm start -- --compare https://site1.com https://site2.com --html ./reports/comparison.html --open

Using config file (config.json):

{
  "urls": ["https://site1.com", "https://site2.com"],
  "headers": {
    "Authorization": "Bearer token"
  },
  "output": {
    "html": "./reports/comparison.html",
    "openHtml": true
  }
}

Then run:

npm start -- --config config.json

API Reference

Main Functions

`scrapeMetadata(url, headers?, options?)`

Scrapes metadata from a given URL.

url: string - The URL to scrape
headers: object (optional) - Custom HTTP headers
options: object (optional) - Request options
Returns: Promise

interface Metadata {
  basic: {
    title: string;
    description: string;
    keywords: string[];
    // ...
  };
  opengraph: {
    title: string;
    description: string;
    image: string;
    // ...
  };
  twitter: {
    card: string;
    title: string;
    description: string;
    // ...
  };
  scripts: {
    gtm: {
      ids: string[];
      implementation: string;
    };
    analytics: {
      ga: string[];
      gtag: string[];
      ua: string[];
      ga4: string[];
    };
    other: string[];
  };
  structured: any[];
}

Advanced Examples

Extract only Google Analytics information:

const { scripts } = await scrapeMetadata("https://example.com");
const { ga4, ua } = scripts.analytics;
console.log("GA4 IDs:", ga4);
console.log("Universal Analytics IDs:", ua);

Extract all JSON-LD data:

const { structured } = await scrapeMetadata("https://example.com");
const jsonLdData = structured.filter((item) =>
  item["@context"]?.includes("schema.org"),
);

Custom error handling:

try {
  const metadata = await scrapeMetadata(
    "https://example.com",
    {},
    {
      timeout: 3000,
      followRedirect: false,
    },
  );
} catch (error) {
  if (error.message.includes("timeout")) {
    console.error("Request timed out");
  } else if (error.message.includes("Invalid URL")) {
    console.error("Invalid URL provided");
  } else {
    console.error("Failed to fetch metadata:", error);
  }
}

Configuration

You can provide configuration either through command line arguments or a config file.

Command Line Arguments

<url> - URL to scrape
--compare <url1> <url2> - Compare two URLs
--html <path> - Generate HTML report
--open - Open HTML report after generation
--config <path> - Use config file

Config File Format

{
  // Single URL mode
  "url": "https://example.com",

  // OR Comparison mode
  "urls": ["https://site1.com", "https://site2.com"],

  // Optional fields
  "headers": {
    "Authorization": "Bearer token"
  },
  "options": {
    "timeout": 5000,
    "followRedirect": true
  },
  "output": {
    "html": "./reports/report.html",
    "openHtml": true
  }
}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.