Package Exports

clean-web-scraper
clean-web-scraper/main.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (clean-web-scraper) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Web Content Scraper

A powerful Node.js web scraper that extracts clean, readable content from websites while keeping everything nicely organized. Perfect for creating AI training datasets! 🤖

✨ Features

🌐 Smart recursive web crawling of internal links
📝 Clean content extraction using Mozilla's Readability
🧹 Smart content processing and cleaning
🗂️ Maintains original URL structure in saved files
🚫 Excludes unwanted paths from scraping
🔄 Handles relative and absolute URLs like a pro
🎯 No duplicate page visits
📊 Generates JSONL output file for ML training
📊 AI-friendly clean text and csv output (perfect for LLM fine-tuning!)

🛠️ Prerequisites

Node.js (v18 or higher)
npm

📦 Dependencies

axios - HTTP requests master
jsdom - DOM parsing wizard
@mozilla/readability - Content extraction genius

🚀 Installation

npm i clean-web-scraper

# OR

git clone https://github.com/mlibre/Clean-Web-Scraper
cd Clean-Web-Scraper
npm install

💻 Usage

const WebScraper = require('clean-web-scraper');

const scraper = new WebScraper({
  baseURL: 'https://example.com',       // Required: The website to scrape
  scrapResultPath: './output',          // Required: Where to save the content
  excludeList: ['/admin', '/private'],  // Optional: Paths to exclude
  exactExcludeList: ['/specific-page'], // Optional: Exact URLs to exclude
  jsonlPath: 'output.jsonl',            // Optional: Custom JSONL output path
  textOutputPath: "./dataset/texts",    // Optional: Custom text output path
  csvPath: "./dataset/train.csv"        // Optional: Custom CSV output path
});

scraper.start();

node example-usage.js

📤 Output

Your AI-ready content is saved in a clean, structured format:

📁 Base folder: ./folderPath/example.com/
📑 Files preserve original URL paths
📝 Pure text format, perfect for LLM training and fine-tuning
🤖 No HTML, no mess - just clean, structured text ready for AI consumption
📊 JSONL output for ML training
📈 CSV output with clean text content

🤖 AI/LLM Training Ready

The output is specifically formatted for AI training purposes:

Clean, processed text without HTML markup
Consistent formatting across all documents
Structured content perfect for fine-tuning LLMs
Ready to use in your ML pipelines

Standing with Palestine 🇵🇸

This project supports Palestinian rights and stands in solidarity with Palestine. We believe in the importance of documenting and preserving Palestinian narratives, history, and struggles for justice and liberation.

Free Palestine 🇵🇸