JSPM

Analyzes license information for multiple node.js modules (package.json files) as part of your software project.

JavaScript module detecting bots/crawlers/spiders via user-agent

A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Automatically extracts structured information from webpages

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

Extract data from a pdf with pure javascript

Easily create XML sitemaps for your website.

Stop website fingerprinting techniques

Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

JavaScript SDK for Firecrawl API

A port of n0madic/twitter-scraper to Node.js.

Simple, lightweight and expressive web scraping with Node.js

Node.js agent for Sqreen, please see https://www.sqreen.io/

x-ray's crawler

A web scraper for NodeJs

HTTP request module customized for crawlers.

http request for web scraping

Create xml sitemaps from the command line.

Beautiful-dom is a lightweight library that mirrors the capabilities of the HTML DOM API needed for parsing crawled HTML/XML pages. It models the methods and properties of HTML nodes that are relevant for extracting data from HTML nodes. It is written in

[](https://www.npmjs.com/package/recrawl) [](https://github.com/aleclarson/recrawl/actions/workflows/release.yml) [![codeco

Node.js SDK for Crawlab

Stop website fingerprinting techniques playwright edition

blocklet crawler lib

Hyperbrowser Model Context Protocol Server

This plugin links your Netlify site with Algolia's Crawler. It will trigger a crawl on each successful build.

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

Dependency free module for scraping and crawling websites using [Crawlbase](https://crawlbase.com) API

A Model Context Protocol (MCP) server implementation that provides real-time web search capabilities through a simple API

Bright CLI is a CLI tool that can initialize, stop, poll and maintain scans in Bright solutions.

Shared functionality for implementing device detection engine for the 51Degrees Pipeline API

Device detection on-premise services for the 51Degrees Pipeline API

Advanced MCP server for web scraping with nested URL fetching and intelligent markdown formatting

Finds broken links and resources on websites

Headless Chrome abstraction to simplify the interaction with the browser. It may be used for crawling sites, test automation, etc

Crawler (spider) of site web pages by domain name

A JavaScript library that allows for the quick transformation of DOM documents into useful formats.

Model Context Protocol (MCP) server for pure.md, the markdown delivery network for LLMs

A powerful and flexible web scraping library with concurrent processing and DOM hierarchy awareness

Linkd Model Context Protocol Server

Device detection cloud services for the 51Degrees Pipeline API

Next.js robots.tsx generator - Automatically create and serve robots.txt for Next.js applications

Parse HTTP headers to detect the device type, model, operating system, browser, and crawler information

Automatically extracts structured information from webpages

http client module with cheerio & iconv(-lite) & promise

An easy-to-use Node web crawler storing cookies, following redirects, traversing pages and submitting forms.

🚀 MCP SERVER FIXED v3.7.9! Resolved import errors, middleware conflicts, type hints - NOW WORKING PERFECTLY!

一个强大的网站链接抓取工具，支持深度抓取、认证和页面分析

crawl youtube without api key (search videos channels or get all channel/playlist's videos)

JS client for WebcrawlerAPI

Automatically extracts structured information from webpages

impit-based HTTP client implementation for Crawlee. Impersonates browser requests to avoid bot detection.

Utils library for harvesting RPDE feeds

Take a snapshot of any website.

Recursively read a directory, blazing fast.

Yet another node torrent scraper based on x-ray. (Support iptorrents, torrentleech, torrent9, Yyggtorrent, ThePiratebay, torrentz2, 1337x, KickassTorrent, Rarbg, TorrentProject, Yts, Limetorrents, Eztv)

🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.

A high-performance web crawler powered by Bun that downloads pages and converts them to Markdown

Distributed web crawler powered by Headless Chrome

To install:

Lightweight async scraper for Google News

Fast, token-efficient web content extraction - fetch web pages and convert to clean Markdown

An easy to use CLI for downloading websites for offline usage

Analyzes license information for multiple node.js modules (package.json files) as part of your software project.

JavaScript SDK for Firecrawl API

a Test Node

The [spider](https://github.com/spider-rs/spider) project ported to Node.js

A simple email extractor for obfuscated emails.

The unofficial HLTV Node.js API

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

Verify that a request is from Google using Google's recommended DNS verification steps

An attestate crawler strategy to download and transform Ethereum block event logs

Generic web crawler powered by Node.js

A web crawler for Nodejs.

Webpage crawler for qualweb

Keep Bots Away From Your Express App

Smart SQL injection scanner with crawler and optional Playwright capture.

novel downloader for node-novel style , include site ( dmzj / wenku8 / syosetu / ...etc )

A sophisticated website comparison tool with intelligent content analysis and offset-aware difference detection

dcinside 갤러리 크롤링을 위한 Node.js 라이브러리

Unofficial API lib for Nintendo Switch eShop game listing and pricing information.

crawls a npm package and it's dependencies for their licenses

Extract emails from text and also from a site page

Web crawler for Node.js

Lightweight, runtime-safe crawling → clean Markdown

Efficient SEO-focused server for Wasm-generated pages

Detect search engine crawlers by their User-Agent strings.

Crawl and download Snap Lenses from *lens.snapchat.com* with ease.

x-crawl is a flexible Node.js AI-assisted crawler library.

Unified browser / HTML controller interfaces that support patchright, camoufox, playwright, puppeteer and cheerio

A twitter client for agents with notifications support

An `URL` parser for crawling purpose.

A Node crawler/scrape for retrieving data from websites

Priority based Semantic Web Crawler.

crawler toolbox

Detect SEO Bot Crawler

Web scraper for NodeJS

Scrape public available jobs on Linkedin using headless browser

A web crawler made for the SEO based on plugins. Please wait or contribute ... still in beta

A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.

A library for checking basic SEO signals of a website

Web scraper that scraping web pages by LetsScrapeData XML template

A crawler, to download web-site

A robust GitHub API crawler that walks a queue of GitHub entities retrieving and storing their contents.

A client for fetching stock data from the Tehran Stock Exchange (TSETMC). Works in Browser, Node and as CLI.

Crawls information from public netatmo stations

Fetch web data as easy as possible

Crawl web as easy as possible

gRPC tokio based web crawler

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

A web crawler module designed to scarp data from Ptt.

The unofficial HLTV Node.js API

TypeScript types for defining custom metrics

`algoliasearch-netlify-frontend` is the front-end bundle we recommend to use with our Netlify plugin.

Detect user-agent is a bot/spider/crawler

Generate AI-ready optimization files for websites, including robots.txt, sitemaps, and AI manifests

Node.js SDK for interacting with WebsiteCrawler.org API

A comprehensive web scraping library with resumable operations, middleware support, and built-in rate limiting

NodeJS User-Agent String Parser based on Udger SQLite databases https://udger.com/products/local_parser

Secure your puppeteer for scraping

deepcrawl cli

Basic page crawler written in nodejs

Super configurable async web spider

linkedin scraper for 2020 website

An extremely lightweight HTTP request client for the command-line. Supports: http, https, proxy, redirects, cookies, content-encoding, multipart/form-data, multi-threading, recursive website crawling and mirroring.

An easiest crawling and scraping module for NestJS

Distributed web crawler powered by Headless Chrome

Simple, flexible, delightful web crawler/spider package

Node.js web crawler to get all internal links from a website.

Crawler queue creation tool for paging

Crawl a site to generate a backstopjs config

A web page content extractor for News websites

Official TypeScript SDK for Crawlbyte – create tasks, poll results, and integrate data scraping into your JavaScript/TypeScript applications.

Typed TypeScript/JavaScript SDK for Supacrawler API (scrape, jobs, screenshots, watch)

MCP server for Supadata video & web scraping integration. Features include YouTube, TikTok, Instagram, Twitter, and file video transcription, web scraping, batch processing and structured data extraction.

🛡️ Nuxt 3 middleware to block suspicious bots, protect SEO crawlers with reverse DNS checks, and enforce User-Agent rules.

Vercel integration for SnapCrawl. Serve pre-rendered HTML to crawlers in Next.js middleware or Edge Functions for static SPAs and Express apps.

proxy manager used to scrape data

Deterministic link harvesting for QA and website migration testing

DOM Document Object Artifact Collector

A Node.js module for downloading a single image or multiple images to disk from a given Url (checking if url exist and detecting image type)

A powerful website downloader with GUI support

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

spankbang.com api implementation

Web service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv.

A library for checking basic SEO signals of a website

CrawleeOne is a framework built on top of Crawlee and Apify for writing robust and highly configurable web scrapers

A node crawler that return all links/href from website

🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.

Multi environment web page parser

Crawler(robots) decision middleware for Express

A CLI tool to crawl documentation sites and create a search index for Upstash Search.

A library for crawling a filesystem tree, based on Effect-ts

Generate comprehensive PDFs of entire websites, ideal for RAG.

Tool to crawl events, leagues and statistics from WBSC based websites.

Behaviour Assertion Sheets: CSS-like declarative syntax for client-side integration testing and quality assurance.

SyphonX is a tool that extracts data from HTML data, transforming it into JSON of any shape or size. It combines the power of CSS Selectors and jQuery, Regular Expressions, and Javascript into a declarative template format to elegantly solve the simplest

Stealth mode: Applies various techniques to make detection of headless puppeteer harder.

extract emails address from website by following links

A utility library to make downloading & extracting specific content from a URL easy

MCP server for Firecrawl web scraping integration. Supports both cloud and self-hosted instances. Features include web scraping, batch processing, structured data extraction, and LLM-powered content analysis.

Fetch the pre-rendered content, meta, links and Open Graph of a webpage, especially Single-Page Application (SPA)

Get Aliexpress product details as a json reponse including feedbacks, variants, description, images, etc.,

Crawl and download Snap Lenses from *lens.snapchat.com* with ease.

The directory crawler library for Node.JS

Easily create XML sitemaps for your website.

Web Download CLI

A Vue3 plugin to improve SEO and crawler accessibility

Web crawler MCP server for extracting text content from web pages

Stealth mode: Applies various techniques to make detection of headless puppeteer harder.

Lightweight, runtime-safe crawling → clean Markdown

Simple manga scrapper for famous online manga websites.

Nodejs library that provides an Api for obtaining the movies information from FlixHQ website.

Web page scraper with a jQuery-like syntax for Node.

Lightweight and easy to use crawling solution for websites.

A highly configurable website crawler for automatically testing a website for accessibility issues using the axe-core library. Uses selenium and headless Chrome to load pages, inject axe-core, and run tests. Generates an html summary report in addition

Environment variable collector

Node.js module for crawling the web

A crawler implemented using a headless browser (Chrome).

A util tool

xvideos.com api implementation.

Smart MCP tool to find and validate movie/tv-show resources with multiple sources support

use google play protobuf api in node

A Node.JS Web crawler using the API Fetch

A crawler for a dhis2 instance to pull data from the site.

A TypeScript library for scraping stories from various Vietnamese websites

A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format.

Generic node service to handle SSR for SPA made with any kind of frontend framework

Core crawler framework functionality - TypeScript web crawling library

Utility functions for web crawling - sitemap processing, link extraction, system info

xvideos.com api implementation.

A promised based node module to scrape TV shows, episodes and torrent info from EZTV.

Crawlyx is an open-source command-line interface (CLI) based web crawler built using Node.js. It is designed to crawl websites and extract useful information like links, images, and text. It is lightweight, fast, and easy to use.

🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.

API crawler for REST and GraphQL endpoint crawling with auto-detection

A socks and http proxy by nodejs for you to over GWF

A util tool

HTTP crawler for basic web scraping without JavaScript execution

爬虫下载插件

Scrape data from any webpage.

Easily build flexible, scalable, and distributed, web crawlers.

crawler with nodejs

A crawler to retrieve store data from the Epic Games store

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

An extremely simple module to web scrapper a DOM element(s)

Access Google Play by logging in and making requests as an Android device!

self twitter scraper

Floodesh is a distributed web spider/crawler written with Nodejs.

Funções para os rastreadores ABSOLUT Mobile

MCP server for documentation crawling, indexing, and retrieval

Express middleware that returns the resulting html after executing javascript, allowing crawlers to read on the page

Botify Rest API SDK middlewares

Multi-thread crawler engine.

Get Aliexpress product details as a json reponse including feedbacks, variants, description, images, etc.,

Auxiliary package of functions for the TypeScript framework Botmation

A TypeScript library for detecting and categorizing bots from user agent strings

A simple license crawler for crediting open source work

一个专门用于爬取csdn文章的爬虫/A JS library for Crawl CSDN Article.

Get information using the string of the specified rule

google search crawler

Youtube Crawler with no API that returns 3 first videos.

A robots.txt reader, parser and matcher.

Playwright-based crawler for full browser automation and JavaScript rendering

Puppeteer-based crawler for Chrome automation and dynamic content scraping

A util tool

MCP server for Crawlbase API - enables web scraping through Model Context Protocol

web crawler based on Puppeteer

crawler

crawls mysql database and creates insert queries or returns data from multiple table depending on the relationship information of the tables provided

A bitcoin news crawler

Web crawler to use as API

轻小说文库下载器

A simple, configurable and functional content scraper

Fast Implemented TikTok Scraping API

爬虫公共代码

The simple and fast crawling framework. So you can focus on scraping.

A flexible and powerful library designed to extract and transform data from HTML documents using user-defined schemas

Model Context Protocol server for WebScraping.AI API. Provides LLM-powered web scraping tools with Chromium JavaScript rendering, rotating proxies, and HTML parsing.

Command-line interface for creating and managing crawler projects

A library and CLI tool to recursively collect links from a given initial URL and output them as structured data

A job search and crawling tool built with Model Context Protocol

Gracefully handle timeout and network error with auto retry.

Tool for easy scraping data from websites

command line interface to crawl a rippled network

Walk an entire directory. Fast, simple, and asynchronous.

web spider, support puppeteer, cheerio and so on, include task-queue and dispatcher