JSPM

Simplify data extraction from document

This CLI tool will find same domain urls in a web page and requesting them to find even more urls until server crash (or at the end of benchmark). It is used to test maximun capacity of server or finding for glitches that users might encounter.

A configuration - based crawler framework

Web crawling tool

Jacob's Crawler, the CLI version.

A web scraper for the Bosnian listings site olx.ba

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

This library provides support for traversing objects and their values while providing information on the traversal state, pathing to target values, and the ability to manipulate said pathing to easily move to related values.

Get product info by barcode

A fast crawler cli with pyppteer, this crawler can crawl SPA(single page application)

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

a website-crawler library for nodejs

A web scraper for the Serbian listings site KupujemProdajem

Sitemap plugin

A helper that build a x-ray based on a schema

A Bing command line dictionary, which obtains the query results of bing dictionary by crawler.

Crawl the log of any command

A crawler framework based on NodeJS.

here is a package for crawling and downloading videos from youtube

A fast and stable DHT crawler.

To scrap the content from the web site

A util tool

Parkour the web like a yamakazi

Website schema based crawler

User Agent

Powerful Scraping and Crawling library with anti-scraping, scalability, storage, static/dynamic contents, monitoring UI and more. Ready to deploy on cloud instances or serverless.

Nexstack Nodejs library that provides an Api for obtaining the movies information website.

HTTP request module customized for crawlers.

x-ray's crawler

Get information using the string of the specified rule

crawler

基于 MCP 的网页爬取服务器，内置 Puppeteer 无头浏览器支持

A powerful and flexible web scraper library built with TypeScript

Generate LLM-friendly text files from Next.js applications by crawling sitemaps and extracting content

A web crawler. Automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Templates for the crawlee projects

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Crawl an instagram profile by id

The unofficial HLTV Node.js API

Provides APIs by simple configuration.

Just Another Sitemap Generator

High-performance web crawler implemented in Go with JavaScript bindings

a module used for scraping with either an electron browser (webview, BrowserWindow, BrowserView) or http requests (axios)

Fetch the pre-rendered content, meta, links and Open Graph of a webpage, especially Single-Page Application (SPA)

A lightweight and simple API for web crawling built on chromium puppeteer

Convert a website to static markdown.

A tool to allow for quick running of JSON-based scrapers using request-promise and jsonframe-cheerio.

A search and crawler for Wikipedia articles

A SoundCloud Downloader made with Node and Typescript

Verify that a request is from Google crawlers using Google's DNS verification steps

Verify that a request is from Baidu crawlers using Baidu's DNS verification

Distributed web crawler powered by Headless Chrome

Memory efficient and synchronous downloader of map tiles. Allows for a fast and easy approach to make map tiles (from a WMS) available offline.

Webcrawler for data mining and unification purposes

模仿scrapy的node爬虫框架

Functional web scraping in typescript

A twitter client for agents

过早客论坛信息获取 MCP (Model Context Protocol) 服务器

A sophisticated website comparison tool with intelligent content analysis and offset-aware difference detection