JSPM

Just Another Sitemap Generator

High-performance web crawler implemented in Go with JavaScript bindings

a module used for scraping with either an electron browser (webview, BrowserWindow, BrowserView) or http requests (axios)

Fetch the pre-rendered content, meta, links and Open Graph of a webpage, especially Single-Page Application (SPA)

A lightweight and simple API for web crawling built on chromium puppeteer

Convert a website to static markdown.

A tool to allow for quick running of JSON-based scrapers using request-promise and jsonframe-cheerio.

A search and crawler for Wikipedia articles

A SoundCloud Downloader made with Node and Typescript

Verify that a request is from Google crawlers using Google's DNS verification steps

Verify that a request is from Baidu crawlers using Baidu's DNS verification

Distributed web crawler powered by Headless Chrome

Memory efficient and synchronous downloader of map tiles. Allows for a fast and easy approach to make map tiles (from a WMS) available offline.

Webcrawler for data mining and unification purposes

模仿scrapy的node爬虫框架

Functional web scraping in typescript

A twitter client for agents

过早客论坛信息获取 MCP (Model Context Protocol) 服务器

A sophisticated website comparison tool with intelligent content analysis and offset-aware difference detection