simplecrawler
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
Found 14 results for simplecrawler
semantic-crawler
Priority based Semantic Web Crawler.
web-spider
web-spider is a simple and fast web spider written with Nodejs!
simplecrawler-zic
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
simplecrawler-mongo-queue
MongoDB queue implementation for simplecrawler
mongodb-simplecrawler-queue
MongoDB FetchQueue Implementation for Simplecrawler
ispider
iSpider is a simple and fast web spider written with Nodejs!
spiderhunt
Very straightforward web crawler. Uses EventEmitter. Based on Simplecrawler but using a distributed queue system.
simplecrawler-referrer-filter
Very straigntforward web crawler. Uses EventEmitter. Generates queue statistics and has a basic cache mechanism with extensible backend. This version is forked in order to add the ability to filter by referrer url
simplecrawler-fork
Forked version of https://github.com/ciffard/node-simplecrawler.
jes-spider
A web crawler forked from simplecrawler.
sqlite-simplecrawler-queue
SQLite FetchQueue Implementation for Simplecrawler
@gombosg/simplecrawler
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
jespider
A web crawler forked from simplecrawler.