Package Exports

sqlite-simplecrawler-queue

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (sqlite-simplecrawler-queue) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

SQLite Implementation of FetchQueue Interface for Simplecrawler

This is an implementation of FetchQueue Interface for simplecrawler queue with SQLite usage as backend.

Preferences:

Possibility to pause/stop/kill/terminate running job without queue state losing
Possibility to run one crawler job in parallel with several crawler instances using one queue.

Installation

Install from github

npm install git+https://github.com/LeMoussel/SQLite-simplecrawler-queue#master

Install from npm

npm install --save SQLite-simplecrawler-queue

Usage

All you need is

the database information such as database file
the name of the queue table

try {
  const sqliteDatabaseName = 'crawlsite.sqlite3'

  // Drop Database if exist
  SQLiteFetchQueue.dropDatabase(sqliteDatabaseName)

  // Connect to a disk file database, you pass the path to the database file.
  const crawlerQueue = new SQLiteFetchQueue(sqliteDatabaseName, 'queue')

  // Initialization of the database
  crawlerQueue.init()

  // Initializing simplecrawler
  const crawler = new Crawler('https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Accueil_principal')
  crawler.maxDepth = 3
  crawler.allowInitialDomainChange = false
  crawler.filterByDomain = true
  crawler.queue = crawlerQueue
  crawler.start()
} catch (err) {
  console.error(err)
}

Additional utilities

You can fully drop

the queue using dropQueue method.

// Connect to a disk file database, you pass the path to the database file.
const crawlerQueue = new SQLiteFetchQueue('sqliteDatabaseName', 'queue')
// Drop 'queue' table
crawlerQueue.dropQueue
// Initialization of the database
crawlerQueue.init()

drop the databse using SQLiteFetchQueue.dropDatabase static method.

// Drop Database if exist
SQLiteFetchQueue.dropDatabase('sqliteDatabaseName')
// Connect to a disk file database, you pass the path to the database file.
const crawlerQueue = new SQLiteFetchQueue('sqliteDatabaseName', 'queue')
// Initialization of the database
crawlerQueue.init()