flight-scrappper

Web scraper made with nodejs and webdriverjs that gathers flight data and stores it in a mongodb database.

Requirements

• node
• npm
• chrome
• mongodb

Installing

$ npm install flight-scrappper

Options

The following options can be defined as an argument of the FlightScrappper.run() method.

This can be done passing an object {option1:'abc',option2:'abc',...}.

If an option is not defined, a default value will be used instead.

These are the default values:

let defaultDateFormat = 'DD-MM-YYYY';
var defaultOptions = {
    periods: 1,
    interval: 48,
    routes: [{
        from: 'LIS',
        to: 'PAR'
    }],
    currency: 'USD',
    directFlight: 'false',
    dateFormat: defaultDateFormat,
    targetDate: Utils.getDefaultDateString(defaultDateFormat),
    database: 'localhost:27017/flight-scrappper',
    collection: 'flight-data',
    timeout: 50000,
    browser: 'chrome'
};

During the start, a new parameter dates will be generated. This array will contain dates in string form in the options.dateFormat format.

This dates are calculated with the following formula targetDate + options.interval x options.periods times.

Example: Setting periods to 2, interval to 24 and targetDate to 5/01/2000 will generate an array such as ['5/01/2000','07/01/2000'].

Running

First, start your mongodb database. You can find more information on how to do this here.

If you want to scrap flights, without storing data, you can set database to 'none'.

If you installed mongodb in the default directory you can run $ npm run mongo-win (for windows) or $ npm run mongo-mac (for mac) to run a database that will use a folder named mongo-db in the current directory.

To start the flight-scrappper with the default values just type $ npm start.

If you want to run with diferent options just add arguments as specified in Options.

If you want to get feedback in the console please check Debugging.

Output

FlightScrappper.run will return a promise wich will resolve into the number of inserted documents or into an error.

The resulting data that will be stored in the database has the following fields:

{
    _id, 		
    from,
    to,	
    airline,
    stops,
    time: {
        date,
        departure,
        duration,
        queried
    },
    price:{
        amount,
        currency
    }
}

Tests

$ npm test

Debugging

$ npm run debug to have console output.

Contributing

Contributions, requests or pull requests are welcome & appreciated!

Send me an email if you have questions regarding possible contributions.