Package Exports

flipkart-scraper

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (flipkart-scraper) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Flipkart Scraper

This package will help us to scrape all Flipkart products through Flipkart affiliate API.

Please refer API Documentation here.

See the Examples here

Installation

Install using 'npm'

npm i flipkart-scraper

Install using 'yarn'

yarn add flipkart-scraper

Usage

import { FlipkartScraper } from "flipkart-scraper";

const scraper = new FlipkartScraper(
  "<Affiliate-Id-Here>",
  "<Affiliate-Token-Here>"
);

// 'data' event handler
scraper.on("data", (data) => {
  console.log(data.products.length);
});

// Start the scraper
scraper.start();

Class: FlipkartScraper

extends EventEmitter

This module will help us to scrape all the Flipkart products.

constructor(affiliateId, affiliateToken, [options])

This will create instance of the FlipkartScraper class with required authentication params.

options {object}
- concurrency? {number} Number for parallel processing in the queue, Default set to 2
- maxRequest? {number} Maximum request to Flipkart affliate server, Default set to 0 - means unlimited
- maxPage? {number} Maximum number of pages to scrape per category, Default set to 0 - means unlimited

Example

const scraper = new FlipkartScraper(
  "<Affiliate-Id-Here>",
  "<Affiliate-Token-Here>",
  {
    /**
     * It will make 5 parallel request to Flipkart.
     * This is optional param, default is set to 2
     **/
    concurrency: 5,
    /**
     * It will make only 500 request to Flipkart. After that program ends.
     * This is optional param, default is set to 0 means unlimited
     **/
    maxRequest: 500,
    /**
     * Maximum 3 request per category
     **/
    maxPage: 3,
  }
);

scraper.start([categoriesToScrape])

This method will start scraping through Flipkart affiliate API.

categoriesToScrape? {string[]} Pass the list of categories that you want to scrape. Default set to [] which means all categories.

Example

scraper.start(["telivision", "mobiles"]); // It will scrape only specified categories

scraper.stats(showAsNumbers?=false)

This method will show the stats of scraper. By default stats will come as numerals like (3.1k, 1.45GB)

showAsNumbers? {boolean} Stats will return as number instead of numerals.

Sample Stats

/*
{
  "startTime": "2021-02-21T08:14:29.445Z",
  "endTime": undefined, // endTime will be available once scraping finished
  "status": "inprogress",
  "concurrency": 30,
  "waitingRequests": 0,
  "productsCount": "19.54k",
  "elapsed": "0:00:06 10ms",
  "durationPerMillionProducts": "0:05:08 558ms",
  "productsPerSec": "3.25k products/sec",
  "avgResponseTime": "147ms",
  "requestPerSec": "7/sec",
  "requestedCount": "43.00",
  "processedCount": "41.00",
  "errorCount": "0.0",
  "retryCount": "0.0",
  "retryHaltCount": "0.0",
  "pendingCategory": 2,
  "completedCategory": 1,
  "downloadSize": "84.78MB",
  "downloadedSpeed": "14.11MB/sec",
  "info": {
    "pendingCategories": [
      {
        "category": "mens_clothing",
        "startTime": "2021-02-21T08:14:29.602Z",
        "noOfPages": 18,
        "elapsed": 5668,
        "totalProducts": 9000
      },
      {
        "category": "mobiles",
        "startTime": "2021-02-21T08:14:29.603Z",
        "noOfPages": 13,
        "elapsed": 5771,
        "totalProducts": 6500
      }
    ],
    "completedCategories": [
      {
        "category": "laptops",
        "noOfPages": 10,
        "totalProducts": 4500,
        "elapsed": 3878
      }
    ]
  }
}
*/

events: 'response'

Emitted when successful HTTP response from the Flipkart Affiliate server.

Example

// 'response' event handler
scraper.on("response", (response) => {
  console.log(response);
});

events: 'data'

Emitted when products returned from Flipkart affiliate API.

Example

// 'data' event handler
scraper.on("data", (data) => {
  console.log(data.apiData.products);
});

events: 'categoryCompleted'

Emitted when all products scraped for the category.

Example

// 'categoryCompleted' event handler
scraper.on("categoryCompleted", (completedCategoryInfo) => {
  console.log(completedCategoryInfo);
});

events: 'finished'

Emitted when scraper finished.

Example

// 'finished' event handler
scraper.on("finished", (info) => {
  console.log(info);
});

events: 'error'

Emitted if any errors occured.

Example

// Triggered if any error occured
scraper.on("error", (error) => {
  console.error(error);
});

events: 'retry'

Emitted if any retry occured.

Example

// Triggered if any retry occured
scraper.on("retry", (retryInfo) => {
  console.log(retryInfo);
});

events: 'retryHalted'

Emitted if any retries failed for 10 times.

Example

// Triggerd when retry failed 10 times
scraper.on("retryHalted", (retryHaltInfo) => {
  console.error(retryHaltInfo);
});