Package Exports

@shelf/dynamodb-parallel-scan

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (@shelf/dynamodb-parallel-scan) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

dynamodb-parallel-scan

Scan DynamoDB table concurrently (up to 1,000,000 segments), recursively read all items from every segment

Install

$ yarn add @shelf/dynamodb-parallel-scan

Usage

Fetch everything at once

const {parallelScan} = require('@shelf/dynamodb-parallel-scan');

(async () => {
  const items = await parallelScan(
    {
      TableName: 'files',
      FilterExpression: 'attribute_exists(#fileSize)',
      ExpressionAttributeNames: {
        '#fileSize': 'fileSize',
      },
      ProjectionExpression: 'fileSize',
    },
    {concurrency: 1000}
  );

  console.log(items);
})();

Use as async generator (or streams)

Note: this stream doesn't implement backpressure mechanism just yet, so memory overflow could happen if you don't consume stream fast enough.

const {parallelScanAsStream} = require('@shelf/dynamodb-parallel-scan');

(async () => {
  const stream = await parallelScanAsStream(
    {
      TableName: 'files',
      FilterExpression: 'attribute_exists(#fileSize)',
      ExpressionAttributeNames: {
        '#fileSize': 'fileSize',
      },
      ProjectionExpression: 'fileSize',
    },
    {concurrency: 1000, chunkSize: 10000}
  );

  for await (const items of stream) {
    console.log(items); // 10k items here
  }
})();

Read

Publish

$ git checkout master
$ yarn version
$ yarn publish
$ git push origin master

@shelf/dynamodb-parallel-scan