JSPM

  • ESM via JSPM
  • ES Module Entrypoint
  • Export Map
  • Keywords
  • License
  • Repository URL
  • TypeScript Types
  • README
  • Created
  • Published
  • Downloads 51
  • Score
    100M100P100Q74143F
  • License BSD-3-Clause

Helpers for AWS DynamoDB to build efficient web applications

Package Exports

  • dynamodb-toolkit

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (dynamodb-toolkit) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

dynamodb-toolkit NPM version

No-dependencies opinionated micro-library for AWS DynamoDB to build small efficient RESTful APIs and high-performance command-line utilities with a simple intuitive API.

  • Designed to work with existing code. Your legacy code can be gradually improved. No special requirements for database tables or indices!
  • Battle-proven. Used in mission-critical applications.
  • Provides a flexible way preparing your data objects for storing and reviving them back:
    • Supports complex indexing: design your own queries!
    • Validate objects before storing.
    • Check the consistency of a database before storing objects.
    • Rich set of efficient read/write/delete/clone/move operations.
    • Various low-level and high-level utilities for DynamoDB.
  • Automatically encoding/decoding your beautiful JSON data to and fro DynamoDB internal format.
  • Supports vagaries of mass read/write/delete operations correctly handling throughput-related errors using the recommended exponential back-off algorithm.
    • Supports transactions and batch requests.
  • Implements efficiently on the server:
    • Paging (in offset/limit terms) and sorting on an index (ascending and descending) to interface with list/table visual components.
    • Subsetting AKA projection: read operations can return a subset required of fields instead of the whole shebang which can be huge especially for mass read operations — think lists and tables.
    • Searching AKA filtering: filters results looking for a substring in a predefined set of fields.
    • Patching: only necessary fields are transferred to be updated/deleted.
    • Cloning: making updated copies in a database.
    • Moving: effectively renaming objects.
  • Thoroughly asynchronous. No callbacks.
  • Working with multiple databases at the same time potentially using different credentials.

Extensively documented: see the wiki.

Example

const Adapter = require('dynamodb-toolkit');

const AWS = require('aws-sdk');
AWS.config.update({region: 'us-east-2'});

const adapter = new Adapter({
  client: new AWS.DynamoDB(),
  table: 'test',
  keyFields: ['name']
});

// ...

const planet = await adapter.get({name: 'Alderaan'});
await adapter.patch({name: planet.name, diameter: planet.diameter * 2});
// ...
await adapter.clone({name: planet.name}, planet => {
  const newPlanet = {...planet, name: planet.name + ' Two'};
  return newPlanet;
});
await adapter.delete({name: planet.name});

Adapter

The full documentation: Adapter. Here is a cheatsheet:

What you have to define yourself

  • Data properties:
    • keyFields — a required list of keys starting with the partition key.
  • Methods:

What you may want to define yourself

What you immediately get

Stand-alone utilities

Included utility functions:

  • Helping to create clients with all necessary settings.
  • Various operations on parameters (params) including adding projections, filtering.
  • Batching operations for efficiency.
  • Transactions.
  • Preparing patches.
  • Mass operations: reading by keys, list with pagination, copying, deleting, iterating over, getting totals, writing.
  • See the wiki for more details.

Koa

The library provides a helper for Koa to write HTTP REST servers: KoaAdapter. It takes care of query parameters, extracts POST/PUT JSON bodies, sends responses encoded as JSON with proper HTTP status codes, and prepares parameters for mass operations.

Example

Define a Koa adapter:

const koaAdapter = new KoaAdapter(adapter, {
  sortableIndices: {name: '-t-name-index'},

  augmentItemFromContext(item, ctx) {
    if (ctx.params.planet) {
      item.name = ctx.params.planet;
    }
    return item;
  },

  augmentCloneFromContext(ctx) {
    // how to transform an object when cloning/moving (the default)
    return item => ({...item, name: item.name + ' COPY'});
  },

  async getAll(ctx) {
    // custom getAll(), which supports sorting
    let index, descending;
    if (ctx.query.sort) {
      let sortName = ctx.query.sort;
      descending = ctx.query.sort[0] == '-';
      if (descending) {
        sortName = sortName.substr(1);
      }
      index = this.sortableIndices[sortName];
    }
    const options = this.makeOptions(ctx);
    options.descending = !!descending;
    ctx.body = await this.adapter.getAll(options, null, index);
  },

  async load(ctx) {
    // custom operation
    const data = JSON.parse(zlib.gunzipSync(fs.readFileSync(path.join(__dirname, 'data.json.gz'))));
    await this.adapter.putAll(data);
    ctx.body = {processed: data.length};
  }
});

Most operations were trivial. Some operations take more than a couple of lines for flexibility's sake.

Define the routing table:

const router = new Router();

router
  // mass operations
  .get('/', async ctx => koaAdapter.getAll(ctx))
  .delete('/', async ctx => koaAdapter.deleteAll(ctx))
  .put('/-load', async ctx => koaAdapter.load(ctx))
  .put('/-clone', async ctx => koaAdapter.cloneAll(ctx))
  .put('/-clone-by-names', async ctx => koaAdapter.cloneByNames(ctx))
  .put('/-move', async ctx => koaAdapter.moveAll(ctx))
  .put('/-move-by-names', async ctx => koaAdapter.moveByNames(ctx))
  .get('/-by-names', async ctx => koaAdapter.getByNames(ctx))
  .delete('/-by-names', async ctx => koaAdapter.deleteByNames(ctx))

  // item operations
  .post('/', async ctx => koaAdapter.post(ctx))
  .get('/:planet', async ctx => koaAdapter.get(ctx))
  .put('/:planet', async ctx => koaAdapter.put(ctx))
  .patch('/:planet', async ctx => koaAdapter.patch(ctx))
  .delete('/:planet', async ctx => koaAdapter.delete(ctx))
  .put('/:planet/-clone', async ctx => koaAdapter.clone(ctx))
  .put('/:planet/-move', async ctx => koaAdapter.move(ctx));

module.exports = router;

It cannot be simpler than that!

Documentation

See wiki for the full documentation.

Versions

  • 2.1.0 Added moveXXX() operations, some minor implementation improvements.
  • 2.0.1 Bug fix in the default implementation of revive().
  • 2.0.0 Minor API change, better support for paths, support for AWS.DynamoDB.DocumentClient, and so on.
  • 1.16.0 Switched conversion to AWS.DynamoDB.Convert, added getPath() and setPath() utilities.
  • 1.15.1 Added seq() for sequential asynchronous operations.
  • 1.15.0 Updated API to work with "db-raw" objects from a database.
  • 1.14.0 Updated API to work with "raw" objects from a database.
  • 1.13.9 Added an overridable clone function.
  • 1.13.8 Serialized generic mass operations. Slower but less resource consumption.
  • 1.13.7 Better work with boolean results in mass operations.
  • 1.13.6 Bugfix to return a number of processed records from generic mass operations.
  • 1.13.5 Bugfix to unbreak generic ops.
  • 1.13.4 Bugfix in generic operations.
  • 1.13.3 Added addKeyFields(), simplified getAllByParams(), fixed genericDeleteAllByParams().
  • 1.13.2 Bugfix in pagination with filtering.
  • 1.13.1 Added an alternative way to go over database records: iterateList().
  • 1.13.0 Refactored xxxList() functions to use getBatch() and applyBatch().
  • 1.12.0 Updated getTotal() API and added a no-limit version of paginateList().
  • 1.11.2 Typo fix in Adapter.makeCheck().
  • 1.11.1 Bugfix in Adapter related to transactions.
  • 1.11.0 New style of batches, support for transactional consistency checks.
  • 1.10.6 Bugfix in applyTransaction().
  • 1.10.5 Added generic implementations of compound operations.
  • 1.10.4 Added checkExistence() helper.
  • 1.10.3 Minor improvements, added batch makers working on raw keys and items.
  • 1.10.2 Restructured names to keys methods of KoaAdapter.
  • 1.10.1 Added xxxByNames methods to KoaAdapter.
  • 1.10.0 Added support for multi-table batches and transactions. Minor overall improvements.
  • 1.9.2 Added profile-related utilities.
  • 1.9.1 Bugfix: added missing normalization for scanAllByParams(), updated dev deps.
  • 1.9.0 Added: projecting sub-objects, new way to clone objects, and unit tests.
  • 1.8.2 Bugfix in scanAllByParams().
  • 1.8.1 Added makeListParams() and scanAllByParams().
  • 1.8.0 New signature for readList(), added streamable reading.
  • 1.7.2 Technical release.
  • 1.7.1 Minor performance tweaks, bugfixes in the test case.
  • 1.7.0 Added deleteByNames() and cloneByNames(), renamed getAllByKeys() to getByKeys().
  • 1.6.3 Added updateParams() to add writing conditions.
  • 1.6.2 Bugfix in KoaAdapter's mass methods: augmenting instead of copying.
  • 1.6.1 Return of clone() and cloneAll() as default methods.
  • 1.6.0 Added doClone().
  • 1.5.0 Refactored, simplified, added more canned list-based operations.
  • 1.4.1 Added ability to return undefined for mapping functions.
  • 1.4.0 Added helper methods: toDynamo(), toDynamoKey(), fromDynamo().
  • 1.3.0 clone() added to KoaAdapter. A related bug was fixed.
  • 1.2.0 Major refactoring of internals. Some API changes.
  • 1.1.1 Bugfix: added the missing index file.
  • 1.1.0 Made a search prefix a parameter.
  • 1.0.0 The initial public release

License

The 3-Clause BSD License

Acknowledgements

The test JSON file containing planets of the Star Wars universe is extracted from SWAPI and used under BSD license.