Package Exports

mongodb-data-sync

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (mongodb-data-sync) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

mongodb-data-sync

Duplicate data in multiple collections is a common thing in MongoDB. It is efficient for searching, sorting and even projects fields.

Handling duplicate data can be a pain. you will have to create jobs to sync the data, or do updates in place what makes the reference collection need to know about all the collections needed data from him . and we all know the bugs that can lead to.

mongodb-data-sync comes to solve this problem by letting you declare the dependencies in a logical place in your applications (for instance where you declare your schemas ) and sync the data in almost real-time.

mongodb-data-sync was designed to do all the updates and synchronization with minimum overhead on the database and do most of the checks in memory.

Notice

mongodb-data-sync is still experimental and hasn't been tested on production yet

Pros and cons of having duplicate data in multiple collection

pros

1. you won't have to do joins. 2. you can do index on all the fields. 3. faster and easier searching and sorting

cons

1. having duplicate data means more storage usage. 2. hard do maintenance and keep in track all the connections(this is what mongodb-data-sync comes to solve). 3. add write operations, every update will have to update multiple collections

requirements

MongoDB v3.6 or higher replaica set
nodejs 7.6 or higher

Architecture

mongodb-data-sync built from 2 seprate parts.

The server application(there can only be one)- this what runs all the updates logic,don't use more than 1 application, it was designed to work as a single process and knows from where to continue after restart, crash so don't try to do auto-scaling or set 2 containers for high availability
The SDK - responsible for managing the database dependencies of your application,

Instructions

The Instructions will address the 2 parts separately,

the server that runs the logic.

the SDK that runs the communication between your app and the server

Server

Run

npm install mongodb-data-sync -g

Then in the cmd run

mongodb-data-sync --key "some key" --url "mongodb connection url"

Options:

  -p, --port <port>       server port. 
  
  -d, --dbname <dbname>  the database name for the package. 
  
  -k, --key              API key to used for authentication of the SDK requests, required
  
  -u, --url              MongoDB connection url, required
  
  -h, --help             output usage information

that's it for running the server, let's jump to the SDK

SDK

You can look at the example on github

Install

npm install mongodb-data-sync -save

init

first initialize the client , do it as soon as possible in your app

const SynchronizerClient = require('mongodb-data-sync');

SynchronizerClient.init({
    dbName: String, // the DB name you want the synchronization to work on (required)
    serviceUrl: String, // the URL for the server you run on the previous stage (required),  
    apiKey: String, // this need to be the same key you declared in your server (required)
});

returns a Promise

getInstance

const synchronizerClientInstance = SynchronizerClient.getInstance({dbName: String}); // return an instance related to your db(its not a mongodb db instance) for dependncies oprations

addDependency

synchronizerClientInstance.addDependency({
   dependentCollection: String,// the dependent collection (required)
   refCollection: String, //the referenced collection (required)
   localField: String, // the dependent collection field to connect with (required)
   foreignField:String , // the referenced collection field to connect with, default _id ,using other field then _id will cuz an extra join for each check (optional)
   fieldsToSync: {}// the fields you want to update, the key is the field on the  dependentCollection and the value is for the refCollection
});

return Promise with the id of the Dependency

removeDependency

synchronizerClientInstance.removeDependency(id);

return Promise