hyperbeedeebee

A MongoDB-like database built on top of Hyperbee with support for indexing

WIP

Based on this design

Usage

npm i --save hyperbeedeebee

const Hyperbee = require('hyperbee')
// This module handles networking and storage of hypercores for you
const SDK = require('hyper-sdk')
const {DB} = require('hyperbeedeebee')

const {Hypercore} = await SDK()

// Initialize a hypercore for loading data
const core = new Hypercore('example')
// Initialize the Hyperbee you want to use for storing data and indexes
const bee = new Hyperbee(core)

// Create a new DB
const db = new DB(bee)

// Open up a collection of documents and insert a new document
const doc = await db.collection('example').insert({
  hello: 'Wrold!'
})

// doc._id gets set to an ObjectId if you don't specify it
console.log(doc)

// Iterate through data as it's loaded (streaming)
// Usually faster and more memory / CPU efficient
for await (let doc of db.collection('example').find({
  clout: {
$gt: 9000
  },
})) {
  console.log(doc)
}

// Get all results in an array
// Can skip some results and limit total for pagination
const killbots = await db.collection('example')
  .find({type: 'killbot'})
  .skip(30)
  .limit(100)

// Get a single document that matches the query
const eggbert = await db.collection('example').findOne({name: 'Eggbert'})

Data Types

HyperbeeDeeBee uses MongoDB's BSON data types for encoding data. You can import the bson library bundled with HyperbeeDeeBee using the following code:

const { BSON } = require('hyperbeedeebee')

From there you can access any of the following data types:

Binary,
Code,
DBRef,
Decimal128,
Double,
Int32,
Long,
UUID,
Map,
MaxKey,
MinKey,
ObjectId,
BSONRegExp,
BSONSymbol,
Timestamp

TODO:

• Sketch up API
• Insert (with BSON encoding)
• Find all docs
• Find by _id
• Find by field eq (no index)
• Find by array field includes
• Find by number field $gt/$gte/$lt/$lte
  • Numbers
  • Dates
• Find using $in operator
• Find using $all operator
• Index fields (must specify BSON type, rebuild?)
• Flatten array for indexes
• Sort by index (with find)
• Indexed find by field (only allow find for indexed fields)
• Indexed find by number field (only allow find for indexed fields)
• Get field values from index key without getting the doc
• Choose best index (hint API?)
• Find on fields that aren't indexed
• Test if iterators clean up properly
• Detect when data isn't available from peers and emit an error of some sort instead of waiting indefinately.

Important Differences From MongoDB

• There is a single writer for a hyperbee and multiple readers
• The indexing means that readers only need to download small subsets of the full dataset (if you index intelligently)
• No way to do "projections" so keep in mind you're always downloading the full document to disk
• Subset of find() API is implemented, no Map Reduce API, no $or/$and since it's difficult to optimize
• Fully open source under AGPL-3.0 and with mostly MIT dependencies.