heineiuo/rippledb

Embeddable key-value database engine in pure TypeScript, based on LSM-Tree

Rippledb ·

Rippledb is an embeddable key-value database engine in pure TypeScript, based on LSM-Tree, Inspired by LevelDB.

• Pure TypeScript: Rippledb is totally written in TypeScript, and runs on different platforms after being compiled to JavaScript.
• Lightweight: Rippledb has only 7k+ source code, and smaller than 1MB after compiled.Rippledb use zero third party modules.
• Embeddable: Rippledb can be embedded in node.js application (or other JavaScript Runtime Environments) very easily.

Installation

Install with npm:

npm install rippledb

Install with Yarn:

yarn add rippledb

Documentation

You can find the React documentation on the website.

Check out the Get Started page for a quick overview.

Examples

import path from 'path'
import { Database } from 'rippledb'

async function main(){
  const db = new Database(path.resolve(__dirname, './db'))
  await db.put('foo', 'bar')
  console.log(
    new TextDecoder().decode(await db.get('foo'))
  ) // 'bar'
}

main()

Roadmap

• Release 1.0 (2020-7-7)
• Support Deno (2020-9-1)

Benchmark

environment : GitHub Action
key         : 16 bytes
value       : 100 bytes
total       : 10000
runners     : 10 
fillrandom  : 823.87 ms total; 82.39 us/op

Compatibility

node.js	Deno
>=v10.0.0	WIP

License

MIT License

rippledb's Issues

Maybe top level API should use Uint8Array instead of Buffer

Release 1.0

• Support transaction #94
• Support Database.destroy #114
• Support reverse iterator #118
• Support Database.repair #119

I prefer release at Jul 6, 2020, 1 year after totally rewritten.

Support Database.repair

Support Database.destroy

• set ok to shutdown error
• close logfile fd
• close tablecache fd
• close infolof fd
• release memory： memtable，
• compaction check db ok status

Refactor: Refactor Status

Status is just a wrapper for Promise

Current Status:

class Status {

async ok():Promise<boolean> {
    constructor(promise:Promise) {
this.promise  = promise
}

    try {
     await this.promise
     return true
    catch(e){
     return false
   }
 }
}

const status = new Status(promise)
if (await status.ok()) {
   // do stuff
}

refactor:

async function method(){
  let error:void|Error
  try {
      await promise
   } catch(e){
      error = e
   }

   return e
}

if (!(await method())) {
    // do stuff
}

Refactor: Rename destroy API to close

Because in LevelBD, there is an API called DestroyDB which will physically delete files and folders. In current version of Rippledb, destroy method just do stuffs like close file descriptors and release memories.

Getting `ReferenceError: TextEncoder is not defined` when used in Node.js v10

Buffer.js makes reference to TExtEncoder, but never imports it:

rippledb/src/Buffer.ts

Line 95 in b885204

const ab = new TextEncoder().encode(str);

This causes an exception on Node.js

I was using the following script to try and convert a Level DB to a Ripple DB:

#!/usr/bin/env node

const fs = require('fs');
const path = require('path');
const level = require('level');
const { Database } = require('rippledb');

// https://github.com/heineiuo/rippledb/issues/148
global.TextEncoder = require("util").TextEncoder;

if(process.argv.length !== 4) {
  console.error("Usage: node level-convert.js <path_to_level_db> <path_to_ripple_db>");
  process.exit(1)
}

let levelDbPath = process.argv[2];
let rippleDbPath = process.argv[3];

try {
  const stats = fs.statSync(levelDbPath);
  if(!stats.isDirectory()) {
    console.error("No such leveldb database");
    process.exit(1);
  }
} catch(e) {
  console.error("No such leveldb database");
  process.exit(1);
}

try {
  const levelDb = level(levelDbPath, {valueEncoding: 'json'});
  const rippleDb = new Database(path.resolve(__dirname, rippleDbPath));
  const stream = levelDb.createReadStream();
  stream.on('data', function({ key, value }) {
    rippleDb.put(key, value);
    console.log(key, value);
  })
  stream.on('end', function() {
    process.exit(0);  
  })
} catch(e) {
  console.error("Could not open leveldb database");
  process.exit(1);
}

Migrate to TypeScript

SequenceNumber should deeply use BigInt

• _value should be bigint
• use get number() to get number
• compare based on bigint

ci: did not create tag for rippledb-deno

Support reverse iterator

TypeError: Cannot read property 'userKey' of undefined

    TypeError: Cannot read property 'userKey' of undefined

      1048 |     let level = 0
      1049 |     if ((await status.ok()) && meta.fileSize > 0) {
    > 1050 |       const minUserKey = meta.smallest.userKey
           |                                        ^
      1051 |       const maxUserKey = meta.largest.userKey
      1052 |       if (!!base) {
      1053 |         level = base.pickLevelForMemTableOutput(minUserKey, maxUserKey)

      at Database.userKey [as writeLevel0Table] (src/Database.ts:1050:40)
      at Database.compactMemTable (src/Database.ts:982:21)
      at Database.backgroundCompaction (src/Database.ts:727:7)
      at Database.backgroundCall (src/Database.ts:713:5)
      at Database.maybeScheduleCompaction (src/Database.ts:707:7)
      at Database.makeRoomForWrite (src/Database.ts:674:9)
      at Database.write (src/Database.ts:597:20)
      at Database.manualCompactMemTable (src/Database.ts:1232:5)
      at Database.compactRange (src/Database.ts:1166:5)

Recover failed with Error: AssertError: E

(node:1) UnhandledPromiseRejectionWarning: Error: AssertError: E
    at assert (<...>/rippledb/build/src/DBHelper.js:22:11)
    at Skiplist.put (<...>/rippledb/build/src/Skiplist.js:157:26)
    at MemTable.add (<...>/rippledb/build/src/MemTable.js:129:16)
    at Function.insert (<...>/rippledb/build/src/WriteBatch.js:37:11)
    at Database.recoverLogFile (<...>/rippledb/build/src/Database.js:342:38)
    at async Database.recover (<...>/rippledb/build/src/Database.js:295:23)
    at async Status.wait (<...>/rippledb/build/src/Status.js:68:7)
    at async Status.ok (<...>/rippledb/build/src/Status.js:78:5)
    at async Database.recoverWrapper (<...>/rippledb/build/src/Database.js:177:11)
    at async Status.wait (<...>/rippledb/build/src/Status.js:68:7)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 6)

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value: (0, _assert.default)(buf.length >= 4)

AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  (0, _assert.default)(buf.length >= 4)

    at decodeFixed32 (/node_modules/node-level/build/Coding.js:47:23)
    at SSTableBlock.restartPointIterator (/node_modules/node-level/build/SSTableBlock.js:82:39)

AssertionError [ERR_ASSERTION]: new key must bigger then last key

There maybe something incorrect in MemTable

Support transaction

Refrerences:

• https://stackoverflow.com/questions/54316452/transaction-modes-in-rocksdb

Add Lockfile

DB failed to run after DBRepairer

Compatible with Deno

https://deno.land
https://github.com/denoland/deno