blinkdb-js/blinkdb

🗃️ An in-memory JS database optimized for large scale storage on the frontend.

blinkdb's Issues

Read performance

I'm looking to switch to blinkdb for its typescript support, but when I benchmark indexed search, blinkdb comes out substantially slower than lokijs:

var Benchmark = require('benchmark')
const Loki = require('lokijs')
const { createDB, createTable } = require( "blinkdb")
const { internalInsertMany } = require('blinkdb/dist/core/insertMany')
const { internalFirst } = require('blinkdb/dist/core/first')

const items = require('./mock.json')

const blinkdb = createDB()
const blinktable = createTable(blinkdb, 'citekeys')({
  primary: "id",
  indexes: ["firstmail"],
});
internalInsertMany(blinktable, items)

const DB = new Loki('better-bibtex', {})
const coll = DB.addCollection('citekeys', {
  indices: [ 'id', 'firstname' ],
  unique: [ 'id' ],
})
coll.insert(items)

var suite = new Benchmark.Suite;
suite
  .add('loki', function() { coll.findOne({ firstname: { $eq: `${Math.random()}` } }) })
  .add('blink', function() { internalFirst(blinktable, { where: { firstname: `${Math.random()}` } }) })
  .on('cycle', function(event) { console.log(String(event.target)) })
  .on('complete', function() { console.log('Fastest is ' + this.filter('fastest').map('name')); })
  .run();

(where mock.json has this content)

returns

loki x 1,466,141 ops/sec ±1.39% (84 runs sampled)
blink x 1,279 ops/sec ±1.49% (89 runs sampled)

and even when I turn off indexes in loki, it still comes out ahead:

loki x 18,342 ops/sec ±0.97% (95 runs sampled)
blink x 1,313 ops/sec ±1.29% (92 runs sampled)
Fastest is loki

I haven't tested more complex searches but search by single indexed field is something I'd be doing a fair bit of. Is this just not a good use-case for blinkdb?

Docs: watch is async

In the example code in the docs, the return value of watch is being used as if the function ran synchronously, where in reality it returns a Promise. Also, if watch is not awaited, inserting items cause race conditions and will not be properly handled by the callback passed to watch.

Wrong (how it is currently)

const watcher = watch(userTable, (users) => {
  console.log("All users: ", users);
});

watcher.stop();

Correct

const watcher = await watch(userTable, (users) => {
  console.log("All users: ", users);
});

watcher.stop();

benchmarks don't run

When I run npm run start in the benchmarks folder, I get

> start
> ts-node ./src/index.ts

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /Users/emile/github/blinkdb/packages/benchmarks/src/index.ts
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:160:9)
    at defaultGetFormat (node:internal/modules/esm/get_format:203:36)
    at defaultLoad (node:internal/modules/esm/load:143:22)
    at async nextLoad (node:internal/modules/esm/hooks:749:22)
    at async nextLoad (node:internal/modules/esm/hooks:749:22)
    at async Hooks.load (node:internal/modules/esm/hooks:382:20)
    at async MessagePort.handleMessage (node:internal/modules/esm/worker:199:18) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}
npm ERR! Lifecycle script `start` failed with error:
npm ERR! Error: command failed
npm ERR!   in workspace: @blinkdb/benchmarks
npm ERR!   at location: /Users/emile/github/blinkdb/packages/benchmarks

Is there a way to do sync queries?

The async methods all seems to consist of wrapping the result of what are essentially sync functions in a Promise.resolve. Is there a way to skip this? There are parts of my application where I cannot run async code.

Is there a way to filter for not-equal?

How to count the entries effectively?

I can't find any example how to get the total rows in collections effectively.

Currently, I am using

const total = await many(users).then(x => x.length)

But obviously it is not a good approach.

Compatibility with recent typescript changes

I have following issues when using blinkdb with bun.sh

• https://github.com/mindreframer/blinkdb-bun

$ bun run index.ts
SyntaxError: Indirectly exported binding name 'Ids' is not found.

The fix is to export Ids as from db/remove.ts as type, like here:

export { clear } from "./clear";
export { count } from "./count";
export * from "./createDB";
export * from "./createTable";
export * from "./errors";
export { first } from "./first";
export { insert } from "./insert";
export type { Create } from "./insert";
export { insertMany } from "./insertMany";
export { many } from "./many";
export { one } from "./one";
export { remove } from "./remove";
export type { Ids } from "./remove";
export { removeMany } from "./removeMany";
export { removeWhere } from "./removeWhere";
export { update } from "./update";
export type { Diff } from "./update";
export { updateMany } from "./updateMany";
export { updateWhere } from "./updateWhere";
export { upsert } from "./upsert";
export { upsertMany } from "./upsertMany";
export * from "./use";
export * from "./uuid";
export * from "./watch";

The other issue is with BTree, that I have posted here:

• qwertie/btree-typescript#36

Not sure, if it's would require code changes in Blinkdb to make it work with bun.sh.

Document `Cannot access "buffer.Buffer" in client code` workaround

While following up on the installation and usage guide, I bumped into this issue - basically, buffer is not available on client side and we need to manually install it. A simple yarn add buffer or npm i buffer and restarting the dev server should fix it for most (stack overflow thread).

Perhaps buffer should be added as a dependency of blinkdb?

removeMany stops mid-remove when non-existent keys are passed

I'm pretty sure I must be missing something dead obvious, but I have a script where I do two removeManys that should have identical effect (I think) that don't in practice. Sorry for the newbie questions, but I've been trying to figure out all day what I'm doing wrong, and I just don't see it.

const { createDB , createTable, BlinkKey, insertMany, removeMany, clear, many, key } = require( "blinkdb")

const blinkdb = createDB({ clone: true })
let table = createTable(blinkdb, "users")({
  primary: "itemID",
  indexes: ["itemKey", "libraryID", "citationKey"]
})

const data = [
  {
    "citationKey": "shelahStableTheories1969",
    "itemID": 2,
    "itemKey": "6STQAJ5F",
    "libraryID": 1,
    "pinned": false
  },
  {
    "citationKey": "shelahNoteMinmaxProblem1969",
    "itemID": 4,
    "itemKey": "KG9BBSIH",
    "libraryID": 1,
    "pinned": false
  },
]

const deletes = [
  { "itemID": 2 },
  { "itemID": 5 },
  { "itemID": 4 },
]

async function main() {
  await insertMany(table, data)

  // this works partially
  await removeMany(table, deletes)
  console.log('remaining:', (await many(table)).length)
  console.log('ghosts:', (await many(table, { where: { itemID: { in: deletes.map(d => d.itemID) } } })).length)

  await clear(table)
  await insertMany(table, data)
  // this does work
  deletes.sort((a, b) => a.itemID - b.itemID)
  await removeMany(table, deletes)
  console.log('remaining:', (await many(table)).length)
  console.log('ghosts:', (await many(table, { where: { itemID: { in: deletes.map(d => d.itemID) } } })).length)
}

main().catch(err => console.log(err))

Error with hyperid

code to reproduce error

Code to Reproduce Error

in js vite project

import { createDB } from "blinkdb";
const db = createDB();
const wordsTable = createTable(db, "words")();

Error

hyperid.js:7  Uncaught TypeError: Cannot read properties of undefined (reading 'from')
    at node_modules/hyperid/hyperid.js (hyperid.js:7:30)
    at __require (chunk-NIBQISYW.js?v=a227823d:9:50)
    at node_modules/blinkdb/dist/core/uuid.js (uuid.js:7:35)
    at __require (chunk-NIBQISYW.js?v=a227823d:9:50)
    at node_modules/blinkdb/dist/core/index.js (index.js:54:14)
    at __require (chunk-NIBQISYW.js?v=a227823d:9:50)
    at node_modules/blinkdb/dist/index.js (index.js:17:14)
    at __require (chunk-NIBQISYW.js?v=a227823d:9:50)
    at index.js:20:47
    
 

Export Query types

I'm trying to build some caching layer using blinkdb on top of some file-based data structures and so far it's doing good. The only issue is that Query types are not exported for some reason.

value.some is not a function

I'm getting an odd issue, where if I pass an object for the contain query {contain: 'bob'} it throws an error. If I make it an array as the error seems to suggest [ {contains: 'bob'}] for example, I don't get an error, but I don't get results.

I'm following the documentation formatting:

I code in imba which compiles to vanilla js. Not sure if this has anything to do with it not being typescript.

Is there a facility to generate auto-increment unique IDs instead of guids?

When using `clone: false` watchers are not triggered with correct entities

Hello,

I have a watcher on a table and the db is initialized with clone: false. If I, for example, upsert an entity, the callback of the watcher is called without the changed entity. It works if clone is set to true (which is the default).

Property does not exist on type 'Where | Or | And'

When I have this table:

type CitekeyRecord = {
  itemID: number
  libraryID: number
  itemKey: string
  citationKey: string
  pinned: boolean | 0 | 1
}

const keys keys = createTable<CitekeyRecord>(createDB({ clone: true }), 'citationKeys')({
  primary: 'itemID',
  indexes: ['itemKey', 'libraryID', 'citationKey'],
})

const where: Query<CitekeyRecord, 'itemID'> = {
        where: {
          pinned: { in: [0, false] },
          citationKey: { eq: key.citationKey },
          libraryID: { eq: key.libraryID }
        },
      }
if (Preference.keyScope === 'global') delete where.where.libraryID

I get the error:

Property 'libraryID' does not exist on type 'Where | Or | And'.
Property 'libraryID' does not exist on type 'Or'.

I get the same error when I do

const where: Query<CitekeyRecord, 'itemID'> = {
        where: {
          pinned: { in: [0, false] },
          citationKey: { eq: key.citationKey },
        },
      }
if (Preference.keyScope === 'global') where.where.libraryID = { eq: key.libraryID }

How can I declare the query variable?

Links on the sidebar of CRUD & Filters Docs go to `localhost` not the `#` hash

Docs: how to save and load the database?

What is the recommended way to save and load a database with blinkdb?

Cannot use template parameter for an entity type

I tested with the recent version and unfortunately got a typing error.

The following example fails with an error stating that MyEntity cannot be assigned to Entity<T>

import { createDB, createTable } from 'blinkdb';

const database = createDB({
  clone: false
});

function abc<T extends MyEntity>(): void {
  createTable<T>(database, 'test')();
}

type MyEntity = {
  id: string;
}

abc();

I don't know how to resolve this..

insertMany very slow

The insertMany is currently very slow in benchmarks. That doesn't seem very realistic.

blinkdb/insert-many.ts --- lokijs is 417.44x faster than blinkdb
┌─────────┬───────────┬────────────────────┬────────────────────┬──────────┬─────────┐
│ (index) │   name    │      ops/sec       │ Average Time (ns)  │  Margin  │ Samples │
├─────────┼───────────┼────────────────────┼────────────────────┼──────────┼─────────┤
│    0    │ 'lokijs'  │ 203398.47530091717 │ 4916.457699697863  │ '±1.64%' │ 101700  │
│    1    │ 'blinkdb' │ 487.25611966443955 │ 2052308.7543542264 │ '±4.94%' │   244   │
└─────────┴───────────┴────────────────────┴────────────────────┴──────────┴─────────┘


blinkdb/upsert-many.ts --- lokijs is 3.87x faster than blinkdb
┌─────────┬───────────┬────────────────────┬────────────────────┬──────────┬─────────┐
│ (index) │   name    │      ops/sec       │ Average Time (ns)  │  Margin  │ Samples │
├─────────┼───────────┼────────────────────┼────────────────────┼──────────┼─────────┤
│    0    │ 'lokijs'  │ 1633.0642639853138 │ 612345.7735579922  │ '±4.32%' │   817   │
│    1    │ 'blinkdb' │ 421.87318580370237 │ 2370380.5637585605 │ '±7.20%' │   211   │
└─────────┴───────────┴────────────────────┴────────────────────┴──────────┴─────────┘


blinkdb/update.ts --- lokijs is 3.34x faster than blinkdb
┌─────────┬───────────┬────────────────────┬────────────────────┬───────────┬─────────┐
│ (index) │   name    │      ops/sec       │ Average Time (ns)  │  Margin   │ Samples │
├─────────┼───────────┼────────────────────┼────────────────────┼───────────┼─────────┤
│    0    │ 'lokijs'  │ 268725.01220459485 │ 3721.2762287964697 │ '±4.79%'  │ 134364  │
│    1    │ 'blinkdb' │ 80449.77230128765  │ 12430.115976648878 │ '±20.82%' │  40354  │
└─────────┴───────────┴────────────────────┴────────────────────┴───────────┴─────────┘

Data not updating when changing query with useMany

Hello and thank you for this wonderful library! It's super easy to use and very performant.

My question is: is there a way to update the query in the useMany hook?

It seems it will only re-query the data when something is changed in the DB, as it's using watch.

My use case is having a checkbox to toggle the display of deleted items, e.g.:

export const useItems = (includeDeleted = true) => {
  const itemsTable = useTable((model: Model) => model.items)

  const query = includeDeleted ? {} : { where: { deleted: false } }

  const { data, error, state } = useMany(itemsTable, query)

  return data
}

But the data is always the same as the first returned values.

Is this expected? Or am I doing something wrong?

Thank you very much in advance! 🙇

Would it be possible to include substring filters?

The reason i'm trying blinikDB is to have fast queryable data in memory.
I was sad to see no substring search filters.

I'd love to see something like the following

where: {
	prefix: "hell"
}
# returns hello

or

where: {
	substring: "no"
}

# returns: note, gnome, monosodium, etc.

Dexie.js has a function for prefix filtering, but no substring filter.

• https://dexie.org/docs/WhereClause/WhereClause.startsWith()
• https://dexie.org/docs/WhereClause/WhereClause.startsWithIgnoreCase()

Cannot use classes like `ItemNotFoundError` because target is set to `es5`

I cannot check an error with instanceof ItemNotFoundError because in the compiled file classes get converted into functions. I think it is because you target es5 in the tsconfig.json.

I don't know if it is desired/necessary to target es5 for some reason, otherwise would it be possible to change it to es6?

Thank you for this great project! 🙂

cross session, browser-refresh persistence with indexedDB?

It would be really useful if BlinkDB, could have browser-refresh persistence built-in.
I believe Dexie.js does this with Indexed DB.
I managed to add persistence my BlinkDB app using IndexedDB, but it took a bit of work, since I'm not a super experienced js dev.