seald/nedb

The JavaScript Database, for Node.js, nw.js, electron and the browser

seald

nedb's Issues

Maximum call stack size exceeded at BinarySearchTree.compareThings

On database ~500MB with 10K records in a single table, there are frequent errors:

/home/vlado/dev/pigallery/node_modules/.pnpm/@[email protected]/node_modules/@seald-io/nedb/lib/model.js:175
const compareThings = (a, b, _compareStrings) => {
                      ^

RangeError: Maximum call stack size exceeded
    at BinarySearchTree.compareThings [as compareKeys] (/home/vlado/dev/pigallery/node_modules/.pnpm/@[email protected]/node_modules/@seald-io/nedb/lib/model.js:175:23)
    at BinarySearchTree.delete (/home/vlado/dev/pigallery/node_modules/.pnpm/@[email protected]/node_modules/@seald-io/binary-search-tree/lib/bst.js:348:14)
    at BinarySearchTree.delete (/home/vlado/dev/pigallery/node_modules/.pnpm/@[email protected]/node_modules/@seald-io/binary-search-tree/lib/bst.js:354:40)
    at BinarySearchTree.delete (/home/vlado/dev/pigallery/node_modules/.pnpm/@[email protected]/node_modules/@seald-io/binary-search-tree/lib/bst.js:354:40)
    at BinarySearchTree.delete (/home/vlado/dev/pigallery/node_modules/.pnpm/@[email protected]/node_modules/@seald-io/binary-search-tree/lib/bst.js:354:40)

This most commonly happens during calls to delete function, but also happens during any number of other operation.

Example that causes error - record itself is perfectly valid and can be read without issues, just cannot be deleted
note this happens to only small number of records in database, but its fully reproducible with specific record in question

db.findOne({ image: param }, (err, res) => {
  if (res) db.remove({ image: data.image }, { multi: false }, (err, count) => {
    console.log(err, count);
  });
});

Workaround is to run node --stack-size=8192 and then there are no issues,
But this really should not be necessary for a DB of this size - I'm guessing recursive calls for large number of cells need some rethinking, perhaps using something like setTimeout(() => { next(); }); to give node a chance to clear a stack before diving into next recursion.

Environment: NodeJS 16.0.0 on Ubuntu 21.04, @seald-io/nedb 2.0.3

Crosslinking issue with jrop/nedb-promise#14

[FR] Add types to make TypeScript compile

https://github.com/seald/nedb
sandbox: https://codesandbox.io/s/declarator-demo-ivwep

I cannot get this to run on TypeScript without completely disabling it. More strangely, all the known workarounds seem to fail for this package.

and index.d.ts should fix this.

[ERROR] 21:46:18 ⨯ Unable to compile TypeScript:
src/models/users.model.ts:3:18 - error TS7016: Could not find a declaration file for module '@seald-io/nedb'. '/sandbox/node_modules/@seald-io/nedb/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/seald-io__nedb` if it exists or add anew declaration (.d.ts) file containing `declare module '@seald-io/nedb';`

3 import NeDB from '@seald-io/nedb';

NeDB types are here https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/nedb

Promise support

Is your feature request related to a problem? Please describe.
For complex requests implementation, like for data migration, I need to fetch data from many databases and use them to update others.

Using the current callback system is quite limiting, forcing me to chain the data fetch callbacks until the update one.

Describe the solution you'd like

Instead of callback, make this library embracing promises.

Using promises also allow us to use the async/await keywords to have a more imperative code structure that can be easier to read and understand on some cases. For example:

const user = await models.users.findOne(...);
await model.orders.update({ _id: 42 }, { customerId: user._id });

Describe alternatives you've considered
I don't see any alternative.

Additional context

This was a subject already evoked on the original project (louischatriot#673) but without any official answer.

Add dropDatabase method

We should add a dropDatabase static method to Datastore to delete the file(s) of a Datastore which would work for the browser, node and react-native.

Compound Indexes

Is your feature request related to a problem? Please describe.
Today it is only possible to create an index on one single field, but it is often required to index on multiple fields.

Describe the solution you'd like

I implemented compound indexes a while ago on my own nedb fork. (rmanibus#1)

this is how it works:

db.ensureIndex({ fieldName: ["field1", "field2"] }, function (err) {});

my branch would need to be rebased but I think this can easily be integrated in that project.

did you look at this commit for bigger databases?

it could be good for big databases
louischatriot#463

Race condition & database corruption may happen when calling `ensureIndex`

The datastore.js#ensureIndex and datastore.js#removeIndex functions are explicitly said to have an optional callback only for consistency in the API, but it appears it does asynchronous operations via persistence.js#persistNewState.

There are two problems with that:

1. the callback is necessary since it does an FS operation asynchronously, we should enforce it / at least document it properly and use it properly in the tests;
2. it does asynchronous operations on the database file outside the executor, which can sometimes corrupt the database: it can lead to having lines that look like: {"_id":"ccc","z":"3","nested":{"today":{"$$date":1634636472464}}}{"$$indexCreated":{"fieldName":"z"}} which are most probably the result of a persistCachedDatabase running concurrently with the persistNewState. In memory, it poses no problem. However, on disk, if it happens it will corrupt the line.

Short-term, I recommend we:

1. change the documentation regarding how to use the ensureIndex / removeIndex functions to explicitly say the callback should be called;
2. change all the tests that use those functions to ensure they call they use the callback;
3. add either these functions in the executor loop to ensure there are no race conditions.

Long-term, I recommend we:

1. remove [email protected] which was released in 2013;
2. either move to async-es or use a simple Promise-based waterfall pattern and shim the callback API.

[feat] binary data

Is your feature request related to a problem? Please describe.
I want to store binary files as a database field without resorting to creating and managing real files on a filesystem.

Describe the solution you'd like
Moving to a binary-encoded json-like format or adding a way to reference a separate database with the actual binary data.

Describe alternatives you've considered
Creating a package to serve as a store for binary files

Github Actions CI publish always tags as beta

${{ github.event.release.tag_name }} does not seem to work

Add notice about nedb-promises

This, to be exact https://www.npmjs.com/package/nedb-promises

loadDatabase with callback option cannot be used from Typescript

I'm trying to use nedb from typescript.
But I get an error.

nedb version
v2.2.1

sample code

import Nedb from 'nedb';
const DB = new Nedb({ filename: 'path/to/datafile' })
DB.loadDatabase(function (err) { 
    // callback
})

Symptoms
typescript error occurs

Error details
Expected 0 arguments, but got 1.ts(2554)

Compatibility with React-native

There is this repo which does the job, but it'd be more convenient if it were natively included into nedb: https://github.com/antoniopresto/react-native-local-mongodb/