Giter Site home page Giter Site logo

abstract-leveldown's Introduction

level

Universal abstract-level database for Node.js and browsers. This is a convenience package that exports classic-level in Node.js and browser-level in browsers, making it an ideal entry point to start creating lexicographically sorted key-value databases.

📌 Which module should I use? What is abstract-level? Head over to the FAQ.

level badge npm Node version Test Coverage Standard Common Changelog Community Donate

Table of Contents

Click to expand

Usage

If you are upgrading: please see UPGRADING.md.

const { Level } = require('level')

// Create a database
const db = new Level('example', { valueEncoding: 'json' })

// Add an entry with key 'a' and value 1
await db.put('a', 1)

// Add multiple entries
await db.batch([{ type: 'put', key: 'b', value: 2 }])

// Get value of key 'a': 1
const value = await db.get('a')

// Iterate entries with keys that are greater than 'a'
for await (const [key, value] of db.iterator({ gt: 'a' })) {
  console.log(value) // 2
}

All asynchronous methods also support callbacks.

Callback example
db.put('a', { x: 123 }, function (err) {
  if (err) throw err

  db.get('a', function (err, value) {
    console.log(value) // { x: 123 }
  })
})

TypeScript type declarations are included and cover the methods that are common between classic-level and browser-level. Usage from TypeScript requires generic type parameters.

TypeScript example
// Specify types of keys and values (any, in the case of json).
// The generic type parameters default to Level<string, string>.
const db = new Level<string, any>('./db', { valueEncoding: 'json' })

// All relevant methods then use those types
await db.put('a', { x: 123 })

// Specify different types when overriding encoding per operation
await db.get<string, string>('a', { valueEncoding: 'utf8' })

// Though in some cases TypeScript can infer them
await db.get('a', { valueEncoding: db.valueEncoding('utf8') })

// It works the same for sublevels
const abc = db.sublevel('abc')
const xyz = db.sublevel<string, any>('xyz', { valueEncoding: 'json' })

Install

With npm do:

npm install level

For use in browsers, this package is best used with browserify, webpack, rollup or similar bundlers. For a quick start, visit browserify-starter or webpack-starter.

Supported Platforms

At the time of writing, level works in Node.js 12+ and Electron 5+ on Linux, Mac OS, Windows and FreeBSD, including any future Node.js and Electron release thanks to Node-API, including ARM platforms like Raspberry Pi and Android, as well as in Chrome, Firefox, Edge, Safari, iOS Safari and Chrome for Android. For details, see Supported Platforms of classic-level and Browser Support of browser-level.

Binary keys and values are supported across the board.

API

The API of level follows that of abstract-level. The documentation below covers it all except for Encodings, Events and Errors which are exclusively documented in abstract-level. For options and additional methods specific to classic-level and browser-level, please see their respective READMEs.

An abstract-level and thus level database is at its core a key-value database. A key-value pair is referred to as an entry here and typically returned as an array, comparable to Object.entries().

db = new Level(location[, options])

Create a new database or open an existing database. The location argument must be a directory path (relative or absolute) where LevelDB will store its files, or in browsers, the name of the IDBDatabase to be opened.

The optional options object may contain:

  • keyEncoding (string or object, default 'utf8'): encoding to use for keys
  • valueEncoding (string or object, default 'utf8'): encoding to use for values.

See Encodings for a full description of these options. Other options (except passive) are forwarded to db.open() which is automatically called in a next tick after the constructor returns. Any read & write operations are queued internally until the database has finished opening. If opening fails, those queued operations will yield errors.

db.status

Read-only getter that returns a string reflecting the current state of the database:

  • 'opening' - waiting for the database to be opened
  • 'open' - successfully opened the database
  • 'closing' - waiting for the database to be closed
  • 'closed' - successfully closed the database.

db.open([callback])

Open the database. The callback function will be called with no arguments when successfully opened, or with a single error argument if opening failed. If no callback is provided, a promise is returned. Options passed to open() take precedence over options passed to the database constructor. The createIfMissing and errorIfExists options are not supported by browser-level.

The optional options object may contain:

  • createIfMissing (boolean, default: true): If true, create an empty database if one doesn't already exist. If false and the database doesn't exist, opening will fail.
  • errorIfExists (boolean, default: false): If true and the database already exists, opening will fail.
  • passive (boolean, default: false): Wait for, but do not initiate, opening of the database.

It's generally not necessary to call open() because it's automatically called by the database constructor. It may however be useful to capture an error from failure to open, that would otherwise not surface until another method like db.get() is called. It's also possible to reopen the database after it has been closed with close(). Once open() has then been called, any read & write operations will again be queued internally until opening has finished.

The open() and close() methods are idempotent. If the database is already open, the callback will be called in a next tick. If opening is already in progress, the callback will be called when that has finished. If closing is in progress, the database will be reopened once closing has finished. Likewise, if close() is called after open(), the database will be closed once opening has finished and the prior open() call will receive an error.

db.close([callback])

Close the database. The callback function will be called with no arguments if closing succeeded or with a single error argument if closing failed. If no callback is provided, a promise is returned.

A database may have associated resources like file handles and locks. When the database is no longer needed (for the remainder of a program) it's recommended to call db.close() to free up resources.

After db.close() has been called, no further read & write operations are allowed unless and until db.open() is called again. For example, db.get(key) will yield an error with code LEVEL_DATABASE_NOT_OPEN. Any unclosed iterators or chained batches will be closed by db.close() and can then no longer be used even when db.open() is called again.

db.supports

A manifest describing the features supported by this database. Might be used like so:

if (!db.supports.permanence) {
  throw new Error('Persistent storage is required')
}

db.get(key[, options][, callback])

Get a value from the database by key. The optional options object may contain:

  • keyEncoding: custom key encoding for this operation, used to encode the key.
  • valueEncoding: custom value encoding for this operation, used to decode the value.

The callback function will be called with an error if the operation failed. If the key was not found, the error will have code LEVEL_NOT_FOUND. If successful the first argument will be null and the second argument will be the value. If no callback is provided, a promise is returned.

db.getMany(keys[, options][, callback])

Get multiple values from the database by an array of keys. The optional options object may contain:

  • keyEncoding: custom key encoding for this operation, used to encode the keys.
  • valueEncoding: custom value encoding for this operation, used to decode values.

The callback function will be called with an error if the operation failed. If successful the first argument will be null and the second argument will be an array of values with the same order as keys. If a key was not found, the relevant value will be undefined. If no callback is provided, a promise is returned.

db.put(key, value[, options][, callback])

Add a new entry or overwrite an existing entry. The optional options object may contain:

  • keyEncoding: custom key encoding for this operation, used to encode the key.
  • valueEncoding: custom value encoding for this operation, used to encode the value.

The callback function will be called with no arguments if the operation was successful or with an error if it failed. If no callback is provided, a promise is returned.

db.del(key[, options][, callback])

Delete an entry by key. The optional options object may contain:

  • keyEncoding: custom key encoding for this operation, used to encode the key.

The callback function will be called with no arguments if the operation was successful or with an error if it failed. If no callback is provided, a promise is returned.

db.batch(operations[, options][, callback])

Perform multiple put and/or del operations in bulk. The operations argument must be an array containing a list of operations to be executed sequentially, although as a whole they are performed as an atomic operation.

Each operation must be an object with at least a type property set to either 'put' or 'del'. If the type is 'put', the operation must have key and value properties. It may optionally have keyEncoding and / or valueEncoding properties to encode keys or values with a custom encoding for just that operation. If the type is 'del', the operation must have a key property and may optionally have a keyEncoding property.

An operation of either type may also have a sublevel property, to prefix the key of the operation with the prefix of that sublevel. This allows atomically committing data to multiple sublevels. Keys and values will be encoded by the sublevel, to the same effect as a sublevel.batch(..) call. In the following example, the first value will be encoded with 'json' rather than the default encoding of db:

const people = db.sublevel('people', { valueEncoding: 'json' })
const nameIndex = db.sublevel('names')

await db.batch([{
  type: 'put',
  sublevel: people,
  key: '123',
  value: {
    name: 'Alice'
  }
}, {
  type: 'put',
  sublevel: nameIndex,
  key: 'Alice',
  value: '123'
}])

The optional options object may contain:

  • keyEncoding: custom key encoding for this batch, used to encode keys.
  • valueEncoding: custom value encoding for this batch, used to encode values.

Encoding properties on individual operations take precedence. In the following example, the first value will be encoded with the 'utf8' encoding and the second with 'json'.

await db.batch([
  { type: 'put', key: 'a', value: 'foo' },
  { type: 'put', key: 'b', value: 123, valueEncoding: 'json' }
], { valueEncoding: 'utf8' })

The callback function will be called with no arguments if the batch was successful or with an error if it failed. If no callback is provided, a promise is returned.

chainedBatch = db.batch()

Create a chained batch, when batch() is called with zero arguments. A chained batch can be used to build and eventually commit an atomic batch of operations. Depending on how it's used, it is possible to obtain greater performance with this form of batch(). On browser-level however, it is just sugar.

await db.batch()
  .del('bob')
  .put('alice', 361)
  .put('kim', 220)
  .write()

iterator = db.iterator([options])

Create an iterator. The optional options object may contain the following range options to control the range of entries to be iterated:

  • gt (greater than) or gte (greater than or equal): define the lower bound of the range to be iterated. Only entries where the key is greater than (or equal to) this option will be included in the range. When reverse is true the order will be reversed, but the entries iterated will be the same.
  • lt (less than) or lte (less than or equal): define the higher bound of the range to be iterated. Only entries where the key is less than (or equal to) this option will be included in the range. When reverse is true the order will be reversed, but the entries iterated will be the same.
  • reverse (boolean, default: false): iterate entries in reverse order. Beware that a reverse seek can be slower than a forward seek.
  • limit (number, default: Infinity): limit the number of entries yielded. This number represents a maximum number of entries and will not be reached if the end of the range is reached first. A value of Infinity or -1 means there is no limit. When reverse is true the entries with the highest keys will be returned instead of the lowest keys.

The gte and lte range options take precedence over gt and lt respectively. If no range options are provided, the iterator will visit all entries of the database, starting at the lowest key and ending at the highest key (unless reverse is true). In addition to range options, the options object may contain:

  • keys (boolean, default: true): whether to return the key of each entry. If set to false, the iterator will yield keys that are undefined. Prefer to use db.keys() instead.
  • values (boolean, default: true): whether to return the value of each entry. If set to false, the iterator will yield values that are undefined. Prefer to use db.values() instead.
  • keyEncoding: custom key encoding for this iterator, used to encode range options, to encode seek() targets and to decode keys.
  • valueEncoding: custom value encoding for this iterator, used to decode values.

📌 To instead consume data using streams, see level-read-stream and level-web-stream.

keyIterator = db.keys([options])

Create a key iterator, having the same interface as db.iterator() except that it yields keys instead of entries. If only keys are needed, using db.keys() may increase performance because values won't have to fetched, copied or decoded. Options are the same as for db.iterator() except that db.keys() does not take keys, values and valueEncoding options.

// Iterate lazily
for await (const key of db.keys({ gt: 'a' })) {
  console.log(key)
}

// Get all at once. Setting a limit is recommended.
const keys = await db.keys({ gt: 'a', limit: 10 }).all()

valueIterator = db.values([options])

Create a value iterator, having the same interface as db.iterator() except that it yields values instead of entries. If only values are needed, using db.values() may increase performance because keys won't have to fetched, copied or decoded. Options are the same as for db.iterator() except that db.values() does not take keys and values options. Note that it does take a keyEncoding option, relevant for the encoding of range options.

// Iterate lazily
for await (const value of db.values({ gt: 'a' })) {
  console.log(value)
}

// Get all at once. Setting a limit is recommended.
const values = await db.values({ gt: 'a', limit: 10 }).all()

db.clear([options][, callback])

Delete all entries or a range. Not guaranteed to be atomic. Accepts the following options (with the same rules as on iterators):

  • gt (greater than) or gte (greater than or equal): define the lower bound of the range to be deleted. Only entries where the key is greater than (or equal to) this option will be included in the range. When reverse is true the order will be reversed, but the entries deleted will be the same.
  • lt (less than) or lte (less than or equal): define the higher bound of the range to be deleted. Only entries where the key is less than (or equal to) this option will be included in the range. When reverse is true the order will be reversed, but the entries deleted will be the same.
  • reverse (boolean, default: false): delete entries in reverse order. Only effective in combination with limit, to delete the last N entries.
  • limit (number, default: Infinity): limit the number of entries to be deleted. This number represents a maximum number of entries and will not be reached if the end of the range is reached first. A value of Infinity or -1 means there is no limit. When reverse is true the entries with the highest keys will be deleted instead of the lowest keys.
  • keyEncoding: custom key encoding for this operation, used to encode range options.

The gte and lte range options take precedence over gt and lt respectively. If no options are provided, all entries will be deleted. The callback function will be called with no arguments if the operation was successful or with an error if it failed. If no callback is provided, a promise is returned.

sublevel = db.sublevel(name[, options])

Create a sublevel that has the same interface as db (except for additional methods specific to classic-level or browser-level) and prefixes the keys of operations before passing them on to db. The name argument is required and must be a string.

const example = db.sublevel('example')

await example.put('hello', 'world')
await db.put('a', '1')

// Prints ['hello', 'world']
for await (const [key, value] of example.iterator()) {
  console.log([key, value])
}

Sublevels effectively separate a database into sections. Think SQL tables, but evented, ranged and real-time! Each sublevel is an AbstractLevel instance with its own keyspace, events and encodings. For example, it's possible to have one sublevel with 'buffer' keys and another with 'utf8' keys. The same goes for values. Like so:

db.sublevel('one', { valueEncoding: 'json' })
db.sublevel('two', { keyEncoding: 'buffer' })

An own keyspace means that sublevel.iterator() only includes entries of that sublevel, sublevel.clear() will only delete entries of that sublevel, and so forth. Range options get prefixed too.

Fully qualified keys (as seen from the parent database) take the form of prefix + key where prefix is separator + name + separator. If name is empty, the effective prefix is two separators. Sublevels can be nested: if db is itself a sublevel then the effective prefix is a combined prefix, e.g. '!one!!two!'. Note that a parent database will see its own keys as well as keys of any nested sublevels:

// Prints ['!example!hello', 'world'] and ['a', '1']
for await (const [key, value] of db.iterator()) {
  console.log([key, value])
}

📌 The key structure is equal to that of subleveldown which offered sublevels before they were built-in to abstract-level. This means that an abstract-level sublevel can read sublevels previously created with (and populated by) subleveldown.

Internally, sublevels operate on keys that are either a string, Buffer or Uint8Array, depending on parent database and choice of encoding. Which is to say: binary keys are fully supported. The name must however always be a string and can only contain ASCII characters.

The optional options object may contain:

  • separator (string, default: '!'): Character for separating sublevel names from user keys and each other. Must sort before characters used in name. An error will be thrown if that's not the case.
  • keyEncoding (string or object, default 'utf8'): encoding to use for keys
  • valueEncoding (string or object, default 'utf8'): encoding to use for values.

The keyEncoding and valueEncoding options are forwarded to the AbstractLevel constructor and work the same, as if a new, separate database was created. They default to 'utf8' regardless of the encodings configured on db. Other options are forwarded too but abstract-level (and therefor level) has no relevant options at the time of writing. For example, setting the createIfMissing option will have no effect. Why is that?

Like regular databases, sublevels open themselves but they do not affect the state of the parent database. This means a sublevel can be individually closed and (re)opened. If the sublevel is created while the parent database is opening, it will wait for that to finish. If the parent database is closed, then opening the sublevel will fail and subsequent operations on the sublevel will yield errors with code LEVEL_DATABASE_NOT_OPEN.

chainedBatch

chainedBatch.put(key, value[, options])

Queue a put operation on this batch, not committed until write() is called. This will throw a LEVEL_INVALID_KEY or LEVEL_INVALID_VALUE error if key or value is invalid. The optional options object may contain:

  • keyEncoding: custom key encoding for this operation, used to encode the key.
  • valueEncoding: custom value encoding for this operation, used to encode the value.
  • sublevel (sublevel instance): act as though the put operation is performed on the given sublevel, to similar effect as sublevel.batch().put(key, value). This allows atomically committing data to multiple sublevels. The key will be prefixed with the prefix of the sublevel, and the key and value will be encoded by the sublevel (using the default encodings of the sublevel unless keyEncoding and / or valueEncoding are provided).

chainedBatch.del(key[, options])

Queue a del operation on this batch, not committed until write() is called. This will throw a LEVEL_INVALID_KEY error if key is invalid. The optional options object may contain:

  • keyEncoding: custom key encoding for this operation, used to encode the key.
  • sublevel (sublevel instance): act as though the del operation is performed on the given sublevel, to similar effect as sublevel.batch().del(key). This allows atomically committing data to multiple sublevels. The key will be prefixed with the prefix of the sublevel, and the key will be encoded by the sublevel (using the default key encoding of the sublevel unless keyEncoding is provided).

chainedBatch.clear()

Clear all queued operations on this batch.

chainedBatch.write([options][, callback])

Commit the queued operations for this batch. All operations will be written atomically, that is, they will either all succeed or fail with no partial commits.

There are no options (that are common between classic-level and browser-level). Note that write() does not take encoding options. Those can only be set on put() and del().

The callback function will be called with no arguments if the batch was successful or with an error if it failed. If no callback is provided, a promise is returned.

After write() or close() has been called, no further operations are allowed.

chainedBatch.close([callback])

Free up underlying resources. This should be done even if the chained batch has zero queued operations. Automatically called by write() so normally not necessary to call, unless the intent is to discard a chained batch without committing it. The callback function will be called with no arguments. If no callback is provided, a promise is returned. Closing the batch is an idempotent operation, such that calling close() more than once is allowed and makes no difference.

chainedBatch.length

The number of queued operations on the current batch.

chainedBatch.db

A reference to the database that created this chained batch.

iterator

An iterator allows one to lazily read a range of entries stored in the database. The entries will be sorted by keys in lexicographic order (in other words: byte order) which in short means key 'a' comes before 'b' and key '10' comes before '2'.

A classic-level iterator reads from a snapshot of the database, created at the time db.iterator() was called. This means the iterator will not see the data of simultaneous write operations. A browser-level iterator does not offer such guarantees, as is indicated by db.supports.snapshots. That property will be true in Node.js and false in browsers.

Iterators can be consumed with for await...of and iterator.all(), or by manually calling iterator.next() or nextv() in succession. In the latter case, iterator.close() must always be called. In contrast, finishing, throwing, breaking or returning from a for await...of loop automatically calls iterator.close(), as does iterator.all().

An iterator reaches its natural end in the following situations:

  • The end of the database has been reached
  • The end of the range has been reached
  • The last iterator.seek() was out of range.

An iterator keeps track of calls that are in progress. It doesn't allow concurrent next(), nextv() or all() calls (including a combination thereof) and will throw an error with code LEVEL_ITERATOR_BUSY if that happens:

// Not awaited and no callback provided
iterator.next()

try {
  // Which means next() is still in progress here
  iterator.all()
} catch (err) {
  console.log(err.code) // 'LEVEL_ITERATOR_BUSY'
}

for await...of iterator

Yields entries, which are arrays containing a key and value. The type of key and value depends on the options passed to db.iterator().

try {
  for await (const [key, value] of db.iterator()) {
    console.log(key)
  }
} catch (err) {
  console.error(err)
}

iterator.next([callback])

Advance to the next entry and yield that entry. If an error occurs, the callback function will be called with an error. Otherwise, the callback receives null, a key and a value. The type of key and value depends on the options passed to db.iterator(). If the iterator has reached its natural end, both key and value will be undefined.

If no callback is provided, a promise is returned for either an entry array (containing a key and value) or undefined if the iterator reached its natural end.

Note: iterator.close() must always be called once there's no intention to call next() or nextv() again. Even if such calls yielded an error and even if the iterator reached its natural end. Not closing the iterator will result in memory leaks and may also affect performance of other operations if many iterators are unclosed and each is holding a snapshot of the database.

iterator.nextv(size[, options][, callback])

Advance repeatedly and get at most size amount of entries in a single call. Can be faster than repeated next() calls. The size argument must be an integer and has a soft minimum of 1. There are no options at the moment.

If an error occurs, the callback function will be called with an error. Otherwise, the callback receives null and an array of entries, where each entry is an array containing a key and value. The natural end of the iterator will be signaled by yielding an empty array. If no callback is provided, a promise is returned.

const iterator = db.iterator()

while (true) {
  const entries = await iterator.nextv(100)

  if (entries.length === 0) {
    break
  }

  for (const [key, value] of entries) {
    // ..
  }
}

await iterator.close()

iterator.all([options][, callback])

Advance repeatedly and get all (remaining) entries as an array, automatically closing the iterator. Assumes that those entries fit in memory. If that's not the case, instead use next(), nextv() or for await...of. There are no options at the moment. If an error occurs, the callback function will be called with an error. Otherwise, the callback receives null and an array of entries, where each entry is an array containing a key and value. If no callback is provided, a promise is returned.

const entries = await db.iterator({ limit: 100 }).all()

for (const [key, value] of entries) {
  // ..
}

iterator.seek(target[, options])

Seek to the key closest to target. Subsequent calls to iterator.next(), nextv() or all() (including implicit calls in a for await...of loop) will yield entries with keys equal to or larger than target, or equal to or smaller than target if the reverse option passed to db.iterator() was true.

The optional options object may contain:

  • keyEncoding: custom key encoding, used to encode the target. By default the keyEncoding option of the iterator is used or (if that wasn't set) the keyEncoding of the database.

If range options like gt were passed to db.iterator() and target does not fall within that range, the iterator will reach its natural end.

iterator.close([callback])

Free up underlying resources. The callback function will be called with no arguments. If no callback is provided, a promise is returned. Closing the iterator is an idempotent operation, such that calling close() more than once is allowed and makes no difference.

If a next() ,nextv() or all() call is in progress, closing will wait for that to finish. After close() has been called, further calls to next() ,nextv() or all() will yield an error with code LEVEL_ITERATOR_NOT_OPEN.

iterator.db

A reference to the database that created this iterator.

iterator.count

Read-only getter that indicates how many keys have been yielded so far (by any method) excluding calls that errored or yielded undefined.

iterator.limit

Read-only getter that reflects the limit that was set in options. Greater than or equal to zero. Equals Infinity if no limit, which allows for easy math:

const hasMore = iterator.count < iterator.limit
const remaining = iterator.limit - iterator.count

keyIterator

A key iterator has the same interface as iterator except that its methods yield keys instead of entries. For the keyIterator.next(callback) method, this means that the callback will receive two arguments (an error and key) instead of three. Usage is otherwise the same.

valueIterator

A value iterator has the same interface as iterator except that its methods yield values instead of entries. For the valueIterator.next(callback) method, this means that the callback will receive two arguments (an error and value) instead of three. Usage is otherwise the same.

sublevel

A sublevel is an instance of the AbstractSublevel class, which extends AbstractLevel and thus has the same API as documented above. Sublevels have a few additional properties.

sublevel.prefix

Prefix of the sublevel. A read-only string property.

const example = db.sublevel('example')
const nested = example.sublevel('nested')

console.log(example.prefix) // '!example!'
console.log(nested.prefix) // '!example!!nested!'

sublevel.db

Parent database. A read-only property.

const example = db.sublevel('example')
const nested = example.sublevel('nested')

console.log(example.db === db) // true
console.log(nested.db === db) // true

Contributing

Level/level is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Donate

Support us with a monthly donation on Open Collective and help us continue our work.

License

MIT

abstract-leveldown's People

Contributors

achingbrain avatar andrewrk avatar calvinmetcalf avatar deanlandolt avatar dependabot[bot] avatar dominictarr avatar flatheadmill avatar greenkeeper[bot] avatar greenkeeperio-bot avatar hden avatar huan avatar hugomrdias avatar juliangruber avatar kesla avatar mafintosh avatar marcuslyons avatar max-mapper avatar mcollina avatar meirionhughes avatar nolanlawson avatar ralphtheninja avatar raynos avatar rvagg avatar sandersn avatar shama avatar tapppi avatar timoxley avatar vweevers avatar watson avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

abstract-leveldown's Issues

nonErrorValues() test breaks leveldown

A typo was fixed here:

ecd41a7

This actually introduces 40 new tests (leveldown went from 672 tests to 712) that were not being run before and some of them break, see:

https://travis-ci.org/Level/leveldown/builds/60216894#L384

More concretely these tests fail:

https://github.com/Level/abstract-leveldown/blob/master/abstract/put-get-del-test.js#L140-L144

And they fail in this location:

https://github.com/Level/abstract-leveldown/blob/master/abstract/put-get-del-test.js#L50

The result variable is a buffer and it's being compared (===) with an empty string, which obviously fails :)

The question now is where to fix this. Are the tests correctly implemented?

/cc @rvagg @juliangruber

changelog

someone needs to start a changelog...

AbstractChainedBatch tests should commit and inspect db rather than sniff `_operations` buffer

Nothing about AbstractChainedBatch requires it to keep operations around in memory (in the _operations buffer), but some of its tests sniff this buffer rather than committing these values and reading from the db. The tests are mostly around just verifying the _serialize[Key|Value] behavior, so in this case they would probably be better off hooking _put and _del. I do have to do all this nonsense here) from outside the tests, but from within we could just hook the _put and _del methods on the batch directly.

Regardless, these tests should also commit these values and verify what ends up in the db w/ collectEntries. The only exception might be test custom _serialize*, since what will happen when this hits the db is undefined behavior. But this just suggests to me we should add companion _deserialize[Key|Value] methods to get us a hook to reliably verify expected results.

/cc @ralphtheninja @juliangruber

silently drops batch calls

If db.batch is called and there is no _batch specified it does nothing, and fails silently.

I'd expect it to either

  • throw an error
  • have a sane default that async.map's to _put and _del

Perhaps batch guarantees being atomic. If we are unable to provide a sane default for that, then we should throw an error on a batch call

verbose mode

I think it would be really useful to have a flag you could turn on when debugging that made leveldown write something like this to stdout:

OPEN "test"
PUT "ÿfooÿhello", "world", "String"
GET "ÿfooÿhello", "world", "String"
GET "ÿfooÿweeee", "Key not found in database [ÿfooÿweeee]", "NotFoundError"

I could monkeypatch to implement this today as a third party module but I wanted feedback on where in the stack something like this could go. I imagine that simple if (verbose) checks would get inlined by v8 etc

Thoughts on exposing the test suite?

I'm writing a backend for an internal DB and I would like to use the test suite in here in my module so that I know that I'm doing the right thing.

Any thoughts on exposing this test suite in a way that things that use it can also use the test suite?

Auto casting to String on set??

I ran into this with my latest LevelDown abstraction:
https://github.com/rvagg/abstract-leveldown/blob/master/abstract-leveldown.js#L87

if (!this._isBuffer(value) && !process.browser)
    value = String(value)

Anyway we can make that an option and have process.browser default it to true? This way the abstraction can deal with the encoding?

My issue is that my backend server fully supports JSON encoding. Even if I said encoding: 'json' in my options, it's still casting that to a string on set. I'd like to have it just send the raw data to my abstraction and let me deal with the encoding.

If this is ok, I'll work on a PR just wanted to ask first :)

Decide whether gte/gt/lte/lt should take precedence over start/end or ..?

See dominictarr/ltgt#1 for details

Basically there exists a discrepancy between memdown and leveldown where memdown will decide to prefer lte and lt, discarding end if it exists, while leveldown takes the minim of the options. Same would go for gte, gt and start (and the reverse combinations as well).

One of them should be wrong, we need to decide which and implement conformance tests for it here.

open, close, and open

I don't believe there is a test that opens a db, closes it and then opens it again, came up for me with sqldown

An in-range update of sinon is breaking the build 🚨

Version 4.1.0 of sinon was just published.

Branch Build failing 🚨
Dependency sinon
Current Version 4.0.2
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

sinon is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 22 commits.

  • c0a71c6 Update docs/changelog.md and set new release id in docs/_config.yml
  • a2b873a Add release documentation for v4.1.0
  • 0a6a660 4.1.0
  • 3b36972 Update History.md and AUTHORS for new release
  • 201a652 Issue 1598 (Feature Request): Implemented sandbox.createStubInstance, tests, and documentation.
  • d49180d Merge pull request #1603 from mroderick/fix-more-markdown
  • 2d2631c Docs: fix pre commit hook
  • 9fa87e7 Docs: remove trailing quote from heading
  • 46ffad3 Docs: verify documentation using markdownlint
  • aa10bb7 Docs: remove use of element
  • 294ada0 Docs: remove use of
     tag
  • 77e5d31 Docs: reduce unnecessary inline HTML
  • b14a261 Docs: fix invalid syntax of backticks in headers
  • 579e029 Docs: fix trailing punctuation in headers
  • 7b04012 Docs: remove extraneous blank lines

There are 22 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

using empty location + open causes weird results

If I do the following:

var leveldown = require('leveldown')
var db = leveldown('')
db.open(console.log.bind(console))

I get the following output:

[Error: IO error: /LOCK: Permission denied]

Which is kind of odd. I propose that we make AbstractLevelDOWN throw an error if a zero length string is given, instead of finding out in open() that we couldn't open that location. We already know it will fail and might as well error in the constructor.

Enhance the test suite with PouchDB

With PouchDB and PouchDB Server, we're quickly reaching a level of stability where we can test against the various *DOWN backends and say with confidence that, if there's a bug, it's in that library rather than our own.

In fact I just finished an audit of various server-side *DOWN backends and noticed that a lot of them are failing our test suite. I'm not too surprised, though – with index-js, localstoragedown, and MemDOWN, we occasionally ran into cases where a test failed in PouchDB but not in abstract-leveldown. It's just really hard to catch all the cases, and our test suite has gotten pretty huge (somewhere north of 700 tests now).

Obviously in an ideal world the abstract-leveldown test suite would be sufficient for rooting out these bugs, and I could try to be a better LevelUP citizen and contribute some of the failing tests back to this repo. But I think maybe an easier and more effective solution would be if we just offered an quick way for *DOWN implementers to run the PouchDB test suite against their code. Since we've got PouchDB Server working now, we can even set it up so that it uses their adapter on both the server and the client, which makes for a pretty badass test. (Here's an example of MemDOWN vs. MemDOWN, although sadly it's failing.)

Is this something that would be interesting for the LevelUP community, and if so, how should we go about offering those tests to you? Would a simple bash script that npm installs all the dependencies and runs the test be enough? Or would you prefer something else?

Remove test: serialization of object key in approximateSize()

I propose to remove this abstract test:

test('test _serialize object', function (t) {
t.plan(3)
var db = leveldown(testCommon.location())
db._approximateSize = function (start, end, callback) {
t.equal(Buffer.isBuffer(start) ? String(start) : start, '[object Object]')
t.equal(Buffer.isBuffer(end) ? String(end) : end, '[object Object]')
callback()
}
db.approximateSize({}, {}, function (err, val) {
t.error(err)
})
})

We have enough coverage on the _serialize functions, including the extensibility of _serializeKey() in combination with approximateSize(). This test has no added value anymore and assumes that object keys are stringified.

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.

Since we did not receive a CI status on the greenkeeper/initial branch, we assume that you still need to configure it.

If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/.

We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

Once you have installed CI on this repository, you’ll need to re-trigger Greenkeeper’s initial Pull Request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organiszation’s settings page, under Installed GitHub Apps.

genericize test suite to make sense in browser

this almost passes all the leveldown tests (except iterators, havent worked on those yet): https://github.com/maxogden/level.js

there are two fundamental differences though:

  • indexeddb natively supports storing all JS data types (num, bool, string, typed arrays, array buffers etc). with leveldb its either a string of a buffer, and the test suite is currently set up to always expect either a string or a buffer, which means if you store a bool in the browser the test suite converts it to a string
  • Buffer doesn't exist in the browser. instead of using buffer-browserify (which if fundamentally flawed IMO) i'd rather return ArrayBuffers as they are the equivalent primitive binary data type in browsers. the problem with this is that the test suite currently does a lot of Buffer.toString() checking but toString() returns '[object ArrayBuffer]' on ArrayBuffers. instead you have to do String.fromCharCode.apply(null, new Uint16Array(arraybuffer))

so, is it cool if I add a bunch of conditional browser specific stuff to the test suite? is there a better 'paradigm' we could use for return value checking?

ES6 is ending up in the browserify builds

The const keyword is ending up the browserify builds for memdown (e.g. see https://wzrd.in/standalone/memdown@latest), and it is breaking various browsers, notably in the Hoodie tests (Hoodie uses PouchDB uses MemDOWN uses AbstractLevelDOWN). pouchdb/pouchdb#4215

I'm not sure what the most elegant way to fix this is, but for the time being I would suggest just removing const from the source, or else making it a build step to only output ES5-compatible Node code.

Iterator as Async Function

It ought be possible to create an async function (function *iterator or function *async_iterator) alternative to leveldown's own .next() wielding iterator. It is likely beneficial for inclusion.

Remove gaps from batch array?

If you do db.batch([null]), or any other falsy value, should abstract-leveldown filter it?

I noticed that memdown has its own !array[i] check in _batch(), so I wondered if that's a job for abstract-leveldown. We currently do have a typeof array[i] !== 'object' check, which I didn't git blame yet, but this doesn't catch null.

@juliangruber @ralphtheninja

Put implementation converts `null` to `"null"` not `""`.

There are new tests that assert that when a null or undefined value is put into the database that it will be retrieved as an empty string. I assume this means it should be converted to an empty string before insertion. The _serializeValue method converts using String(value) which converts null to "null" and undefined to "undefined".

make location optional

The recent change to validate that a location isn't an empty string was actually a breaking change, as modules like memdown here may pass an empty string.

I'm for making location optional as it has shown that enough backends don't need one.

get rid of process.browser checks

It makes the tests a lot harder to understand. Not exactly sure how we would do it and how it will affect implementations, need time and help from the community to get this right. A suggestion would be to

  1. check the commit history, why were the process.browser checks added in the first place?
  2. if we remove them, what would the consequences be?
  3. rewrite/update implementations that rely on this based in 1. and 2.

See the following comments:

What is the next callback structure?

The abstract down and iterator in here doesn't seems to define what the callback for the iterator next should be. And the implementation implies the callback args could be anything.

The only place where the structure for the next callback is expected to be a certain way seems to be in the stream converter on levelup: https://github.com/Level/iterator-stream/blob/master/index.js#L25 which has it as: function (err, key, value)

Can it be anything? or should it be function (err, key, value, ...more)?

cc: @ralphtheninja

Clean up test and testBuffer globals

This is an investigative issue, will add points as I find them.

Get rid of test and testBuffer "globals", pass them on as function parameters instead

Try to get rid of testBuffer completely, if possible.

Strict abstract leveldown?

Yo!

I've been thinking about an alternative implementation av abstract-leveldown, one that doesn't have nice defaults but throw like crazy if there's anything missing.

I've been fooled a couple of times when working on *DOWN:s by the defaults as is today.

API-wise we could do something like

var util = require('util')
  , StrictAbstractLevelDOWN = require('./').StrictAbstractLevelDOWN

// constructor, passes through the 'location' argument to the AbstractLevelDOWN constructor
function FakeLevelDOWN (location) {
  StrictAbstractLevelDOWN.call(this, location)
}

//etc etc

Interested in a PR?

Some notes on iterator implementation in browser

@maxogden as I closed the pull request I thought I would move my notes on iterator here.

Currently the error test in simple iterator is returning null not undefined
https://github.com/rvagg/node-abstract-leveldown/blob/master/abstract/iterator-test.js#L125

The tests in "Test setup2" fail if the database from simple iterator is not cleared down
https://github.com/rvagg/node-abstract-leveldown/blob/master/abstract/iterator-test.js#L146
Should we have them as different test files with their own setup and teardown?

Rather than have two of us doing the same thing I am leaving iterator spec with you but
If you want me to have a go ping me.

Snapshot test - do we really need it?

I noticed this test was added recently. It's causing some headaches for me in localstorage-down, but I'm wondering how many other *down authors have actually implemented this thing?

Reading the test, I can understand the problem it's trying to solve. Ideally users should be able to open up a read-only iterator against a database and continue to read from a "snapshot" even as others are writing to it.

However, that's a really, really involved feature (concurrency! transactions!), especially for simple modules like MemDOWN and localstorage-down. And this test doesn't even seem to be very thorough. I could adhere to the letter of the law by just passing this one test, while still not really implementing the proper transactional semantics.

What's the feeling about this? Should *down authors pick-and-choose the tests we know we won't support, or should we rise to the occasion and try to pass tests like this?

binary encoding vs. asBuffer

I'm wondering what the exact purpose of the asBuffer option is (and it's cousins keyAsBuffer and valueAsBuffer). To me it feels redundant since one can use keyEncoding binary and/or valueEncoding binary.

I wonder what to do when using IndexedDB as a backend which has support for native JS types as values and supports some different types for keys (Level/level-js#48). I know in level.js they even have a raw option. I can't really tell the difference between encoding, asBuffer and raw.

/cc @nolanlawson

leveldown on AWS lambda

Has anybody attempted to install leveldown on AWS Lambda? If so how did it go? What is the best strategy for installing the binary?

too strict about error messages

this has broken level.js tests, if you upgrade to the latest abstract-leveldown

but for silly reasons like this:

not ok 44 should have correct error message default_stream.js:12
  --- default_stream.js:12
    operator: equal default_stream.js:12
    expected: "NotFound: " default_stream.js:12
    actual:   "NotFound" default_stream.js:12
    at: Test.equal.Test.equals.Test.isEqual.Test.is.Test.strictEqual.Test.strictEquals (http://localhost:9966/test.js:7440:10) default_stream.js:12

it should test with a regular expression t.ok(/^NotFound/.test(err.message))
I'll make a pull request for this later, though, maybe not before the conference.
/cc @maxogden

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.