web3-storage/pail

🪣 DAG based key value store.

pail

DAG based key value store. Sharded DAG that minimises traversals and work to build shards.

• 📖 Read the SPEC.
• 🎬 Watch the Presentation.

Install

npm install @web3-storage/pail

Usage

import { put, get, del } from '@web3-storage/pail'
import { ShardBlock } from '@web3-storage/pail/shard'
import { MemoryBlockstore } from '@web3-storage/pail/block'

// Initialize a new bucket
const blocks = new MemoryBlockstore()
const init = await ShardBlock.create() // empty root shard
await blocks.put(init.cid, init.bytes)

// Add a key and value to the bucket
const { root, additions, removals } = await put(blocks, init.cid, 'path/to/data0', dataCID0)

console.log(`new root: ${root}`)

// Process the diff
for (const block of additions) {
  await blocks.put(block.cid, block.bytes)
}
for (const block of removals) {
  await blocks.delete(block.cid)
}

Batch operations

If adding many multiple items to the pail together, it is faster to batch them together.

import { put, get, del } from '@web3-storage/pail'
import { ShardBlock } from '@web3-storage/pail/shard'
import { MemoryBlockstore } from '@web3-storage/pail/block'
import * as Batch from '@web3-storage/pail/batch'

// Initialize a new bucket
const blocks = new MemoryBlockstore()
const init = await ShardBlock.create() // empty root shard
await blocks.put(init.cid, init.bytes)

const batch = await Batch.create(blocks, init.cid)

// items is an array of `{ key: string, value: CID }` - the items to add to the pail
for (const item of items) {
  await batch.put(item.key, item.value)
}

const { root, additions, removals } = await batch.commit()

console.log(`new root: ${root}`)

// Process the diff
for (const block of additions) {
  await blocks.put(block.cid, block.bytes)
}
for (const block of removals) {
  await blocks.delete(block.cid)
}

Contributing

Feel free to join in. All welcome. Open an issue!

License

Dual-licensed under MIT or Apache 2.0

pail's Issues

not work with `blockstore-core`

readme suggests we use blockstore-core. but pail expects get(cid) => {cid, bytes}, while blockstore-core.get returns a unit8array see: https://ipfs.github.io/js-stores/classes/blockstore_core.MemoryBlockstore.html#get

Add CRDT delete and list functions

Currently missing.

Add sync API

I think the majority of operations are async because of the async Block.encode, however it only really is like that because the hash interface used to be async only. It's now optinally async and we're just using sha256 which can be sync in Nodejs. We could switch to an Await<T> interface that allows async/sync operation depending on the environment. This might speed up mutations in Nodejs.

Binary search in shard

We currently iterate over the entire shard when searching for a key. We could so a binary search instead to speed it up.

blockstore interface changed?

Looks like the blockstore-core interface may have changed? Get's returned values are currently just Uint8Array instead of { cid, block } objects.

This results in broken usage:

https://github.com/alanshaw/pail/blob/777cbbafada0ca13494934331a6df999a273346c/README.md?plain=1#L22-L28

due to

https://github.com/alanshaw/pail/blob/777cbbafada0ca13494934331a6df999a273346c/src/shard.js#L67-L76

Here block is a Uint8Array and .bytes is undefined

Add user friendly API

The current API is purposely low level and stateless.