Giter Site home page Giter Site logo

surrealdb / surrealkv Goto Github PK

View Code? Open in Web Editor NEW
213.0 18.0 15.0 355 KB

A low-level, versioned, embedded, ACID-compliant, key-value database for Rust

Home Page: https://surrealdb.com

License: Apache License 2.0

Rust 99.91% Makefile 0.09%
embedded-database key-value-database key-value-store kv-store surreal surrealdb surrealkv

surrealkv's Introduction

surrealkv

surrealkv is a versioned, low-level, persistent, embedded key-value database implemented in Rust. This ACID-compliant database offers the following features:

  • Transaction Support: SurrealKV supports rich transactions, allowing multiple items to be inserted, updated, or deleted simultaneously. These transactions are applied atomically, ensuring atomicity and consistency in line with ACID principles. This makes updates invisible until they are committed.

  • Isolation: SurrealKV provides two levels of isolation - Snapshot Isolation and Serializable Snapshot Isolation - to prevent concurrent transactions from interfering with each other.

  • Durability: SurrealKV ensures durability by persisting data on disk, protecting against data loss in the event of a system failure.

  • Multi-Version Concurrency Control (MVCC): SurrealKV employs immutable versioned adaptive radix tries via vart. This allows for any number of concurrent readers and writers to operate without blocking each other.

For more information on the underlying immutable versioned adaptive radix tries used, visit vart.

License

Features

  • In-memory Index
  • Embeddable
  • ACID Semantics
  • Transaction Support:
  • Built-in Item Versioning API
  • Multi-Version Concurrency Control (MVCC) support
  • Multiple Concurrent Readers and Writers
  • Persistence through an append-only File
  • Compaction

Important Notice

This project is actively evolving, and as such, there might be changes to the file format, APIs, and feature set in future releases until reaching stability. Developers are encouraged to stay informed about updates and review future release notes for any breaking changes.

Feel free to contribute, provide feedback, or report issues to help shape the future of surrealkv. Thank you for your interest and involvement in this project!

surrealkv's People

Contributors

arriqaaq avatar ax9d avatar gsserge avatar sgirones avatar tobiemh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

surrealkv's Issues

Avoid excessive copying of keys

Description

There is an issue with excessive copying in surrealkv related to handling snapshots and the vart library. This needs to be optimized to avoid unnecessary allocations and copying.

Details

Whenever setting on a snapshot in surrealkv, the terminate method is called on the key:

let key = &key.terminate();

This terminate method performs an unnecessary copy just to extend the data:

https://github.com/surrealdb/vart/blob/5fe7118140e7da5bdf7c6bcd2d05e0f19725aa88/src/lib.rs#L247

The call to terminate shouldn't happen because, during a set/get operation

.set(&e.key[..].into(), index_value);

, calling on the key invokes the following function:

https://github.com/surrealdb/vart/blob/5fe7118140e7da5bdf7c6bcd2d05e0f19725aa88/src/lib.rs#L284

To optimize, the null byte can be added directly within the function mentioned above to eliminate the unnecessary allocation.

Suggested Solution

  • Remove the call to terminate on the key in surrealkv.
  • Add the null byte directly when doing internal conversion.

This change should reduce unnecessary copying and improve performance.

Publish on Helm

A helm chart release for (which 1:1 versioning) would be good with a CI process for SurrealKV

Range scan does not produce keys in ordered format

Issue Description

Range scan on vart does not produce ordered keys on bulk_insert based on this test. Current fix is to order keys before writing to the index.

    #[tokio::test]
    async fn scan_multiple_keys_within_single_transaction() {
        let (store, _) = create_store(false);
        // Define key-value pairs for the test
        let keys_to_insert = vec![
            Bytes::from("test1"),
            Bytes::from("test2"),
            Bytes::from("test3"),
        ];

        let mut txn = store.begin().unwrap();
        for key in &keys_to_insert {
            txn.set(key, key).unwrap();
        }
        txn.commit().await.unwrap();

        let range = "test1".as_bytes()..="test7".as_bytes();

        let txn = store.begin().unwrap();
        let results = txn.scan(range, None).unwrap();
        assert_eq!(results.len(), 3);
        assert_eq!(results[0].0, keys_to_insert[0]);
        assert_eq!(results[1].0, keys_to_insert[1]);
        assert_eq!(results[2].0, keys_to_insert[2]);
        assert_eq!(results[0].1, keys_to_insert[0]);
        assert_eq!(results[1].1, keys_to_insert[1]);
        assert_eq!(results[2].1, keys_to_insert[2]);
    }

Minimum & Recommended Settings

Here's a list of minimum / recommend settings from current Surreal backends:

Typical deployments of TiKV include a number of components:

3+ TiKV nodes
3+ Placement Driver (PD) nodes
1 Monitoring node
1 or more client application or query layer
System requirements

The minimum specifications for testing or developing TiKV or PD are:

2+ core
8+ GB RAM
An SSD

Production Specifications

The suggested PD specifications for production are:

3+ nodes
4+ cores
8+ GB RAM, with no swap space.
200+ GB Optane, NVMe, or SSD drive
10 Gigabit ethernet (2x preferred)
A Linux Operating System (PD is most widely tested on CentOS 7).
The suggested TiKV specifications for production are:

3+ nodes
16+ cores
32+ GB RAM, with no swap space.
200+ GB Optane, NVMe, or SSD drive (Under 1.5 TB capacity is ideal in our tests)
10 Gigabit ethernet (2x preferred)
A Linux Operating System (TiKV is most widely tested on CentOS 7).

FoundationDB:

FoundationDB minimum specs for production are:

RHEL or CentOS 6 or 7
Ubuntu 12.04 or later
4 GB of RAM per CPU process
Storage as required (more on that below)


All things considered what would the minimum and recommended requirements for SurrealKV look like, even if its just approximate, I believe the requirements above are pretty outrageous considering alternatives such as Postgres and MySQL can run on far less resources out of the box.

Close snapshot iterators post scan

Description

Currently the iterators from a snapshot for scans are not closed, which could lead to unnecessary pointers to older nodes in VART. Close the reader once the scan is done in a transaction.

Make log crate async

Issue Description

The current implementation of the log crate does not utilize any async features, as suggested by @rushmorem. To address this, we need to assess the feasibility of incorporating tokio::fs for async disk writes.

This is not `chore:` is a `feat:`

This is not chore: is a feat:

chore: updating grunt tasks etc; no production code change

I am seeing that in all commits they are using chore, which confused me since this is used for things like ci, tasks in makefiles or cargo make, but not to modify code. Even angular stopped using chore: to use other things like ci: among others.

Please stop using chore, it is not suitable for these commits that modify the code.

G2 Anomaly Fails for SSI for range reads because of write skew

Description

SKV uses SSI for serializability. For individual get/set/delete operations, it avoids write skews. But on range reads, there is still a possibility of write skews which is akin to the g2 (G2: Anti-Dependency Cycles (write skew on predicate read)) test from hermitage.

  "g2")
    echo "Running g2 test."
    tell 0 "begin"
    tell 1 "begin"
    tell 0 "select * from test where value % 3 = 0"
    tell 1 "select * from test where value % 3 = 0"
    tell 0 "insert into test (id, value) values(3, 30)"
    tell 1 "insert into test (id, value) values(4, 42)"
    tell 0 "commit"
    tell 1 "commit" # Rejected with ERROR: FoundationDB commit aborted: 1020 - not_committed
    ;;

SurrealKV limitations

There is a downstream issue I've raised for Surreal: stalwartlabs/mail-server#23 and several points were brought up in context of the KV backends supported by Surreal, specifically on this comment stalwartlabs/mail-server#23 (comment). For some additional context, the particular issue in question is related to Stalwart Mail Server, a rust based fullstack mail solution.

Some points that were brought up were:

SurrealDB is not able to store anything larger than 100KB in FoundationDB and 8MB in TiKV. And, in the case of FoundationDB, any transaction larger than 10MB will fail. These limitations also apply to full-text indexing, storing more than a hundred thousands documents in SurrealDB will also result in errors once the RoaringTreemaps grow over 100KB.

SurrealDB website they list both FDB and TiKV as complete but based on the developer's response it is clear that their support for distributed environments is far from complete.

How does SurrealKV stack up against alternatives? with the comments from the author of the issue in mind. My goal here is to understand the limitations and work around these, specifically where Stalwarts Surreal storage backend is concerned.

Related:

Change error message returned for conflict detection

Description

Currently if there are conflicts between concurrent transactions, skv returns a TransactionReadConflict message. This can be confusing based on the discussion here. Need to change the name to make it generic of if specific, should be either of the two (TransactionReadConflict or TransactionWriteConflict)

Feature: Dockerimage

Since SurrealKV is ready for usage (as beta) with SurrealDB, it'd be useful to have a Dockerimage and have it published on Docker hub

OPFS Support

Is there any intention to support this in the browser via Origin Private File System?

To me, one of the key strengths of Surreal is the ability to run it anywhere. Hopefully that persists with SurrealKV.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.