tikv/client-py

Website | Documentation | Community Chat

TiKV is an open-source, distributed, and transactional key-value database. Unlike other traditional NoSQL systems, TiKV not only provides classical key-value APIs, but also transactional APIs with ACID compliance. Built in Rust and powered by Raft, TiKV was originally created by PingCAP to complement TiDB, a distributed HTAP database compatible with the MySQL protocol.

The design of TiKV ('Ti' stands for titanium) is inspired by some great distributed systems from Google, such as BigTable, Spanner, and Percolator, and some of the latest achievements in academia in recent years, such as the Raft consensus algorithm.

If you're interested in contributing to TiKV, or want to build it from source, see CONTRIBUTING.md.

TiKV is a graduated project of the Cloud Native Computing Foundation (CNCF). If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how TiKV plays a role, read the CNCF announcement.

---

With the implementation of the Raft consensus algorithm in Rust and consensus state stored in RocksDB, TiKV guarantees data consistency. Placement Driver (PD), which is introduced to implement auto-sharding, enables automatic data migration. The transaction model is similar to Google's Percolator with some performance improvements. TiKV also provides snapshot isolation (SI), snapshot isolation with lock (SQL: SELECT ... FOR UPDATE), and externally consistent reads and writes in distributed transactions.

TiKV has the following key features:

• Geo-Replication

  TiKV uses Raft and the Placement Driver to support Geo-Replication.

• Horizontal scalability

  With PD and carefully designed Raft groups, TiKV excels in horizontal scalability and can easily scale to 100+ TBs of data.

• Consistent distributed transactions

  Similar to Google's Spanner, TiKV supports externally-consistent distributed transactions.

• Coprocessor support

  Similar to HBase, TiKV implements a coprocessor framework to support distributed computing.

• Cooperates with TiDB

  Thanks to the internal optimization, TiKV and TiDB can work together to be a compelling database solution with high horizontal scalability, externally-consistent transactions, support for RDBMS, and NoSQL design patterns.

Governance

See Governance.

Documentation

For instructions on deployment, configuration, and maintenance of TiKV,see TiKV documentation on our website. For more details on concepts and designs behind TiKV, see Deep Dive TiKV.

Note:

We have migrated our documentation from the TiKV's wiki page to the official website. The original Wiki page is discontinued. If you have any suggestions or issues regarding documentation, offer your feedback here.

TiKV adopters

You can view the list of TiKV Adopters.

TiKV software stack

• Placement Driver: PD is the cluster manager of TiKV, which periodically checks replication constraints to balance load and data automatically.
• Store: There is a RocksDB within each Store and it stores data into the local disk.
• Region: Region is the basic unit of Key-Value data movement. Each Region is replicated to multiple Nodes. These multiple replicas form a Raft group.
• Node: A physical node in the cluster. Within each node, there are one or more Stores. Within each Store, there are many Regions.

When a node starts, the metadata of the Node, Store and Region are recorded into PD. The status of each Region and Store is reported to PD regularly.

Quick start

Deploy a playground with TiUP

The most quickest to try out TiKV with TiDB is using TiUP, a component manager for TiDB.

You can see this page for a step by step tutorial.

Deploy a playground with binary

TiKV is able to run separately with PD, which is the minimal deployment required.

1. Download and extract binaries.

$ export TIKV_VERSION=v7.5.0
$ export GOOS=darwin  # only {darwin, linux} are supported
$ export GOARCH=amd64 # only {amd64, arm64} are supported
$ curl -O  https://tiup-mirrors.pingcap.com/tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ curl -O  https://tiup-mirrors.pingcap.com/pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz

2. Start PD instance.

$ ./pd-server --name=pd --data-dir=/tmp/pd/data --client-urls="http://127.0.0.1:2379" --peer-urls="http://127.0.0.1:2380" --initial-cluster="pd=http://127.0.0.1:2380" --log-file=/tmp/pd/log/pd.log

3. Start TiKV instance.

$ ./tikv-server --pd-endpoints="127.0.0.1:2379" --addr="127.0.0.1:20160" --data-dir=/tmp/tikv/data --log-file=/tmp/tikv/log/tikv.log

4. Install TiKV Client(Python) and verify the deployment, required Python 3.5+.

$ pip3 install -i https://test.pypi.org/simple/ tikv-client

from tikv_client import RawClient

client = RawClient.connect("127.0.0.1:2379")

client.put(b'foo', b'bar')
print(client.get(b'foo')) # b'bar'

client.put(b'foo', b'baz')
print(client.get(b'foo')) # b'baz'

Deploy a cluster with TiUP

You can see this manual of production-like cluster deployment presented by @c4pt0r.

Build from source

See CONTRIBUTING.md.

Client drivers

• Go (The most stable and widely used)
• Java
• Rust
• C

If you want to try the Go client, see Go Client.

Security

Security audit

A third-party security auditing was performed by Cure53. See the full report here.

Reporting Security Vulnerabilities

To report a security vulnerability, please send an email to TiKV-security group.

See Security for the process and policy followed by the TiKV project.

Communication

Communication within the TiKV community abides by TiKV Code of Conduct. Here is an excerpt:

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Social Media

• Twitter
• Blog
• Reddit
• Post questions or help answer them on Stack Overflow

Slack

Join the TiKV community on Slack - Sign up and join channels on TiKV topics that interest you.

License

TiKV is under the Apache 2.0 license. See the LICENSE file for details.

Acknowledgments

• Thanks etcd for providing some great open source tools.
• Thanks RocksDB for their powerful storage engines.
• Thanks rust-clippy. We do love the great project.

client-py's Issues

python rawclinet batch put meet exception

Here is my code

pairs = {}
for record in r['records']:
    rowkey = urllib.parse.unquote(record['rowkey'])
    for features in record['columns']:
        if features['name'] == columns:
            features = urllib.parse.unquote(features['value'])
            # pairs[rowkey.encode()] = features.encode()
            pairs[b'123'] = b'456'
for key, value in pairs.items():
    print(key, len(value))
client.batch_put(pairs)

then I got exception like this:

Traceback (most recent call last):
  File "test_tikv.py", line 69, in <module>
    func(*sys.argv[2:])
  File "test_tikv.py", line 36, in sync
    client.batch_put(pairs)
  File "/home/work/.jumbo/lib/python3.6/site-packages/tikv_client/__init__.py", line 33, in batch_put
    event_loop.run_until_complete(self.inner.put(pairs, cf))
TypeError: Can't convert {b'123': b'456'} to Sequence

do you guys have a more specific examples?

release to pypi.org

error when using your example

I'm following the 5 minute tutorial and getting the following

Traceback (most recent call last):
  File "ti.py", line 3, in <module>
    client = RawClient.connect("127.0.0.1:2379")
  File "/Users/sean.carey/venv/lib/python3.8/site-packages/tikv_client/__init__.py", line 15, in connect
    asynchronous.RawClient.connect(pd_endpoint))
RuntimeError: no running event loop

Code Example

from tikv_client import TransactionClient

client = TransactionClient.connect("127.0.0.1:2379")

txn = client.begin(pessimistic=True)
txn.put(b"k1", b"v1")
txn.put(b"k2", b"v2")
txn.put(b"k3", b"v3")
txn.put(b"k4", b"v4")
txn.put(b"k5", b"v5")
txn.commit()

snapshot = client.snapshot(client.current_timestamp())
print(snapshot.get(b"k3"))
print(snapshot.batch_get([b"k1", b"k4"]))

for k, v in snapshot.scan(b"k1", end=None, limit=10, include_start=False):
    print(k, v)

Project status

Is there any interest in resurrecting the project? Maturin and pyo3 had many releases and the async/await bindings are less experimental too.

As of today I’m not aware of any HA distributed KV store available from python in a safe and ergonomic way.

rawkv batch_get return None

code

from tikv_client import RawClient

client = RawClient.connect("127.0.0.1:2379")

client.put(b"k1", b"v1")
client.put(b"k2", b"v2")
client.put(b"k3", b"v3")
client.put(b"k4", b"v4")
client.put(b"k5", b"v5")

print(client.batch_get([b"k1", b"k4"]))

output

None

expected

[(b'k4', b'v4'), (b'k1', b'v1')]

When will TiKV-python-client release?

TiKV is amazing!
We decide to adopt TiKV in our project, however, it's python client is not released.
So, can anyone tell me, what's your plan? and when will TiKV-python-client release?

BRs!

Multiple PD endpoints

The official rust client docs say

Always use more than one PD endpoint!

https://tikv.org/docs/3.0/reference/clients/rust/

Looking at the code, it looks straightforward how to change to multiple pd endpoints.

 #[classmethod]
    pub fn connect<'p>(_cls: &PyType, py: Python<'p>, pd_endpoint: String) -> PyResult<&'p PyAny> {
        future_into_py(py, async move {
            let inner = tikv_client::RawClient::new(vec![pd_endpoint], None)

batch_get returns None incorrectly

Minimal Reproduce Step

from tikv_client import RawClient
client = RawClient.connect("127.0.0.1:2379")
client.put(b"k1", b"v1")
client.put(b"k2", b"v2")
client.put(b"k3", b"v3")
client.put(b"k4", b"v4")
client.put(b"k5", b"v5")
print(client.get(b"k1"))
print(client.batch_get([b"k1", b"k4"]))

Expected

b'v1'
[b'v1', b'v4']

Actual

b'v1'
None

Error when writing data with Transaction

I am writing data to TiKV using Transaction:

tikv_txn_client = await _connect_to_txn_tikv()
write_txn = await tikv_txn_client.begin()

await write_txn.put(msgpack.packb(k), msgpack.packb(list(v)))

Where k is a string, and v is a list or set (notice it is being cast to a list).

I am using message pack to convert the k and v to bytes.

I am getting the following error:

Exception: Multiple key errors: [KeyError(KeyError { locked: None, retryable: "", abort: "Error(InvalidKeyMode { cmd: prewrite, storage_api_version: V2, key: \"D92233416D377250FF7743724A55683132FF7959593465646F54FF76364B564E5A6E46FF6831734C00000000FB\" })", conflict: None, already_exist: None, deadlock: None, commit_ts_expired: None, txn_not_found: None, commit_ts_too_large: None, assertion_failed: None })]

Release?

Would it be possible to get a new pypi release?

Release cycle ?

I would like to ask how the release cycle of this project is managed. @andylokandy

Update to the lastest Rust core client

As massive development has been done in Rust client, Python client, the surface wrapper of Rust client, is outdated in core functionalities as well as the surface API.

Release on conda-forge

It's really popular to install packed using conda and the conda-forge repo

Help wanted: add keyspace and raw ttl upgrade from upstream

Rust client has implemented keyspace and raw TTL features (link) recently. The API of python client need to be updated to support such features.

RawClient will try to connect to pd during initialization

>>> from tikv_client.asynchronous import RawClient
>>> client = RawClient('127.0.0.1:23790')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception: [/home/cosven/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/ac96d14/tikv-client-pd/src/cluster.rs:174]: PD cluster failed to respond

How about add another method for RawClient to initialize the rust-tikv-RawClient.

• On the one hand, sending network request during __init__ is not pythonic.
• On the other hand, we can't await RawClient(xxx).

TransactionClient can't connect to pd address

from tikv_client import TransactionClient

client = TransactionClient.connect("127.0.0.1:2379")
Traceback (most recent call last):
File "", line 1, in
AttributeError: type object 'TransactionClient' has no attribute 'connect'

Build tikv-client failed

OS: CentOS Linux release 7.9.2009
rustc: 1.56.0-nightly (50171c310 2021-09-01)

$ maturin build
� Building a mixed python/rust project
� Found pyo3 bindings with abi3 support for Python ≥ 3.6
� Not using a specific python interpreter (With abi3, an interpreter is only required on windows)
� Built source distribution to /data/zy/tikv/target/wheels/tikv_client-0.0.0.tar.gz
   Compiling prost-derive v0.7.0
   Compiling futures-executor v0.3.17
   Compiling tokio-util v0.6.7
   Compiling native-tls v0.2.8
   Compiling pyo3 v0.14.4
   Compiling grpcio-sys v0.9.0+1.38.0 (https://github.com/tikv/grpc-rs.git?branch=master#69abb443)
   Compiling tokio-native-tls v0.3.0
   Compiling futures v0.3.17
error[E0034]: multiple applicable items in scope: prost-derive, grpcio-sys(build), tokio-util, tokio-native-tls, pyo3, futures-executor
   --> /data/zy/database_testing/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-derive-0.7.0/src/lib.rs:109:14
    |
109 |             .intersperse(quote!(|));
    |              ^^^^^^^^^^^ multiple `intersperse` found
    |
    = note: candidate #1 is defined in an impl of the trait `Iterator` for the type `Map<I, F>`
    = note: candidate #2 is defined in an impl of the trait `Itertools` for the type `T`
help: disambiguate the associated function for candidate #1
    |
105 ~         let tags = Iterator::intersperse(field
106 +             .tags()
107 +             .into_iter()
108 +             .map(|tag| quote!(#tag)), {
109 +         let mut _s = $crate::__private::TokenStream::new();
110 +         $crate::quote_each_token!(_s $($tt)*);
  ...
help: disambiguate the associated function for candidate #2
    |
105 ~         let tags = Itertools::intersperse(field
106 +             .tags()
107 +             .into_iter()
108 +             .map(|tag| quote!(#tag)), {
109 +         let mut _s = $crate::__private::TokenStream::new();
110 +         $crate::quote_each_token!(_s $($tt)*);
  ...


   Compiling h2 v0.3.4
error: aborting due to previous error


   For more information about this error, try `rustc --explain E0034`.

error: could not compile `prost-derive` due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error: build failed
� maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `cargo rustc --message-format json --manifest-path Cargo.toml --lib --`

cannnot connect to PD on k8s from local.

I have deployed tikv on my k8s cluster using tidb operator. when i port forward basic-pd:2379 service to my local and try to connect to it using python client.
i am using
client = TransactionClient.connect(["127.0.0.1:2379"])
to connect to PD, which gives error:

Exception: [/root/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/eb1d2da/tikv-client-pd/src/cluster.rs:264]: failed to connect to [Member { name: "basic-pd-0", member_id: 15548971879676424159, peer_urls: ["http://basic-pd-0.basic-pd-peer.tikv.svc:2380/"], client_urls: ["http://basic-pd-0.basic-pd-peer.tikv.svc:2379/"], leader_priority: 0, deploy_path: "", binary_version: "", git_hash: "", dc_location: ""

I think its advertising its service url and that is obviously not accessible from my local.
Although if i exec to any pod on my k8s cluster i am able to connect to tikv.
Any idea on how can i connect from my local to tikv?

Rollback

Is there any reason why rollback is not implemented on the PyO3 implementation?

return list or tuple in client.scan method?

Currently, the RawClient.scan return a dict:

client-py/src/raw.rs

Lines 77 to 78 in 27d14d7

	let kv_pairs = inner?.scan(range, limit).await.map_err(to_py_execption)?;
	Ok(to_py_dict(kv_pairs)?)

However, the dict may be unordered(depending on Python interpreter implementation). As a result, it's hard for us to get the biggest key from the scan response. I want to use the biggest key as the start key of the next scan request.