tikv/client-rust

Rust Client for TiKV.

Website | Documentation | Community Chat

TiKV is an open-source, distributed, and transactional key-value database. Unlike other traditional NoSQL systems, TiKV not only provides classical key-value APIs, but also transactional APIs with ACID compliance. Built in Rust and powered by Raft, TiKV was originally created by PingCAP to complement TiDB, a distributed HTAP database compatible with the MySQL protocol.

The design of TiKV ('Ti' stands for titanium) is inspired by some great distributed systems from Google, such as BigTable, Spanner, and Percolator, and some of the latest achievements in academia in recent years, such as the Raft consensus algorithm.

If you're interested in contributing to TiKV, or want to build it from source, see CONTRIBUTING.md.

TiKV is a graduated project of the Cloud Native Computing Foundation (CNCF). If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how TiKV plays a role, read the CNCF announcement.

---

With the implementation of the Raft consensus algorithm in Rust and consensus state stored in RocksDB, TiKV guarantees data consistency. Placement Driver (PD), which is introduced to implement auto-sharding, enables automatic data migration. The transaction model is similar to Google's Percolator with some performance improvements. TiKV also provides snapshot isolation (SI), snapshot isolation with lock (SQL: SELECT ... FOR UPDATE), and externally consistent reads and writes in distributed transactions.

TiKV has the following key features:

• Geo-Replication

  TiKV uses Raft and the Placement Driver to support Geo-Replication.

• Horizontal scalability

  With PD and carefully designed Raft groups, TiKV excels in horizontal scalability and can easily scale to 100+ TBs of data.

• Consistent distributed transactions

  Similar to Google's Spanner, TiKV supports externally-consistent distributed transactions.

• Coprocessor support

  Similar to HBase, TiKV implements a coprocessor framework to support distributed computing.

• Cooperates with TiDB

  Thanks to the internal optimization, TiKV and TiDB can work together to be a compelling database solution with high horizontal scalability, externally-consistent transactions, support for RDBMS, and NoSQL design patterns.

Governance

See Governance.

Documentation

For instructions on deployment, configuration, and maintenance of TiKV,see TiKV documentation on our website. For more details on concepts and designs behind TiKV, see Deep Dive TiKV.

Note:

We have migrated our documentation from the TiKV's wiki page to the official website. The original Wiki page is discontinued. If you have any suggestions or issues regarding documentation, offer your feedback here.

TiKV adopters

You can view the list of TiKV Adopters.

TiKV software stack

• Placement Driver: PD is the cluster manager of TiKV, which periodically checks replication constraints to balance load and data automatically.
• Store: There is a RocksDB within each Store and it stores data into the local disk.
• Region: Region is the basic unit of Key-Value data movement. Each Region is replicated to multiple Nodes. These multiple replicas form a Raft group.
• Node: A physical node in the cluster. Within each node, there are one or more Stores. Within each Store, there are many Regions.

When a node starts, the metadata of the Node, Store and Region are recorded into PD. The status of each Region and Store is reported to PD regularly.

Quick start

Deploy a playground with TiUP

The most quickest to try out TiKV with TiDB is using TiUP, a component manager for TiDB.

You can see this page for a step by step tutorial.

Deploy a playground with binary

TiKV is able to run separately with PD, which is the minimal deployment required.

1. Download and extract binaries.

$ export TIKV_VERSION=v7.5.0
$ export GOOS=darwin  # only {darwin, linux} are supported
$ export GOARCH=amd64 # only {amd64, arm64} are supported
$ curl -O  https://tiup-mirrors.pingcap.com/tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ curl -O  https://tiup-mirrors.pingcap.com/pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz

2. Start PD instance.

$ ./pd-server --name=pd --data-dir=/tmp/pd/data --client-urls="http://127.0.0.1:2379" --peer-urls="http://127.0.0.1:2380" --initial-cluster="pd=http://127.0.0.1:2380" --log-file=/tmp/pd/log/pd.log

3. Start TiKV instance.

$ ./tikv-server --pd-endpoints="127.0.0.1:2379" --addr="127.0.0.1:20160" --data-dir=/tmp/tikv/data --log-file=/tmp/tikv/log/tikv.log

4. Install TiKV Client(Python) and verify the deployment, required Python 3.5+.

$ pip3 install -i https://test.pypi.org/simple/ tikv-client

from tikv_client import RawClient

client = RawClient.connect("127.0.0.1:2379")

client.put(b'foo', b'bar')
print(client.get(b'foo')) # b'bar'

client.put(b'foo', b'baz')
print(client.get(b'foo')) # b'baz'

Deploy a cluster with TiUP

You can see this manual of production-like cluster deployment presented by @c4pt0r.

Build from source

See CONTRIBUTING.md.

Client drivers

• Go (The most stable and widely used)
• Java
• Rust
• C

If you want to try the Go client, see Go Client.

Security

Security audit

A third-party security auditing was performed by Cure53. See the full report here.

Reporting Security Vulnerabilities

To report a security vulnerability, please send an email to TiKV-security group.

See Security for the process and policy followed by the TiKV project.

Communication

Communication within the TiKV community abides by TiKV Code of Conduct. Here is an excerpt:

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Social Media

• Twitter
• Blog
• Reddit
• Post questions or help answer them on Stack Overflow

Slack

Join the TiKV community on Slack - Sign up and join channels on TiKV topics that interest you.

License

TiKV is under the Apache 2.0 license. See the LICENSE file for details.

Acknowledgments

• Thanks etcd for providing some great open source tools.
• Thanks RocksDB for their powerful storage engines.
• Thanks rust-clippy. We do love the great project.

client-rust's Issues

Client failed to connect

I try to use the client to connect to a tikv running in docker-compose and getting the following error.
/home/user/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/3ab4f95/src/rpc/pd/leader.rs:433]: failed to connect to [name: "pd0" member_id: 2569432990168970607 peer_urls: "http://pd0:2380" client_urls: "http://pd0:2379"] }

I don't exactly know if it's a bug or just wrong usage.
Am I missing anything?

Client Code

use futures::Future;
use tikv_client::{raw::Client, Config};
fn main() {
    let config = Config::new(vec!["172.18.02:2379"]);
    let connect = Client::new(config);
    let client =  connect.wait().unwrap() 
    let key = "a";
    let val = "b";
    let req = client.put(key, val).wait();
}

Docker Compose File

version: '2.1'

services:
  pd0:
    image: pingcap/pd:latest
    ports:
      - "2379"
    command:
      - --name=pd0
      - --client-urls=http://0.0.0.0:2379
      - --peer-urls=http://0.0.0.0:2380
      - --advertise-client-urls=http://pd0:2379
      - --advertise-peer-urls=http://pd0:2380
      - --initial-cluster=pd0=http://pd0:2380
    restart: on-failure
  
  tikv0:
    image: pingcap/tikv:latest
    command:
      - --addr=0.0.0.0:20160
      - --advertise-addr=tikv0:20160
      - --data-dir=/data/tikv0
      - --pd=pd0:2379

    depends_on:
      - "pd0"
    restart: on-failure

UCP: Add unit tests for the timestamp oracle

Description

#92 only contains an integration test for the timestamp oracle.

It is better for us to have unit tests without a real PD instance. With a mock PD, we can also inject failpoints to test failure cases.

Difficulty

• Medium

Score

• 300

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Rust

panic when running the transaction example

I tried to run client-rust's transcation example. And it paniced after scan with the following message:

thread '<unnamed>' panicked at 'no results should be impossible', src/transaction/requests.rs:472:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And if I run it with RUST_BACKTRACE=1, it gave me an empty stack backtrace.

OS: macOS 10.15.4, also occured when tested on ubuntu in docker
tikv version: 4.1.0-alpha
pd version: v4.0.0-rc.2-15-g902babf4

Panicked when running example/raw.rs

Output:

Put key "TiKV", value "Rust".
Get key Key(54694B56) returned value Some(Value("Rust")).
Key: Key(54694B56) deleted
Found values: [KvPair(6B31, "v1"), KvPair(6B32, "v2")] for keys: [Key(6B31), Key(6B32)]
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[Key(6B31)]`,
 right: `[Key(6B31), Key(6B32)]`', src/main.rs:103:5

I think the problem is (in raw.rs):

    // Scanning a range of keys is also possible giving it two bounds
    // it will returns all entries between these two.
    let start = "k1";
    let end = "k2";
    let pairs = client
        .scan(start..=end, 10)
        .key_only()
        .wait()
        .expect("Could not scan");

    let keys: Vec<_> = pairs.into_iter().map(|p| p.key().clone()).collect();
    assert_eq!(&keys, &[Key::from("k1"), Key::from("k2")]);
    println!("Scaning from {:?} to {:?} gives: {:?}", start, end, keys);

Does scan in this case return the right border?

Upgrade gRPC and remove futures compat code

GRPC-rs now supports futures 0.3 so if we upgrade then we shouldn’t need the compatibility shims to go from 0.1 to 0.3.

Failed to import client-rust in tikv

We need to import client-rust in tikv for our hackathon project, and we add tikv-client in dependencies in tikv project, and some compiling error occurs.

error[E0592]: duplicate definitions with name `generate_files`
  --> /Users/zhuangtianyi/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/protobuf-build-0.10.0/src/protobuf_impl.rs:49:5
   |
49 | /     pub fn generate_files(&self) {
50 | |         check_protoc_version(&get_protoc());
51 | |         let mut cmd = Command::new(get_protoc());
52 | |         let desc_file = format!("{}/mod.desc", self.out_dir);
...  |
96 | |         self.replace_read_unknown_fields();
97 | |     }
   | |_____^ duplicate definitions for `generate_files`
   |
  ::: /Users/zhuangtianyi/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/protobuf-build-0.10.0/src/prost_impl.rs:5:5
   |
5  | /     pub fn generate_files(&self) {
6  | |         #[cfg(feature = "grpcio-prost-codec")]
7  | |         {
8  | |             grpcio_compiler::prost_codegen::compile_protos(
...  |
24 | |             .for_each(|path| WrapperGen::new(path, self.wrapper_opts).write());
25 | |     }
   | |_____- other definition for `generate_files`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0592`.
error: Could not compile `protobuf-build`.

To learn more, run the command again with --verbose.

It seems that client-rust and tikv use different feature flag in kvproto import, and generate duplicate generate_files.

And we cannot remove features = [ "prost-codec" ] in client-rust, because it will cause another error:

error[E0599]: no method named `ok_or_else` found for type `protobuf::singular::SingularPtrField<kvproto::protos::metapb::Region>` in the current scope
  --> /Users/zhuangtianyi/client-rust/src/pd/cluster.rs:70:22
   |
70 |                     .ok_or_else(|| Error::region_for_key_not_found(key));
   |                      ^^^^^^^^^^

error[E0308]: mismatched types
  --> /Users/zhuangtianyi/client-rust/src/pd/cluster.rs:72:66
   |
72 |                 future::ready(region.map(move |r| Region::new(r, leader)))
   |                                                                  ^^^^^^ expected enum `std::option::Option`, found struct `protobuf::singular::SingularPtrField`
   |
   = note: expected type `std::option::Option<kvproto::protos::metapb::Peer>`
              found type `protobuf::singular::SingularPtrField<kvproto::protos::metapb::Peer>`

error[E0599]: no method named `ok_or_else` found for type `protobuf::singular::SingularPtrField<kvproto::protos::metapb::Region>` in the current scope
  --> /Users/zhuangtianyi/client-rust/src/pd/cluster.rs:99:42
   |
99 |                 let region = resp.region.ok_or_else(|| Error::region_not_found(id));
   |                                          ^^^^^^^^^^

error[E0308]: mismatched types
   --> /Users/zhuangtianyi/client-rust/src/pd/cluster.rs:101:66
    |
101 |                 future::ready(region.map(move |r| Region::new(r, leader)))
    |                                                                  ^^^^^^ expected enum `std::option::Option`, found struct `protobuf::singular::SingularPtrField`
    |
    = note: expected type `std::option::Option<kvproto::protos::metapb::Peer>`
               found type `protobuf::singular::SingularPtrField<kvproto::protos::metapb::Peer>`

error: aborting due to 4 previous errors

Some errors have detailed explanations: E0308, E0599.
For more information about an error, try `rustc --explain E0308`.
error: Could not compile `tikv-client`.

It seems that swithing the feature flag will cause different method signatures.

Is there any solution?

@nrc

UCP: support resolving locks of specific keys

Description

Normal resolve lock scans the whole region for locks. With resolve lock lite, we resolve only the specified locks. This can reduce TiKV's load when there are a lot of conflicts.

Task breakdown:

• Add an interface resolve_lock_specified_keys, the protocol is:

message ResolveLockRequest {
    Context context = 1;
    uint64 start_version  = 2;
    // If the txn is rolled back, do not set it.
    uint64 commit_version = 3;
    repeated TxnInfo txn_infos = 4;
    // Only resolve specified keys.
    repeated bytes keys = 5;
}

Set specified keys for ResolveLockRequest.

Difficulty

• Medium

Score

• 300

Mentor(s)

• @sticnarf

Recommended Skills

• Rust
• TiKV Transaction Model

Is the PdReactor's StreamingCallSink properly cancelled?

In the futures 0.1 versionof the client, in PdReactor::init in src/rpc/pd/leader.rs, we called sender.get_mut().get_ref().cancel(). However, after migrating to futures 0.3 (#41), we can't get mutable access to the underlying StreamingCallSink to call cancel. I think that the StreamingCallSink will be cancelled when _sender is dropped, but I could not confirm this.

Refactoring: remove Value type

Either just use byte[] or create an alias type Value = byte[]; since we do not enforce any additional invariants for Value over byte[] and to simplify the code. This will also require changes to the KeyValue types, etc.

cc #75 #99

Allow building integration tests without running them

As an alternative, we could check for the PD_ADDR env var and pass the test if it is not present (or ignore it if it is not present at compile time?).

My reasoning is that when working on the library, it is nice to check that the integration tests still compile, even if we don't run them. Given we don't run them by default today, I don't think passing by default will give a false sense of security.

@Hoverbear what do you think?

TSO implementation is broken

I try to get a number of timestamps asynchronously using futures::executor::ThreadPool, but failed.

Test code: https://github.com/sticnarf/tikv-client-rust/blob/enable-async-await/tests/integration_tests.rs

The code just calls the get_timestamp of RetryClient.

Notice that the code finally leads to an illegal instruction error.

thread 'dispatcher thread' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:374:21
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:214
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   8: rust_begin_unwind
             at src/libstd/panicking.rs:311
   9: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  10: core::panicking::panic
             at src/libcore/panicking.rs:49
  11: core::option::Option<T>::unwrap
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/macros.rs:12
  12: tikv_client::rpc::pd::timestamp::PdReactor::tso_request
             at src/rpc/pd/timestamp.rs:183
  13: tikv_client::rpc::pd::timestamp::Task::dispatch
             at src/rpc/pd/timestamp.rs:49
  14: tikv_client::rpc::pd::timestamp::PdReactor::poll::{{closure}}
             at src/rpc/pd/timestamp.rs:133
  15: <futures_util::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next::{{closure}}::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/stream/map.rs:92
  16: core::option::Option<T>::map
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/option.rs:443
  17: <futures_util::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/stream/map.rs:92
  18: core::task::poll::Poll<T>::map
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/task/poll.rs:35
  19: <futures_util::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/stream/map.rs:89
  20: <futures_util::stream::collect::Collect<St,C> as core::future::future::Future>::poll
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/stream/collect.rs:48
  21: <futures_util::future::unit_error::UnitError<Fut> as core::future::future::Future>::poll
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/future/unit_error.rs:33
  22: <F as futures_core::future::TryFuture>::try_poll
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-preview-0.3.0-alpha.17/src/future/mod.rs:78
  23: <futures_util::compat::compat03as01::Compat<Fut> as futures::future::Future>::poll::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/compat/compat03as01.rs:106
  24: futures_util::compat::compat03as01::with_context
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/compat/compat03as01.rs:209
  25: <futures_util::compat::compat03as01::Compat<Fut> as futures::future::Future>::poll
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/compat/compat03as01.rs:106
  26: futures::task_impl::Spawn<T>::poll_future_notify::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:329
  27: futures::task_impl::Spawn<T>::enter::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
  28: futures::task_impl::std::set
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/std/mod.rs:83
  29: futures::task_impl::Spawn<T>::enter
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
  30: futures::task_impl::Spawn<T>::poll_fn_notify
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:291
  31: futures::task_impl::Spawn<T>::poll_future_notify
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:329
  32: tokio_core::reactor::Core::run::{{closure}}::{{closure}}::{{closure}}::{{closure}}::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:236
  33: futures::future::lazy::Lazy<F,R>::get
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/future/lazy.rs:64
  34: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/future/lazy.rs:82
  35: futures::task_impl::Spawn<T>::poll_future_notify::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:329
  36: futures::task_impl::Spawn<T>::enter::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
  37: futures::task_impl::std::set
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/std/mod.rs:83
  38: futures::task_impl::Spawn<T>::enter
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:399
  39: futures::task_impl::Spawn<T>::poll_fn_notify
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:291
  40: futures::task_impl::Spawn<T>::poll_future_notify
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.28/src/task_impl/mod.rs:329
  41: tokio_current_thread::Entered<P>::block_on::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:494
  42: tokio_current_thread::Borrow<U>::enter::{{closure}}::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
  43: tokio_current_thread::CurrentRunner::set_spawn
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:825
  44: tokio_current_thread::Borrow<U>::enter::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
  45: std::thread::local::LocalKey<T>::try_with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:257
  46: std::thread::local::LocalKey<T>::with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:234
  47: tokio_current_thread::Borrow<U>::enter
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:786
  48: tokio_current_thread::Entered<P>::block_on
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:491
  49: tokio_core::reactor::Core::run::{{closure}}::{{closure}}::{{closure}}::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:234
  50: tokio_timer::timer::handle::with_default::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:101
  51: std::thread::local::LocalKey<T>::try_with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:257
  52: std::thread::local::LocalKey<T>::with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:234
  53: tokio_timer::timer::handle::with_default
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:84
  54: tokio_core::reactor::Core::run::{{closure}}::{{closure}}::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:233
  55: tokio_executor::global::with_default::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:209
  56: std::thread::local::LocalKey<T>::try_with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:257
  57: std::thread::local::LocalKey<T>::with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:234
  58: tokio_executor::global::with_default
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:178
  59: tokio_core::reactor::Core::run::{{closure}}::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:232
  60: tokio_reactor::with_default::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.9/src/lib.rs:237
  61: std::thread::local::LocalKey<T>::try_with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:257
  62: std::thread::local::LocalKey<T>::with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:234
  63: tokio_reactor::with_default
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.9/src/lib.rs:217
  64: tokio_core::reactor::Core::run::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:231
  65: scoped_tls::ScopedKey<T>::set
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/scoped-tls-0.1.2/src/lib.rs:155
  66: tokio_core::reactor::Core::run
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:230
  67: tikv_client::rpc::pd::timestamp::PdReactor::poll
             at src/rpc/pd/timestamp.rs:136
  68: tikv_client::rpc::pd::timestamp::PdReactor::start::{{closure}}
             at src/rpc/pd/timestamp.rs:110
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'get_timestamp' panicked at 'called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"', src/libcore/result.rs:1051:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:214
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   8: rust_begin_unwind
             at src/libstd/panicking.rs:311
   9: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  10: core::result::unwrap_failed
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/macros.rs:18
  11: core::result::Result<T,E>::unwrap
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/result.rs:852
  12: tikv_client::rpc::pd::client::Cluster::get_timestamp
             at src/rpc/pd/client.rs:259
  13: <tikv_client::rpc::pd::client::RetryClient as tikv_client::rpc::pd::client::PdClient>::get_timestamp
             at src/rpc/pd/client.rs:111
  14: tikv_client::rpc::client::RpcClient<PdC>::get_timestamp
             at ./src/rpc/client.rs:229
  15: tikv_client::transaction::client::Client::current_timestamp::{{closure}}
             at ./src/transaction/client.rs:103
  16: <std::future::GenFuture<T> as core::future::future::Future>::poll::{{closure}}
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:42
  17: std::future::set_task_context
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:78
  18: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:42
  19: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-preview-0.3.0-alpha.17/src/future/join_all.rs:136
  20: std::future::poll_with_tls_context::{{closure}}
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:120
  21: std::future::get_task_context
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:110
  22: std::future::poll_with_tls_context
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:120
  23: integration_tests::get_timestamp::{{closure}}
             at tests/integration_tests.rs:17
  24: <std::future::GenFuture<T> as core::future::future::Future>::poll::{{closure}}
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:42
  25: std::future::set_task_context
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:78
  26: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:42
  27: futures_executor::local_pool::LocalPool::run_until::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:156
  28: futures_executor::local_pool::run_executor::{{closure}}
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:68
  29: std::thread::local::LocalKey<T>::try_with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:257
  30: std::thread::local::LocalKey<T>::with
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:234
  31: futures_executor::local_pool::run_executor
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:64
  32: futures_executor::local_pool::LocalPool::run_until
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:153
  33: futures_executor::thread_pool::ThreadPool::run
             at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/thread_pool.rs:96
  34: integration_tests::get_timestamp
             at tests/integration_tests.rs:20
  35: integration_tests::get_timestamp::{{closure}}
             at tests/integration_tests.rs:11
  36: core::ops::function::FnOnce::call_once
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ops/function.rs:231
  37: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/boxed.rs:766
  38: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:82
  39: std::panicking::try
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panicking.rs:275
  40: std::panic::catch_unwind
             at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panic.rs:394
  41: test::run_test::run_test_inner::{{closure}}
             at src/libtest/lib.rs:1466
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'get_timestamp' panicked at 'called `Result::unwrap()` on an `Err` value: Any', src/libcore/result.rs:1051:5
stack backtrace:
   0:     0x560f7490c45b - backtrace::backtrace::libunwind::trace::hf65880216e481eb8
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
   1:     0x560f7490c45b - backtrace::backtrace::trace_unsynchronized::h7adaf2661c0eaf8b
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
   2:     0x560f7490c45b - std::sys_common::backtrace::_print::h05083ada7af0ba89
                               at src/libstd/sys_common/backtrace.rs:47
   3:     0x560f7490c45b - std::sys_common::backtrace::print::h244aa103cfd055e5
                               at src/libstd/sys_common/backtrace.rs:36
   4:     0x560f7490c45b - std::panicking::default_hook::{{closure}}::hd67352ee5eb6d31f
                               at src/libstd/panicking.rs:200
   5:     0x560f7490c137 - std::panicking::default_hook::hf4133fce7d0b47a5
                               at src/libstd/panicking.rs:214
   6:     0x560f7490cbd0 - std::panicking::rust_panic_with_hook::hedad13c54bf0b93b
                               at src/libstd/panicking.rs:477
   7:     0x560f7490c752 - std::panicking::continue_panic_fmt::h8d5b6543a65fc881
                               at src/libstd/panicking.rs:384
   8:     0x560f7490c636 - rust_begin_unwind
                               at src/libstd/panicking.rs:311
   9:     0x560f74927afd - core::panicking::panic_fmt::h0a7fdb4854b8ba94
                               at src/libcore/panicking.rs:85
  10:     0x560f73fddd6a - core::result::unwrap_failed::h400636b1d63e8552
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/macros.rs:18
  11:     0x560f73fe190f - core::result::Result<T,E>::unwrap::h401b5f16c0f532a6
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/result.rs:852
  12:     0x560f73fef08d - <tikv_client::rpc::pd::timestamp::PdReactor as core::ops::drop::Drop>::drop::h9c40459429be85ec
                               at src/rpc/pd/timestamp.rs:77
  13:     0x560f73fea085 - core::ptr::real_drop_in_place::h90d9cae9e39efdf1
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  14:     0x560f73fec26e - core::ptr::real_drop_in_place::hccd54b444b4b0224
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  15:     0x560f73fea458 - core::ptr::real_drop_in_place::h937c5ddaffb7bf12
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  16:     0x560f73fd0042 - core::ptr::drop_in_place::h064a84126d862944
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:165
  17:     0x560f73fd0042 - alloc::sync::Arc<T>::drop_slow::hdbffda19a9c900f5
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/sync.rs:560
  18:     0x560f73fd07e8 - <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop::h20ae927fb0fbb7e9
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/sync.rs:1049
  19:     0x560f73fe7dbf - core::ptr::real_drop_in_place::h43cf29a770682c9b
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  20:     0x560f73fe9e20 - core::ptr::real_drop_in_place::h8b430d9fa5d4d507
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  21:     0x560f73fe72be - core::ptr::real_drop_in_place::h32b400447de9e1ff
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  22:     0x560f73fec7f8 - core::ptr::real_drop_in_place::hd9cc67f6c32a25f1
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  23:     0x560f73feb4d1 - core::ptr::real_drop_in_place::hb41be26baee64591
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  24:     0x560f73fcf782 - core::ptr::drop_in_place::h7b4912ca777e560a
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:165
  25:     0x560f73fcf782 - alloc::sync::Arc<T>::drop_slow::h06c052ada6d6f04f
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/sync.rs:560
  26:     0x560f73fd0a88 - <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop::h5307e8092e46d756
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/sync.rs:1049
  27:     0x560f73feb08f - core::ptr::real_drop_in_place::hab609ff0b4325e7f
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  28:     0x560f73fec061 - core::ptr::real_drop_in_place::hc9d59f1c0519669b
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  29:     0x560f73fd0122 - core::ptr::drop_in_place::h121928ddee85ec51
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:165
  30:     0x560f73fd0122 - alloc::sync::Arc<T>::drop_slow::hdc2861db02af1cce
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/sync.rs:560
  31:     0x560f73fd0d28 - <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop::hedbb8ef45e377785
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/sync.rs:1049
  32:     0x560f73f79abf - core::ptr::real_drop_in_place::h6aa4bbd7ada6601a
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  33:     0x560f73f79d4e - core::ptr::real_drop_in_place::h8d2b3d5580b34c9a
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ptr/mod.rs:175
  34:     0x560f73f85f39 - integration_tests::get_timestamp::{{closure}}::h1fe7978d7efe5c52
                               at tests/integration_tests.rs:18
  35:     0x560f73f8b7eb - <std::future::GenFuture<T> as core::future::future::Future>::poll::{{closure}}::hd04596d0c0d323ba
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:42
  36:     0x560f73f8b39b - std::future::set_task_context::h69a3cd26620124f6
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:78
  37:     0x560f73f8b77a - <std::future::GenFuture<T> as core::future::future::Future>::poll::hff766850b0ebc795
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/future.rs:42
  38:     0x560f73f8be67 - futures_executor::local_pool::LocalPool::run_until::{{closure}}::hc6766659662e3666
                               at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:156
  39:     0x560f73f8bc1a - futures_executor::local_pool::run_executor::{{closure}}::h631e929b1c618d80
                               at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:68
  40:     0x560f73f8a490 - std::thread::local::LocalKey<T>::try_with::ha6eb0af014dc1a3c
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:257
  41:     0x560f73f89ee4 - std::thread::local::LocalKey<T>::with::h9c967060bf815f24
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/local.rs:234
  42:     0x560f73f8bb21 - futures_executor::local_pool::run_executor::h2658d591d67cb916
                               at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:64
  43:     0x560f73f8bda4 - futures_executor::local_pool::LocalPool::run_until::h7ec92cc6e6f64e22
                               at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/local_pool.rs:153
  44:     0x560f73f88670 - futures_executor::thread_pool::ThreadPool::run::h4e432f4e91316b04
                               at /home/yilin/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-preview-0.3.0-alpha.17/src/thread_pool.rs:96
  45:     0x560f73f7ce98 - integration_tests::get_timestamp::h16ebc5ec3dd75dfe
                               at tests/integration_tests.rs:20
  46:     0x560f73f85ade - integration_tests::get_timestamp::{{closure}}::h979f3fdd7f626849
                               at tests/integration_tests.rs:11
  47:     0x560f73f78c5e - core::ops::function::FnOnce::call_once::hc98ec6a96fb0fc64
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ops/function.rs:231
  48:     0x560f73f9c0df - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::hdd2839af31295b4f
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/boxed.rs:766
  49:     0x560f7490feca - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:82
  50:     0x560f73fb69e8 - std::panicking::try::h896452ec5805b861
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panicking.rs:275
  51:     0x560f73fb69e8 - std::panic::catch_unwind::h46170accf265e731
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panic.rs:394
  52:     0x560f73fb69e8 - test::run_test::run_test_inner::{{closure}}::h5a54d74ef3d6835a
                               at src/libtest/lib.rs:1466
  53:     0x560f73f91495 - std::sys_common::backtrace::__rust_begin_short_backtrace::hcf834ea9ccbacdf6
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/sys_common/backtrace.rs:77
  54:     0x560f73f95585 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h18081e1df3cce1ca
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/mod.rs:470
  55:     0x560f73f95585 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::he44de7f021bb3f7c
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panic.rs:315
  56:     0x560f73f95585 - std::panicking::try::do_call::h18151d20bdb933b9
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panicking.rs:296
  57:     0x560f7490feca - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:82
  58:     0x560f73f95c32 - std::panicking::try::h88d44f82c87a9dac
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panicking.rs:275
  59:     0x560f73f95c32 - std::panic::catch_unwind::hb6da1a08a6b4cbc3
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/panic.rs:394
  60:     0x560f73f95c32 - std::thread::Builder::spawn_unchecked::{{closure}}::h00f1b8952678c6d8
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libstd/thread/mod.rs:469
  61:     0x560f73f95c32 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h878a2a58f646bd5c
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/libcore/ops/function.rs:231
  62:     0x560f749015bf - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h5b2090eaf14d13e6
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/boxed.rs:766
  63:     0x560f7490f220 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::he0ec03f2ca3672cd
                               at /rustc/4b65a86ebace8600c8e269e8bfe3365cdc460e68/src/liballoc/boxed.rs:766
  64:     0x560f7490f220 - std::sys_common::thread::start_thread::h79a82acf349ba96a
                               at src/libstd/sys_common/thread.rs:13
  65:     0x560f7490f220 - std::sys::unix::thread::Thread::new::thread_start::ha5923a049bad6775
                               at src/libstd/sys/unix/thread.rs:79
  66:     0x7f57a7be357f - start_thread
  67:     0x7f57a7af70e3 - __clone
  68:                0x0 - <unknown>
thread panicked while panicking. aborting.
error: process didn't exit successfully: `/home/yilin/Projects/tikv-client-rust/target/debug/deps/integration_tests-2d38dae8e43a0c9f get_timestamp --nocapture` (signal: 4, SIGILL: illegal instruction)

UCP: Add backoff and retry limit to requests

Description

We shouldn't allow infinite or immediate retries. It is better to add backoff retry delay.

Difficulty

• Easy

Score

• 50

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Rust

Examples with command line args

In #14 we talked about adding command line parameters to the example so users can pass in their own PD addresses.

Use clap to do this, please only add it to dev-dependencies.

Key and Value should be coercible from &[u8]

Since we can coerce from both Vec<u8> and &str. This will require some new From impls in kv.rs.

UCP: Implement RawBatchScan

Description

Implement the RawBatchScan interface. The protocol is:

message RawBatchScanRequest {
    Context context = 1;
    repeated KeyRange ranges = 2; // scanning range
    uint32 each_limit = 3; // max number of returning kv pairs for each scanning range
    bool key_only = 4;
    string cf = 5;
    bool reverse = 6;
}

message RawBatchScanResponse {
    errorpb.Error region_error = 1;
    repeated KvPair kvs = 2;
}

Difficulty

• Easy

Score

• 50

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Rust

Add nightly CI proptest job

Add a nightly Travis CI proptest job.

Retry when getting a timestamp fails

client-rust/src/rpc/pd/client.rs

Line 111 in 2733e28

// FIXME: retry or reconnect on error

The RetryClient currently does not retry get_timestamp.

Scan() command ignores start_key and end_key

I ran the raw API example code on a fresh cluster and I get an assertion error here:
https://github.com/tikv/client-rust/blob/master/examples/raw.rs#L95-L97

Put key "TiKV", value "Rust".
Get key `TiKV` returned value Some(Value("Rust")).
Key: `TiKV` deleted
Found values: [KvPair(6B31, "v1"), KvPair(6B32, "v2")] for keys: [Key(6B31), Key(6B32)]
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[Key(6B31), Key(6B32), Key(6B33)]`,
 right: `[Key(6B31), Key(6B32)]`', src/main.rs:101:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

So I added two println! in src/raw/requests.rs to get more info about when those start and end keys where lost, here is the output:

Put key "TiKV", value "Rust".
Get key `TiKV` returned value Some(Value("Rust")).
Key: `TiKV` deleted
Found values: [KvPair(6B31, "v1"), KvPair(6B32, "v2")] for keys: [Key(6B31), Key(6B32)]
>>> in new_raw_scan_request : req=RawScanRequest { context: None, start_key: [107, 49], limit: 10, key_only: true, cf: "", reverse: false, end_key: [107, 50, 0] }
>>> in make_rpc_request: req=RawScanRequest { context: Some(Context { region_id: 2, region_epoch: Some(RegionEpoch { conf_ver: 5, version: 1 }), peer: Some(Peer { id: 7, store_id: 5, is_learner: false }), term: 0, priority: Normal, isolation_level: Si, not_fill_cache: false, sync_log: false, handle_time: false, scan_detail: false, replica_read: false, resolved_locks: [], max_execution_duration_ms: 0, applied_index: 0 }), start_key: [], limit: 10, key_only: true, cf: "", reverse: false, end_key: [] }
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[Key(6B31), Key(6B32), Key(6B33)]`,
 right: `[Key(6B31), Key(6B32)]`', src/main.rs:101:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

We can see that in the first object the start_key, end_key, and limit fields are correctly set, but in the second object, start_key and end_key became empty but limit kept its correct value (10).

Can not compile successfully after adding tikv-client dependency in `Cargo.toml`

I want to use this crate in my project, and I have added tikv-client = { git = "https://github.com/tikv/client-rust.git" } in my Cargo.toml. However, when I compile, it outputs a lot of errors. Can you tell me how to fix this? Thanks.

The errors are as follows:

error[E0432]: unresolved import `kvproto::pdpb::PdClient`
  --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/client.rs:12:21
   |
12 | use kvproto::{pdpb, pdpb::PdClient as RpcClient};
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `PdClient` in `pdpb`

error[E0432]: unresolved import `kvproto::tikvpb::TikvClient`
  --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/tikv/client.rs:11:33
   |
11 | use kvproto::{errorpb, kvrpcpb, tikvpb::TikvClient};
   |                                 ^^^^^^^^^^^^^^^^^^ no `TikvClient` in `tikvpb`

error[E0433]: failed to resolve: could not find `PdClient` in `pdpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/leader.rs:359:56
    |
359 |     let client = security_mgr.connect(env, addr, pdpb::PdClient::new)?;
    |                                                        ^^^^^^^^ could not find `PdClient` in `pdpb`

error[E0412]: cannot find type `PdClient` in module `pdpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/leader.rs:225:23
    |
225 |     pub client: pdpb::PdClient,
    |                       ^^^^^^^^ not found in `pdpb`
help: possible candidates are found in other modules, you can import them into scope
    |
3   | use crate::rpc::pd::client::PdClient;
    |
3   | use kvproto::pdpb_grpc::PdClient;
    |

error[E0412]: cannot find type `PdClient` in module `pdpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/leader.rs:304:20
    |
304 | ) -> Result<(pdpb::PdClient, pdpb::GetMembersResponse)> {
    |                    ^^^^^^^^ not found in `pdpb`
help: possible candidates are found in other modules, you can import them into scope
    |
3   | use crate::rpc::pd::client::PdClient;
    |
3   | use kvproto::pdpb_grpc::PdClient;
    |

error[E0412]: cannot find type `PdClient` in module `pdpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/leader.rs:358:20
    |
358 | ) -> Result<(pdpb::PdClient, pdpb::GetMembersResponse)> {
    |                    ^^^^^^^^ not found in `pdpb`
help: possible candidates are found in other modules, you can import them into scope
    |
3   | use crate::rpc::pd::client::PdClient;
    |
3   | use kvproto::pdpb_grpc::PdClient;
    |

error[E0412]: cannot find type `PdClient` in module `pdpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/leader.rs:373:20
    |
373 | ) -> Result<(pdpb::PdClient, pdpb::GetMembersResponse)> {
    |                    ^^^^^^^^ not found in `pdpb`
help: possible candidates are found in other modules, you can import them into scope
    |
3   | use crate::rpc::pd::client::PdClient;
    |
3   | use kvproto::pdpb_grpc::PdClient;
    |

error[E0412]: cannot find type `PdClient` in module `pdpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/leader.rs:393:20
    |
393 | ) -> Result<(pdpb::PdClient, pdpb::GetMembersResponse)> {
    |                    ^^^^^^^^ not found in `pdpb`
help: possible candidates are found in other modules, you can import them into scope
    |
3   | use crate::rpc::pd::client::PdClient;
    |
3   | use kvproto::pdpb_grpc::PdClient;
    |

error[E0412]: cannot find type `GetRegionByIdRequest` in module `pdpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/pd/client.rs:104:58
    |
104 |         let mut req = pd_request!(self.cluster_id, pdpb::GetRegionByIdRequest);
    |                                                          ^^^^^^^^^^^^^^^^^^^^ help: a struct with a similar name exists: `GetRegionByIDRequest`

error[E0412]: cannot find type `GcResponse` in module `kvrpcpb`
  --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/tikv/client.rs:88:28
   |
88 | has_region_error!(kvrpcpb::GcResponse);
   |                            ^^^^^^^^^^ help: a struct with a similar name exists: `GCResponse`

error[E0412]: cannot find type `GcResponse` in module `kvrpcpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/tikv/client.rs:120:25
    |
120 | has_key_error!(kvrpcpb::GcResponse);
    |                         ^^^^^^^^^^ help: a struct with a similar name exists: `GCResponse`

error[E0412]: cannot find type `GcResponse` in module `kvrpcpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/tikv/client.rs:440:47
    |
440 |     ) -> impl Future<Output = Result<kvrpcpb::GcResponse>> {
    |                                               ^^^^^^^^^^ help: a struct with a similar name exists: `GCResponse`

error[E0412]: cannot find type `GcRequest` in module `kvrpcpb`
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/tikv/client.rs:441:54
    |
441 |         let mut req = txn_request!(context, kvrpcpb::GcRequest);
    |                                                      ^^^^^^^^^ help: a struct with a similar name exists: `GCRequest`

error[E0308]: mismatched types
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/tikv/client.rs:535:23
    |
535 |         req.set_pairs(Self::convert_to_grpc_pairs(pairs));
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `protobuf::repeated::RepeatedField`, found struct `std::vec::Vec`
    |
    = note: expected type `protobuf::repeated::RepeatedField<kvproto::protos::kvrpcpb::KvPair>`
               found type `std::vec::Vec<kvproto::protos::kvrpcpb::KvPair>`

error[E0308]: mismatched types
   --> /home/karn/.cargo/git/checkouts/client-rust-5a1ccd35a54db20f/c13f633/src/rpc/tikv/client.rs:618:24
    |
618 |         req.set_ranges(Self::convert_to_grpc_ranges(ranges));
    |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `protobuf::repeated::RepeatedField`, found struct `std::vec::Vec`
    |
    = note: expected type `protobuf::repeated::RepeatedField<kvproto::protos::kvrpcpb::KeyRange>`
               found type `std::vec::Vec<kvproto::protos::kvrpcpb::KeyRange>`

error: aborting due to 15 previous errors

Test `util::tests::test_global_timer` fails intermittently on CI

Example output

thread 'util::tests::test_global_timer' panicked at 'assertion failed: timer.elapsed() >= ::std::time::Duration::from_millis(100)', src/util.rs:72:9

stack backtrace:

   0: backtrace::backtrace::libunwind::trace

             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88

   1: backtrace::backtrace::trace_unsynchronized

             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66

   2: std::sys_common::backtrace::_print

             at src/libstd/sys_common/backtrace.rs:47

   3: std::sys_common::backtrace::print

             at src/libstd/sys_common/backtrace.rs:36

   4: std::panicking::default_hook::{{closure}}

             at src/libstd/panicking.rs:200

   5: std::panicking::default_hook

             at src/libstd/panicking.rs:214

   6: std::panicking::rust_panic_with_hook

             at src/libstd/panicking.rs:477

   7: std::panicking::begin_panic

             at /rustc/dddb7fca09dc817ba275602b950bb81a9032fb6d/src/libstd/panicking.rs:411

   8: tikv_client::util::tests::test_global_timer

             at src/util.rs:72

   9: tikv_client::util::tests::test_global_timer::{{closure}}

             at src/util.rs:66

  10: core::ops::function::FnOnce::call_once

             at /rustc/dddb7fca09dc817ba275602b950bb81a9032fb6d/src/libcore/ops/function.rs:235

  11: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once

             at /rustc/dddb7fca09dc817ba275602b950bb81a9032fb6d/src/liballoc/boxed.rs:770

  12: __rust_maybe_catch_panic

             at src/libpanic_unwind/lib.rs:80

  13: std::panicking::try

             at /rustc/dddb7fca09dc817ba275602b950bb81a9032fb6d/src/libstd/panicking.rs:275

  14: std::panic::catch_unwind

             at /rustc/dddb7fca09dc817ba275602b950bb81a9032fb6d/src/libstd/panic.rs:394

  15: test::run_test::run_test_inner::{{closure}}

             at src/libtest/lib.rs:1471

note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Microbenchmark Prost vs Rust-protobuf

Maybe it can help us diagnose problems.

Test with current nightly when Clippy is distributed

We are currently locked to an older nightly so that we can run Clippy tests (in .travis.yml). We should unlock when Clippy is available again.

UCP: Add backoff and retry limit to lock resolving

Description

Now we use a fixed delay before resolving locks. It is better to add backoff retry delay and retry limit.

The task is to resolve the TODO here:

client-rust/src/request.rs

Line 65 in f3f3db3

// TODO: backoff

Difficulty

• Easy

Score

• 50

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Rust

cargo doc fails due to too deep recursion

cargo doc produces errors like this:

error[E0275]: overflow evaluating the requirement `alloc::raw_vec::RawVec<(syn::Lifetime, syn::token::Add)>: std::marker::Unpin`
  |
  = help: consider adding a `#![recursion_limit="128"]` attribute to your crate
  = note: required because it appears within the type `std::vec::Vec<(syn::Lifetime, syn::token::Add)>`
  = note: required because it appears within the type `syn::punctuated::Punctuated<syn::Lifetime, syn::token::Add>`
  = note: required because it appears within the type `syn::LifetimeDef`
  = note: required because it appears within the type `(syn::LifetimeDef, syn::token::Comma)`
  = note: required because it appears within the type `*const (syn::LifetimeDef, syn::token::Comma)`
  = note: required because it appears within the type `std::ptr::Unique<(syn::LifetimeDef, syn::token::Comma)>`
  = note: required because it appears within the type `alloc::raw_vec::RawVec<(syn::LifetimeDef, syn::token::Comma)>`
  = note: required because it appears within the type `std::vec::Vec<(syn::LifetimeDef, syn::token::Comma)>`
  = note: required because it appears within the type `syn::punctuated::Punctuated<syn::LifetimeDef, syn::token::Comma>`
  = note: required because it appears within the type `syn::BoundLifetimes`
  = note: required because it appears within the type `std::option::Option<syn::BoundLifetimes>`
  = note: required because it appears within the type `syn::PredicateType`
  = note: required because it appears within the type `syn::WherePredicate`
  = note: required because it appears within the type `(syn::WherePredicate, syn::token::Comma)`
  = note: required because it appears within the type `*const (syn::WherePredicate, syn::token::Comma)`
  = note: required because it appears within the type `std::ptr::Unique<(syn::WherePredicate, syn::token::Comma)>`
  = note: required because it appears within the type `alloc::raw_vec::RawVec<(syn::WherePredicate, syn::token::Comma)>`
  = note: required because it appears within the type `std::vec::Vec<(syn::WherePredicate, syn::token::Comma)>`
  = note: required because it appears within the type `syn::punctuated::Punctuated<syn::WherePredicate, syn::token::Comma>`
  = note: required because it appears within the type `syn::WhereClause`
  = note: required because it appears within the type `std::option::Option<syn::WhereClause>`
  = note: required because it appears within the type `syn::Generics`
  = note: required because it appears within the type `syn::ItemType`
  = note: required because it appears within the type `syn::Item`
  = note: required because it appears within the type `syn::Stmt`
  = note: required because it appears within the type `*const syn::Stmt`
  = note: required because it appears within the type `std::ptr::Unique<syn::Stmt>`
  = note: required because it appears within the type `alloc::raw_vec::RawVec<syn::Stmt>`
  = note: required because it appears within the type `std::vec::Vec<syn::Stmt>`
  = note: required because it appears within the type `syn::Block`
  = note: required because it appears within the type `syn::ExprIf`
  = note: required because it appears within the type `syn::Expr`
  = note: required because it appears within the type `syn::TypeArray`
  = note: required because it appears within the type `syn::Type`
  = note: required because it appears within the type `syn::GenericArgument`
  = note: required because it appears within the type `(syn::GenericArgument, syn::token::Comma)`
  = note: required because it appears within the type `*const (syn::GenericArgument, syn::token::Comma)`
  = note: required because it appears within the type `std::ptr::Unique<(syn::GenericArgument, syn::token::Comma)>`
  = note: required because it appears within the type `alloc::raw_vec::RawVec<(syn::GenericArgument, syn::token::Comma)>`
  = note: required because it appears within the type `std::vec::Vec<(syn::GenericArgument, syn::token::Comma)>`
  = note: required because it appears within the type `syn::punctuated::Punctuated<syn::GenericArgument, syn::token::Comma>`
  = note: required because it appears within the type `syn::AngleBracketedGenericArguments`
  = note: required because it appears within the type `syn::PathArguments`
  = note: required because it appears within the type `syn::PathSegment`
  = note: required because it appears within the type `(syn::PathSegment, syn::token::Colon2)`
  = note: required because it appears within the type `*const (syn::PathSegment, syn::token::Colon2)`
  = note: required because it appears within the type `std::ptr::Unique<(syn::PathSegment, syn::token::Colon2)>`
  = note: required because it appears within the type `alloc::raw_vec::RawVec<(syn::PathSegment, syn::token::Colon2)>`
  = note: required because it appears within the type `std::vec::Vec<(syn::PathSegment, syn::token::Colon2)>`
  = note: required because it appears within the type `syn::punctuated::Punctuated<syn::PathSegment, syn::token::Colon2>`
  = note: required because it appears within the type `syn::Path`
  = note: required because it appears within the type `syn::Attribute`
  = note: required because it appears within the type `*const syn::Attribute`
  = note: required because it appears within the type `std::ptr::Unique<syn::Attribute>`
  = note: required because it appears within the type `alloc::raw_vec::RawVec<syn::Attribute>`
  = note: required because it appears within the type `std::vec::Vec<syn::Attribute>`
  = note: required because it appears within the type `syn::TypeParam`
  = note: required because it appears within the type `syn::GenericParam`
  = note: required because it appears within the type `*const syn::GenericParam`
  = note: required because it appears within the type `std::ptr::Unique<syn::GenericParam>`
  = note: required because it appears within the type `alloc::raw_vec::RawVec<syn::GenericParam>`
  = note: required because it appears within the type `std::vec::Vec<syn::GenericParam>`
  = note: required because it appears within the type `Structure<'a>`

Related to rust-lang/rust#62059. It is solved by rust-lang/rust#62450. So simply upgrading the compiler can solve the problem

Is `ColumnFamily::default()` reasonable?

In raw::Client based requests there is an optional cf(...) value. Most requests contain a Option<ColumnFamily>. Since by default requests should be for the default ColumnFamily we may be able to impl Default for ColumnFamily and only hold a ColumnFamily instead.

UCP: Implement transactional scan

Description

The scan interface is not implemented for transactions. The project is to implement this interface. The protocol is:

message ScanRequest {
    Context context = 1;
    bytes start_key = 2;
    uint32 limit = 3;
    uint64 version = 4;
    bool key_only = 5;
    bool reverse = 6;
    // For compatibility, when scanning forward, the range to scan is [start_key, end_key), where start_key < end_key;
    // and when scanning backward, it scans [end_key, start_key) in descending order, where end_key < start_key.
    bytes end_key = 7;
}

message KvPair {
    KeyError error = 1;
    bytes key = 2;
    bytes value = 3;
}

message ScanResponse {
    errorpb.Error region_error = 1;
    repeated KvPair pairs = 2;
}

Difficulty

• Easy

Score

• 50

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Rust

Transactional API

In #14 we enabled the Raw client, and scaffolded much of the code needed for the transactional client. This issue tracks the completion of the first release of the Transactional API.

In order to close this issue you'll need to replace all unimplemented!() in the src/transaction.rs file with working code. You can reference the src/raw.rs code to help with this, however transactions are quite a bit more complicated than the raw client.

Deciding whether a key exists by the value's emptiness is inaccurate

It is actually possible and reasonable to put an empty value into TiKV. Then, we cannot distinguish an empty value from a non-existent key.

After tikv/tikv#5240, TiKV returns not_found field in raw_get and kv_get. Then, we can rely on this field to decide whether the key exists. However, there is a problem that this change is not backported to TiKV release 3.0.

Before the next major release of TiKV comes out, BatchGet may be a good alternative. Non-existent keys are not included in the returned pair array. And after the next major release of TiKV, maybe we can release a new version of this client which allows breaking changes (like 0.1 to 0.2), and switch to using the point get APIs.

Using regular expression in `trim_start_matches`

Can we use a regex here? Like https?://

#40 (diff)

Originally posted by @ice1000 in #40

Question: Converting &[u8] to Key

Most of the existing key-val interface take key/val as &[u8] or &str. how do I convert to Key/Val without cloning the data?

Transaction Drop Behavior

As part of #15, one question to consider is how to handle it when a transaction is dropped?

Transaction should have #[must_use] on it to create a warning for users who might accidentally drop it, but in the case of something like let _ = client.begin() we should consider if it is appropriate to just drop the memory. Should we attempt to rollback?

Refactoring: replace ColumnFamily type with an enum

Only a few distinguished names are accepted as column families, so we should make those an enum: Default, Lock, Write, Version ("versioncf").

Add error handling to KvRequest::execute

Firstly, we have to check the errors in the returned RPC response. The simpliest approach is propagating the error.

To improve user experience, we'd better retry inside the library. Especially when there is a region error, we should just get latest region info from PD and resend the request.

`raw_batch_scan` unimplemented

I have a hacky implementation on this branch. It also changes the API from returning a flattened Vec<KvPair> to Vec<Vec<KvPair>> (which adds to the "hackiness" because TiKV returns it as a flattened vector).
This fit my particular use case better, but I'm not sure it is desirable in general.

Are there plans to implement this feature? I could clean up my implementation and open a pull request.

build failed with kvproto

here is the error message

error: no matching package named `protobuf` found
location searched: https://github.com/nrc/rust-protobuf?branch=v2.8
perhaps you meant: protoc
required by package `kvproto v0.0.2 (https://github.com/pingcap/kvproto.git?rev=2fc6229ed6097b59dbe51525c7a65b19543a94ca#2fc6229e)`
    ... which is depended on by `tikv-client v0.0.0 (/data/git/rust/ping_cap/client-rust)`

found it can build after changing

kvproto = { git = "https://github.com/pingcap/kvproto.git", rev = "2fc6229ed6097b59dbe51525c7a65b19543a94ca", features = [ "prost-codec" ], default-features = false }

to

kvproto = { git = "https://github.com/pingcap/kvproto.git", features = [ "prost-codec" ], default-features = false }

maybe it needs a new rev

Should we allow empty values?

cc #39

They are not permitted to be stored in TiKV, so we could prevent them ever being created. I'm not sure if TiKV ever returns them.

I'm not sure how to make this ergonomic - we could dynamically check easily enough, but a panic is not very friendly (especially if it is not the user's fault), but returning Result from every function which creates a Value is not great either.

whether rust client is available in working_on_pd branch ?

whether rust client is available in working_on_pd branch ?

In rpcclient the `raw_scan` requires bounds

In #25 we found we are taking Option<_> for start, but it must be set. So we can make it always take a value instead.

This can be resolved by changing src/rpc/tikv/client.rs's raw_scan function to accept end_key as Key instead of Option<Key>

Empty values in raw client

I noticed the raw client considers it an error if you put (or batch_put) with an empty value. Is this required by tikv? And if not, can that restriction be removed from the client?

I have a need for writing empty values. I can work around this by just appending a 0x00 on every value, and stripping it out on read. But it would be nice to not have to do this.

Add into_inner function to Key, Value and KvPair

It provides a way of getting the ownership of the inner Vec. It saves one clone when we need the ownership of the data.

An alternative is to implement From<Key> and From<Value> for Vec<u8>. But I prefer the into_inner way.

By the way, is [u8] a better choice for the deref target of Key and Value? [u8] is just a more common deref target in the Rust world. Vec is ok but less comfortable to me and does no good too (I think no one cares about the only addition capacity). This is really of no significance, so I do not open a new issue.

Refactoring: extract low-level kv and pd client crates

For better separation of concerns, and to let users have different options for the abstraction level they wish to use to interact with TiKV.

Tasks:

• convert the project to a Cargo workspace rather than a single-crate project
• extract the kv_client module into a tikv-store-client crate - I think this should be relatively easy since the module is already self-contained
• extract a similar pd-client crate from the pd module - this is more complex since the module currently presents a higher-level API including lots of region-related stuff that should stay in the main crate. The low-level crate should be smaller and basically just wrap the gRPC calls
• ensure that there are no direct dependencies from the main crate to gRPC or any of the protobuf crates (kvproto, raft-proto, etc)

Convert futures to async/await

We have a lot of async code now, but are still using some futures, converting them all (or as much as possible) would make the code clearer.

Should Key and Value implement Deref?

I think not since they are not pointers, nor are they collections deref'ing to slices. However, I'm not sure what the motivation for the implementations is.

UCP: Check if the PD server responds with less timestamps

Description

As discussed in #92 (comment), when the PD server returns less timestamps than requested, there will be more and more pending requests and the timestamp oracle will finally block forever.

For a robust client, we do need extra checks to detect it.

Difficulty

• Easy

Score

• 50

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Rust

cargo build --example raw failed

I just clone project, and use cargo build --example raw on master
output:

    Updating crates.io index
error: failed to select a version for `protobuf`.
    ... required by package `tikv-client v0.0.0 (/root/client-rust)`
versions that meet the requirements `~2.0` are: 2.0.6, 2.0.5, 2.0.4, 2.0.3, 2.0.2, 2.0.1, 2.0.0

all possible versions conflict with previously selected packages.

  previously selected package `protobuf v2.1.0`
    ... which is depended on by `protoc-rust v2.1.0`
    ... which is depended on by `protobuf-build v0.4.1`
    ... which is depended on by `kvproto v0.0.2 (https://github.com/pingcap/kvproto.git?branch=raft-0.5.0#1b8f70f3)`
    ... which is depended on by `tikv-client v0.0.0 (/root/client-rust)`

failed to select a version for `protobuf` which could resolve this conflict

Resolving blockers for publishing

This crate is intended to be published to the Crates.io registry to support our users easily (and correctly) depending on this client.

It will also enable future cargo install tikv-client uses for our command line tool (#17).

The current blockers:

• CI is red (fixed by #22 and #30)
• No README (fixed by #31)
• KVproto is a git based dependency (this is a hard blocker for publishing. pingcap/kvproto#280)

These issues prevent us from reaching a published 0.0.1 release.

UCP: reconnect to PD asynchronously

Description

Currently reconnecting to PD will block the event loop. It's better to do it asynchronously.

Fix https://github.com/tikv/client-rust/blob/master/src/pd/cluster.rs#L205

Difficulty

• Medium

Score

• 300

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Async Rust programming

Command Line Tool

This extends #16 in many ways.

This issue proposes we add a feature command-line which enables a binary (src/bin/client.rs) via the required-features field of [bin].

This binary would allow users to call, for example:

$ tikv-client --pd ${PD} raw set key value
$ tikv-client --pd ${PD} raw get key
value

Closing this issue starts by opening an RFC to define the calling conventions. Due to the size of the issue it's advisable to start with raw first, then add transactional later.

UCP: Add region cache for PD

Description

It costs too much if we query PD for region information in every request. We can save region information in cache and query if necessary (on region error or the region for the key is unknown)

It's a good to idea to use the TiDB implementation as a reference: https://github.com/pingcap/tidb/blob/master/store/tikv/region_cache.go

@nrc has a PR which gives an incomplete implementation: #121. You can continue that PR as well.

Difficulty

• Medium

Score

• 3000

Mentor(s)

• @sticnarf
• @nrc

Recommended Skills

• Rust
• TiKV Architecture

Adding tikv-client to brand new rust package won't build

Manjaro Linux, Release 19.0.2
rustc 1.42.0 (b8cedc004 2020-03-09)
cargo 1.42.0 (86334295e 2020-01-31)
cmake version 3.17.0

Steps to reproduce:

1. Create a new rust lib: cargo new foo --lib.
2. Add tikv-client under [dependencies] in Cargo.toml: tikv-client = { git = "https://github.com/tikv/client-rust.git" }
3. Run cargo build.

Build output:

   Compiling libc v0.2.68
   Compiling cfg-if v0.1.10
   Compiling proc-macro2 v1.0.10
   Compiling unicode-xid v0.2.0
   Compiling cc v1.0.50
   Compiling syn v1.0.17
   Compiling memchr v2.3.3
   Compiling autocfg v1.0.0
   Compiling log v0.4.8
   Compiling bytes v0.5.4
   Compiling lazy_static v1.4.0
   Compiling bitflags v1.2.1
   Compiling pkg-config v0.3.17
   Compiling either v1.5.3
   Compiling getrandom v0.1.14
   Compiling anyhow v1.0.28
   Compiling failure_derive v0.1.7
   Compiling ppv-lite86 v0.2.6
   Compiling rustc-demangle v0.1.16
   Compiling proc-macro2 v0.4.30
   Compiling byteorder v1.3.4
   Compiling slab v0.4.2
   Compiling unicode-segmentation v1.6.0
   Compiling remove_dir_all v0.5.2
   Compiling futures-core v0.3.4
   Compiling unicode-xid v0.1.0
   Compiling syn v0.15.44
   Compiling regex-syntax v0.6.17
   Compiling fixedbitset v0.2.0
   Compiling version_check v0.1.5
   Compiling multimap v0.8.1
   Compiling glob v0.3.0
   Compiling futures v0.1.29
   Compiling fnv v1.0.6
   Compiling proc-macro-nested v0.1.4
   Compiling futures-sink v0.3.4
   Compiling proc-macro-hack v0.5.15
   Compiling version_check v0.9.1
   Compiling itoa v0.4.5
   Compiling pin-utils v0.1.0-alpha.4
   Compiling futures-task v0.3.4
   Compiling futures-io v0.3.4
   Compiling pin-project-lite v0.1.4
   Compiling openssl v0.10.29
   Compiling fixedbitset v0.1.9
   Compiling bindgen v0.51.1
   Compiling foreign-types-shared v0.1.1
   Compiling peeking_take_while v0.1.2
   Compiling shlex v0.1.1
   Compiling smallvec v1.3.0
   Compiling httparse v1.3.4
   Compiling multimap v0.4.0
   Compiling native-tls v0.2.4
   Compiling rustc-hash v1.1.0
   Compiling matches v0.1.8
   Compiling same-file v1.0.6
   Compiling crc32fast v1.2.0
   Compiling openssl-probe v0.1.2
   Compiling try-lock v0.2.2
   Compiling protobuf v2.8.0
   Compiling serde v1.0.106
   Compiling percent-encoding v2.1.0
   Compiling tower-service v0.3.0
   Compiling encoding_rs v0.8.22
   Compiling take_mut v0.2.2
   Compiling rle-decode-fast v1.0.1
   Compiling dtoa v0.4.5
   Compiling adler32 v1.0.4
   Compiling mime v0.3.16
   Compiling base64 v0.11.0
   Compiling hex v0.4.2
   Compiling prometheus v0.8.0
   Compiling spin v0.5.2
   Compiling futures-timer v3.0.2
   Compiling cmake v0.1.42
   Compiling thread_local v1.0.1
   Compiling indexmap v1.3.2
   Compiling itertools v0.8.2
   Compiling backtrace-sys v0.1.35
   Compiling openssl-sys v0.9.55
   Compiling libloading v0.5.2
   Compiling libz-sys v1.0.25
   Compiling heck v0.3.1
   Compiling nom v4.2.3
   Compiling clang-sys v0.28.1
   Compiling futures-channel v0.3.4
   Compiling unicase v2.6.0
   Compiling http v0.2.1
   Compiling petgraph v0.4.13
   Compiling foreign-types v0.3.2
   Compiling unicode-normalization v0.1.12
   Compiling unicode-bidi v0.3.4
   Compiling walkdir v2.3.1
   Compiling idna v0.2.0
   Compiling quote v1.0.3
   Compiling iovec v0.1.4
   Compiling which v3.1.1
   Compiling num_cpus v1.12.0
   Compiling net2 v0.2.33
   Compiling time v0.1.42
   Compiling http-body v0.3.1
   Compiling want v0.3.0
   Compiling aho-corasick v0.7.10
   Compiling quote v0.6.13
   Compiling libflate v0.1.27
   Compiling url v2.1.1
   Compiling bytes v0.4.12
   Compiling rand_core v0.5.1
   Compiling prost-build v0.6.1
   Compiling mio v0.6.21
   Compiling petgraph v0.5.0
   Compiling regex v1.3.6
   Compiling backtrace v0.3.46
   Compiling mime_guess v2.0.3
   Compiling cexpr v0.3.6
   Compiling procfs v0.7.8
   Compiling serde_urlencoded v0.6.1
   Compiling rand_chacha v0.2.2
   Compiling tokio v0.2.16
   Compiling synstructure v0.12.3
   Compiling prost-derive v0.6.1
   Compiling derive-new v0.5.8
   Compiling futures-macro v0.3.4
   Compiling pin-project-internal v0.4.8
   Compiling thiserror-impl v1.0.14
   Compiling serde_derive v1.0.106
   Compiling rand v0.7.3
   Compiling tokio-util v0.3.1
   Compiling tokio-tls v0.3.0
   Compiling futures-util v0.3.4
   Compiling prost v0.6.1
   Compiling thiserror v1.0.14
   Compiling tempfile v3.1.0
   Compiling pin-project v0.4.8
   Compiling failure v0.1.7
   Compiling prost-types v0.6.1
   Compiling which v2.0.1
   Compiling prost-derive v0.5.0
   Compiling h2 v0.2.4
   Compiling futures-executor v0.3.4
   Compiling prost-build v0.5.0
   Compiling futures v0.3.4
   Compiling grpcio-compiler v0.5.0
   Compiling hyper v0.13.4
   Compiling prost v0.5.0
   Compiling protobuf-build v0.11.1
   Compiling grpcio-sys v0.5.2
   Compiling prost-types v0.5.0
   Compiling hyper-tls v0.4.1
   Compiling protobuf-build v0.8.0
   Compiling kvproto v0.0.2 (https://github.com/pingcap/kvproto.git#05af14db)
   Compiling reqwest v0.10.4
   Compiling raft-proto v0.6.0-alpha
   Compiling grpcio-sys v0.5.2
   Compiling grpcio v0.5.1
   Compiling kvproto v0.0.2 (https://github.com/pingcap/kvproto.git#05af14db)
error[E0277]: the trait bound `raft_proto::protos::eraftpb::Entry: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/debugpb.rs:22:28
    |
22  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Entry`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:914:12
    |
914 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encode`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:11:28
    |
11  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Entry: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/debugpb.rs:22:28
    |
22  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Entry`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:929:12
    |
929 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::merge`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:11:28
    |
11  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Entry: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/debugpb.rs:22:28
    |
22  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Entry`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:975:12
    |
975 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encoded_len`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:11:28
    |
11  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Entry: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_cmdpb.rs:219:28
    |
219 | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Entry`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:914:12
    |
914 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encode`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:11:28
    |
11  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Entry: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_cmdpb.rs:219:28
    |
219 | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Entry`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:962:12
    |
962 |         M: Message + Default,
    |            ------- required by this bound in `prost::encoding::message::merge_repeated`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:11:28
    |
11  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Entry: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_cmdpb.rs:219:28
    |
219 | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Entry`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:984:12
    |
984 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encoded_len_repeated`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:11:28
    |
11  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Message: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_serverpb.rs:1:28
    |
1   | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Message`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:914:12
    |
914 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encode`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:48:28
    |
48  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Message: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_serverpb.rs:1:28
    |
1   | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Message`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:929:12
    |
929 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::merge`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:48:28
    |
48  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::Message: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_serverpb.rs:1:28
    |
1   | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::Message`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:975:12
    |
975 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encoded_len`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:48:28
    |
48  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::HardState: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_serverpb.rs:85:28
    |
85  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::HardState`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:914:12
    |
914 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encode`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:77:28
    |
77  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::HardState: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_serverpb.rs:85:28
    |
85  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::HardState`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:929:12
    |
929 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::merge`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:77:28
    |
77  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error[E0277]: the trait bound `raft_proto::protos::eraftpb::HardState: prost::message::Message` is not satisfied
   --> /home/dustin/code/repos/foo/target/debug/build/kvproto-c6c7a7e0753b46b4/out/protos/raft_serverpb.rs:85:28
    |
85  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ the trait `prost::message::Message` is not implemented for `raft_proto::protos::eraftpb::HardState`
    | 
   ::: /home/dustin/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/encoding.rs:975:12
    |
975 |         M: Message,
    |            ------- required by this bound in `prost::encoding::message::encoded_len`
    |
help: trait impl with same name found
   --> /home/dustin/code/repos/foo/target/debug/build/raft-proto-277624fcfd4979d6/out/protos/eraftpb.rs:77:28
    |
77  | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `prost` are being used?

error: aborting due to 12 previous errors

For more information about this error, try `rustc --explain E0277`.
error: could not compile `kvproto`.

I see that #32 is open, but it doesn't mention kvproto not building.