suharev7/clickhouse-rs

Asynchronous ClickHouse client library for Rust programming language.

clickhouse-rs's Issues

Nullable support published?

Hi,

I see that you have nullable support in master, but I don't think it's published yet? Do you know when you'll do that?

streaming

Hello!

I want to thank you again for writing this driver, it's been so helpful to me.

I also wanted to ask if you have any thoughts on streaming the results (i.e. by row).

Thanks again!

Can't parse Option<DateTime<Tz>>

I'm trying to parse

let date: Option<DateTime<Tz>> = result.get(0, "date").expect("invalid date value");

Receive error:

invalid date value: FromSql(InvalidType { src: "DateTime", dst: "$ t" })

without Option null value parsed as 1970-01-01 00:00:00 UTC

Return ClientHandle in case of recoverable errors

Currently the futures return only Error enums even in case of recoverable errors such as concurrent queries hitting the maximum allowed by the ClickHouse server. I propose that we can return the ClientHandle back to some Error variants that corresponds to recoverable errors.

Inserting same column multiple times causes server crash

I've noticed that if I add the same column to a block twice, it will crash the server with the following error:

2019.11.10 19:39:39.280113 [ 44 ] {} <Fatal> BaseDaemon: ########################################
2019.11.10 19:39:39.280191 [ 44 ] {} <Fatal> BaseDaemon: (version 19.16.2.2 (official build)) (from thread 43) Received signal Segmentation fault (11).
2019.11.10 19:39:39.280219 [ 44 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object.
2019.11.10 19:39:39.280244 [ 44 ] {} <Fatal> BaseDaemon: Stack trace: 0x5603c91637d6 0x5603c91cbbb6 0x5603c91d2913 0x5603c604569d 0x5603c604619b 0x5603c60462de 0x5603c604686e 0x5603c6046ae1 0x5603c6048115 0x5603c604839b 0x5603c9ee0fe0 0x5603c9ee16fd 0x5603cb5b78d1 0x5603cb5b567c 0x5603cbd271e0 0x7fbb86c376db 0x7fbb8655488f
2019.11.10 19:39:39.325630 [ 44 ] {} <Fatal> BaseDaemon: 3. 0x5603c91637d6 DB::Block::bytes() const /usr/bin/clickhouse
2019.11.10 19:39:39.325735 [ 44 ] {} <Fatal> BaseDaemon: 4. 0x5603c91cbbb6 DB::BlockStreamProfileInfo::update(DB::Block&) /usr/bin/clickhouse
2019.11.10 19:39:39.325753 [ 44 ] {} <Fatal> BaseDaemon: 5. 0x5603c91d2913 DB::IBlockInputStream::read() /usr/bin/clickhouse
2019.11.10 19:39:39.325774 [ 44 ] {} <Fatal> BaseDaemon: 6. 0x5603c604569d DB::TCPHandler::receiveData() /usr/bin/clickhouse
2019.11.10 19:39:39.325790 [ 44 ] {} <Fatal> BaseDaemon: 7. 0x5603c604619b DB::TCPHandler::receivePacket() /usr/bin/clickhouse
2019.11.10 19:39:39.325804 [ 44 ] {} <Fatal> BaseDaemon: 8. 0x5603c60462de DB::TCPHandler::readDataNext(unsigned long const&, int const&) /usr/bin/clickhouse
2019.11.10 19:39:39.325817 [ 44 ] {} <Fatal> BaseDaemon: 9. 0x5603c604686e DB::TCPHandler::readData(DB::Settings const&) /usr/bin/clickhouse
2019.11.10 19:39:39.325829 [ 44 ] {} <Fatal> BaseDaemon: 10. 0x5603c6046ae1 DB::TCPHandler::processInsertQuery(DB::Settings const&) /usr/bin/clickhouse
2019.11.10 19:39:39.325842 [ 44 ] {} <Fatal> BaseDaemon: 11. 0x5603c6048115 DB::TCPHandler::runImpl() /usr/bin/clickhouse
2019.11.10 19:39:39.325854 [ 44 ] {} <Fatal> BaseDaemon: 12. 0x5603c604839b DB::TCPHandler::run() /usr/bin/clickhouse
2019.11.10 19:39:39.325881 [ 44 ] {} <Fatal> BaseDaemon: 13. 0x5603c9ee0fe0 Poco::Net::TCPServerConnection::start() /usr/bin/clickhouse
2019.11.10 19:39:39.325901 [ 44 ] {} <Fatal> BaseDaemon: 14. 0x5603c9ee16fd Poco::Net::TCPServerDispatcher::run() /usr/bin/clickhouse
2019.11.10 19:39:39.325939 [ 44 ] {} <Fatal> BaseDaemon: 15. 0x5603cb5b78d1 Poco::PooledThread::run() /usr/bin/clickhouse
2019.11.10 19:39:39.325964 [ 44 ] {} <Fatal> BaseDaemon: 16. 0x5603cb5b567c Poco::ThreadImpl::runnableEntry(void*) /usr/bin/clickhouse
2019.11.10 19:39:39.325985 [ 44 ] {} <Fatal> BaseDaemon: 17. 0x5603cbd271e0 ? /usr/bin/clickhouse
2019.11.10 19:39:39.326028 [ 44 ] {} <Fatal> BaseDaemon: 18. 0x7fbb86c376db start_thread /lib/x86_64-linux-gnu/libpthread-2.27.so
2019.11.10 19:39:39.326052 [ 44 ] {} <Fatal> BaseDaemon: 19. 0x7fbb8655488f __clone /lib/x86_64-linux-gnu/libc-2.27.so

I assume this is a direct result of this library using native tcp, and not the safer http interfaces. My code is similar to this:

let mut block = Block::new();
for key in values.iter() {
    block = block.column("key", vec![key.as_str().unwrap()]);
}
...
let done = pool
    .get_handle()
    .and_then(move |c| c.insert(index, block))
    .and_then(move |c| Ok(()))
    .map(|c| info!("flushed"))
    .map_err(|err| eprintln!("database error: {}", err));

let task = tokio::run(done);

Insert JSONEachRow supporting.

I've tried insert JSONEachRow. Like this:

 let query = format!(r#"INSERT INTO payment FORMAT JSONEachRow {"customer_id":"34","amount":6,"account_name":"bar"};"#);
client.execute(query).await.unwrap();

and got a timeout error.
Seems like query and execute methods are not suitable with INSERT.
Could I somehow do INSERT with JSONEachRow?

try to covert to block to Vec<T>

When we want to query data from clickhouse, and covert to struct fast.

so hope to get this code:

#[derive(BlockWrap)]
pub struct Person {
   name: String,
   age: i32
}
let data = block.into() 

How do you think about this?

Reusing handle after dropped stream_blocks leads to panic

Now the following code compiles, but panics, once handle is reused, with a stacktrace leading to PoolBinding::detach:

let blocks = handle.query(some_query).stream_blocks();
// do smth with blocks
drop(blocks);

handle.query(other_query).stream().next().await // <- panic

Haven't tested other combinations of QueryResult functions, those might also trigger the same behavior.

Hanging connections; Pool issue?

Hello,

While using clickhouse in our project, we've been finding that the connection to clickhouse will sometimes hang. While the upcoming v0.1.16 will solve an issue that fixes one of the symptom, I believe that there may be a more fundamental issue.

The minimal example is at https://github.com/hwchen/clickhouse-hang. The basic idea is that when using pool_max=1, running two queries in series will cause the second query to enter reconnect logic (as seen from the logs) and then hang. In v0.1.16, a timeout error will eventually occur, freeing up the connection and a query can be sent again. In v0.1.14, the hang is indefinite.

Assuming that this behavior is real, I think that it's likely that our usage of pool_max much greater than 1 has been hiding it to some extent.

Let me know if you need any more details. Also, @jspeis and I are both happy to investigate further and/or work on a fix.

Sorry, I wasn't able to find yet what code in clickhouse is responsible, but I wanted to point out the behavior first in case you could provide some direction.

Connect to tls-only Clickhouse server

Currently it's only possible to connect to Clickhouse server via TCP, but there is no option to connect via TLS.
I've bumped into this problem while trying to connect to TLS-only Clickhouse installation.

Is there any possibility to implement TLS option?

I can try it myself, if you currently have no time for that :)

In that case I'd ask if by any chance you can provide me with some advices on how you'd like it to be on the API side (I thought about yet another URI option like secure) and on the inner side (like, should there be two different ClickhouseTransports, or is it better to abstract it behind ClickhouseTransport.inner? I'm thinking about latter).

Anyway, thanks for this project, hope you'll find time to look into this issue :)

Can't get examples to complie

Trying this example: https://github.com/suharev7/clickhouse-rs/blob/master/examples/simple.rs

error[E0463]: can't find crate for `futures`
 --> src/main.rs:2:1
  |
2 | extern crate futures;
  | ^^^^^^^^^^^^^^^^^^^^^ can't find crate

Adding futures = "*" to Cargo.toml doesn't help much:

error[E0433]: failed to resolve: use of undeclared type or module `env_logger`
  --> src/main.rs:12:5
   |
12 |     env_logger::init();
   |     ^^^^^^^^^^ use of undeclared type or module `env_logger`

error[E0433]: failed to resolve: use of undeclared type or module `tokio`
  --> src/main.rs:48:5
   |
48 |     tokio::run(done)
   |     ^^^^^ use of undeclared type or module `tokio`

Hope to use macro to build a Block ?

let instance = Block! { customer_id: vec![1_u32,  3,  5,  7,  9], amount: vec![2_u32,  4,  6,  8, 10] };
``

Hope to build a  block in this way. 

DriverError::Timeout is actually a tokio-timer error

Referring to:

clickhouse-rs/src/errors.rs

Line 98 in 5a98c54

Timeout,

By tracing the path to generate this error variant, it appears to correspond to the tokio-timer error when a Delay is registered with the Runtime unsuccessfully during a retry. It is not related to ClickHouse connection state.

However, in tokio 0.2.0 and above, timers will not produce error any more, so it is hopeful that this error variant can be turned into something more relavent to ClickHouse connection status.

build error `0.2.0-alpha.5`

error[E0308]: mismatched types
--> ~/.cargo/registry/src/github.com-1ecc6299db9ec823/clickhouse-rs-0.2.0-alpha.5/src/types/query_result/mod.rs:106:29
|
106 | / ... Box::pin(
107 | | ... stream::iter(Rows {
108 | | ... row: 0,
109 | | ... block_ref,
... |
112 | | ... .map(|row| -> Result<Row<'static, Simple>> { Ok(row) }),
113 | | ... )
| |_______________________^ lifetime mismatch
|
= note: expected enum std::result::Result<types::block::row::Row<'static, _>, _>
found enum std::result::Result<types::block::row::Row<'a, _>, _>
note: the lifetime 'a as defined on the impl at 28:6...
--> ~/.cargo/registry/src/github.com-1ecc6299db9ec823/clickhouse-rs-0.2.0-alpha.5/src/types/query_result/mod.rs:28:6
|
28 | impl<'a> QueryResult<'a> {
| ^^
= note: ...does not necessarily outlive the static lifetime

query timeout?

is there a way to set a timeout limit for a given query so that the query could be cancelled if it doesn't execute in a given time frame?

if it doesn't exist, I'd be happy to try and implement that if you can provide some pointers

Question: Insert value for Nullable(DateTime)

I'm trying to insert a row of data containing a Nullable(DateTime) column using the Vec<(String, Value)> approach:

let mut row: Vec<(String, Value)> = Vec::new();

// get UTC time into DateTime<Tz> expected by clickhouse_rs
let dt: DateTime<Tz> = UTC.from_utc_datetime(&Utc::now().naive_utc()); 

// try to create a clickhouse_rs Value from an Option<DateTime<Tz>>
let column_value = Value::from(Some(dt));
// this fails ^^^

row.push(("column_name_here", column_value));

error:

   |
   |                         (Value::from(Some(dt)))
   |                                      ^^^^^^^^
   |                                      |
   |                                      expected an implementor of trait `std::convert::From<std::option::Option<chrono::datetime::DateTime<chrono_tz::timezones::Tz>>>`
   |                                      help: consider borrowing here: `&Some(dt)`
   |
   = note: required because of the requirements on the impl of `std::convert::From<std::option::Option<chrono::datetime::DateTime<chrono_tz::timezones::Tz>>>` for `clickhouse_rs::types::value::Value`
   = note: required by `std::convert::From::from`

following the compiler's suggestion ^ does not work.

A similar approach works for the other data types I've tried (String, f64, u8, etc.) but not DateTime<Tz> - is there some way I can work around this in order to insert data into a nullable datetime column?

Note - doing this instead:

// old: 
// let column_value = Value::from(Some(dt));
let column_value = Value::from(dt);

compiles, but results in a runtime error:

Error: "From SQL error: `SqlType::DateTime cannot be cast to Nullable(DateTime).`"

Thanks for your help!

Decimal support

If I try to select a decimal value from a table, I get the following error:

Unsupported column type "Decimal(18, 9)".

It would be nice if I could get the internal representation of decimal as i32 or i64.

Blob support

Hello, thanks for the crate!

According to the clickhouse documentation, the String value can contain an arbitrary set of bytes, not only valid utf8. It's useful for storing messages stored in a binary format.

At the moment clickhouse-rs doesn't support it. What about Value{Ref}::Blob?

1.46.0 lifetime error

Fail to compile after updating rust to 1.46.0, error:

clickhouse-rs-0.2.0-alpha.5/src/types/query_result/mod.rs:106:29
    |
106 | / ...                   Box::pin(
107 | | ...                       stream::iter(Rows {
108 | | ...                           row: 0,
109 | | ...                           block_ref,
...   |
112 | | ...                       .map(|row| -> Result<Row<'static, Simple>> { Ok(row) }),
113 | | ...                   )
    | |_______________________^ lifetime mismatch

error compiles when use nightly rustc

error[E0308]: mismatched types
   --> src/types/query_result/mod.rs:106:29
    |
106 | / ...                   Box::pin(
107 | | ...                       stream::iter(Rows {
108 | | ...                           row: 0,
109 | | ...                           block_ref,
...   |
112 | | ...                       .map(|row| -> Result<Row<'static, Simple>> { Ok(row) }),
113 | | ...                   )
    | |_______________________^ lifetime mismatch
    |
    = note: expected enum `std::result::Result<types::block::row::Row<'static, _>, _>`
               found enum `std::result::Result<types::block::row::Row<'a, _>, _>`

Error happend when I use the rustc 1.46.0-nightly (7750c3d46 2020-06-26)

and the 0.2.0-alpha.5

client hello outputs raw server credentials

Would it be possible to disable this line? Because at the moment it seems like it may cause credentials to get logged to the output in plaintext

Support for IPv4 and IPv6 column types

It would be useful to have support for IPv4 and IPv6 columns.

I think the easiest way would be to use std::net::Ipv4Addr/Ipv6Addr objects.

Are there plans to add support for `DateTime64` data type?

As in title. Are there plans to add support for DateTime64 data type?

Tests fail if system timezone is not UTC

Some tests fail for me if the system timezone is not UTC. No failures if I change it to UTC and reboot before running the test.

Datetime confused

I am using the clickhouse and try to insert datetime to the clickhouse,

I try to use the demo in the test but I failed,

error[E0277]: the trait bound `chrono::DateTime<chrono_tz::Tz>: clickhouse_rs::types::stat_buffer::StatBuffer` is not satisfied
   --> src\main.rs:110:29
    |
110 |         .column("datetime", datetimes)
    |                             ^^^^^^^^^ the trait `clickhouse_rs::types::stat_buffer::StatBuffer` is not implemented for `chrono::DateTime<chrono_tz::Tz>`

here is the code

  let uc = UTC.ymd(record[3][0..4].parse::<i32>().unwrap(), record[3][4..6].parse::<u32>().unwrap(), record[3][6..8].parse::<u32>().unwrap())
            .and_hms_micro(record[4][0..2].parse::<u32>().unwrap(), record[4][4..5].parse::<u32>().unwrap(), record[4][7..8].parse::<u32>().unwrap(), record[5].parse::<u32>().unwrap());\

I felt even more confused
why not support multiple API likes using from ?

Connection Pool

Hello,

Are there plans to have a connection pool? Or should I use something like bb8

Insert by Row?

Is there any chance support could be added for insertion by row?

It seems anti-pattern with the other drivers to insert data column by column and not row by row.

Compiling under windows

This is a cool project, but it doesn't seem to build under windows at the moment. In order to get it running on my system, I had to change this line to

#if _WIN32 || _WIN64

New release with tokio 1.0

Hi. Any plans to release a new crate with tokio 1.0 support?

0.2.0-alpha.5 doesn't compile

I tried both last stable (1.46.0) and nightly (1.47.0-nightly 2020-08-25) Rust, both give me the same error

error[E0308]: mismatched types
   --> src/types/query_result/mod.rs:106:29
    |
106 | / ...                   Box::pin(
107 | | ...                       stream::iter(Rows {
108 | | ...                           row: 0,
109 | | ...                           block_ref,
...   |
112 | | ...                       .map(|row| -> Result<Row<'static, Simple>> { Ok(row) }),
113 | | ...                   )
    | |_______________________^ lifetime mismatch
    |
    = note: expected enum `std::result::Result<types::block::row::Row<'static, _>, _>`
               found enum `std::result::Result<types::block::row::Row<'a, _>, _>`
note: the lifetime `'a` as defined on the impl at 28:6...
   --> src/types/query_result/mod.rs:28:6
    |
28  | impl<'a> QueryResult<'a> {
    |      ^^
    = note: ...does not necessarily outlive the static lifetime

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: could not compile `clickhouse-rs`.

Array support

Are there any plans for adding support for Arrays? I would be interested in implementing this myself (with some guidance) if not.

I tried to look if I could implement this using existing building blocks in the library, but I got stuck immediately in types::Value. The first idea was to make the enum recursive by adding Array(List<Value>) to it. However this would allow mixing different types. We could make a new Array type like struct Array (SqlType, List<Value>) which wouldn't allow inserting different values, though that is still pretty inefficient. I think the most efficient way would be to make
Value generic like Value<T = ()> and then we could add an Array(List<T>) to the enum.

I'm not sure what the best approach would be here. What do you think?

version get

Hey brother,
Your project in crate.io version clickhouse-rs 0.2.0-alpha.5 can't compile , I have fix it in the last pr,
I try to use clickhouse-rs={git="https://github.com/suharev7/clickhouse-rs/"}, but I can't get the async-await branch .

could you fix it?

IPv4 address is reversed if inserted in strings format

Hello, I wrote a demo app to demonstrate this-

use clickhouse_rs::{Pool, types::Block};
use tokio::prelude::Future;

fn main() {
    let ddl = "
        CREATE TABLE IF NOT EXISTS test.ipv4
        (
            ip IPv4
        )
        ENGINE = Memory
        ";

    let pool = Pool::new("tcp://tester:[email protected]:9000/test?compression=lz4&ping_timeout=100ms");

    let records_block = Block::new()
            .column("ip", vec!["192.168.2.1", "1.2.3.4"]);

    let client = pool.get_handle()
            .and_then(move |c| c.execute(ddl))
            .and_then(move |c| c.insert("ipv4", records_block))
            .and_then(|_| Ok(()))
            .map_err(|err| eprintln!("database error: {}", err));
    tokio::run(client)
 }

In Clickhouse database,

blah :) select * from test.ipv4;

SELECT *
FROM test.ipv4

┌──────────ip─┐
│ 1.2.168.192 │
│     4.3.2.1 │
└─────────────┘

2 rows in set. Elapsed: 0.004 sec.

The address went from 192.168.2.1 to 1.2.168.192 etc.

clickhouse-rs-cityhash-sys doesn't compile on ARM Linux

Compilation of clickhouse-rs-cityhash-sys fails on ARM based Linux system with a following error:

$ cargo build --verbose
       Fresh cc v1.0.50
   Compiling clickhouse-rs-cityhash-sys v0.1.1 (/home/hombit/clickhouse-rs-cityhash-sys-0.1.1)
     Running `rustc --crate-name clickhouse_rs_cityhash_sys src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 -C metadata=a54df48fff86c350 -C extra-filename=-a54df48fff86c350 --out-dir /home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/deps -C incremental=/home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/incremental -L dependency=/home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/deps -L native=/home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/build/clickhouse-rs-cityhash-sys-5a676c1e89bd3e7c/out -l static=chcityhash -l stdc++`
error[E0308]: mismatched types
  --> src/lib.rs:20:26
   |
20 |     unsafe { CityHash128(buffer.as_ptr() as *const i8, buffer.len()) }
   |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
   |
   = note: expected raw pointer `*const u8`
              found raw pointer `*const i8`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: could not compile `clickhouse-rs-cityhash-sys`.

Caused by:
  process didn't exit successfully: `rustc --crate-name clickhouse_rs_cityhash_sys src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 -C metadata=a54df48fff86c350 -C extra-filename=-a54df48fff86c350 --out-dir /home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/deps -C incremental=/home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/incremental -L dependency=/home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/deps -L native=/home/hombit/clickhouse-rs-cityhash-sys-0.1.1/target/debug/build/clickhouse-rs-cityhash-sys-5a676c1e89bd3e7c/out -l static=chcityhash -l stdc++` (exit code: 1)

I use Rust 1.42.0, uname -a output:

Linux rel 5.5.9-1-default #1 SMP Thu Mar 12 06:33:05 UTC 2020 (70a6377) aarch64 aarch64 aarch64 GNU/Linux

the trait `std::future::Future` is not implemented for `clickhouse_rs::pool::futures::get_handle::GetHandle`

`use std::error::Error;
use futures_util::StreamExt;
use std::str::FromStr;
use clickhouse_rs::{row, types::Block, Pool};
use async_std::task;

async fn execute(database_url: String) -> Result<(), Box> {

let ddl = r"
    CREATE TABLE IF NOT EXISTS payment (
        customer_id  UInt32,
        amount       UInt32,
        account_name Nullable(FixedString(3))
    ) Engine=Memory";

let mut block = Block::with_capacity(5);
block.push(row! { customer_id: 1_u32, amount:  2_u32, account_name: Some("foo") })?;
block.push(row! { customer_id: 3_u32, amount:  4_u32, account_name: None::<&str> })?;
block.push(row! { customer_id: 5_u32, amount:  6_u32, account_name: None::<&str> })?;
block.push(row! { customer_id: 7_u32, amount:  8_u32, account_name: None::<&str> })?;
block.push(row! { customer_id: 9_u32, amount: 10_u32, account_name: Some("bar") })?;
// let pool = Pool::new(database_url());
let pool = Pool::new("tcp://default:[email protected]:9000/darwin?ping_timeout=42ms".to_owned());

let mut client = pool.get_handle().await?;
client.execute(ddl).await?;
client.insert("payment", block).await?;
let mut stream = client.query("SELECT * FROM payment").stream();

while let Some(row) = stream.next().await {
    let row = row?;
    let id: u32 = row.get("customer_id")?;
    let amount: u32 = row.get("amount")?;
    let name: Option<&str> = row.get("account_name")?;
    println!("Found payment {}: {} {:?}", id, amount, name);
}

Ok(())

}

fn main() {
task::block_on(execute("tcp://default:[email protected]:9000/darwin?ping_timeout=42ms".to_owned())).unwrap();
}

[dependencies]
clickhouse-rs = "0.1.21"
futures = "0.3.4"
futures-util = "0.3.5"
async-std = "1.6.2"

`

error[E0277]: the trait bound clickhouse_rs::pool::futures::get_handle::GetHandle: std::future::Future is not satisfied
--> src/main.rs:28:22
|
28 | let mut client = pool.get_handle().await?;
| ^^^^^^^^^^^^^^^^^^^^^^^ the trait std::future::Future is not implemented for clickhouse_rs::pool::futures::get_handle::GetHandle

Release 0.2.x with async/await support

Thank you for the great crate. Do you have any plans to release next version with async/await support?

Migration from failure

failure is deprecated so I would suggest to replace it with either thiserror or snafu, both of them are compatible with std::error::Error. I may help you with a migration itself, but need to decide which one to choose. Personally I like the first one since it smaller and simpler to use, but the former allows more explicit error handling.

Add support for UUID type

I'd be great to add support for UUID types: https://clickhouse.yandex/docs/en/data_types/uuid/

[async branch] Query on empty table crashes client

clickhouse-rs = { version = "0.2.0-alpha.3", default-features = false, features = ["async_std"] }

use clickhouse_rs::Pool;

async fn amain() -> std::io::Result<()> {
    let ddl = r"
        CREATE TABLE IF NOT EXISTS payment (
            customer_id  UInt32,
            amount       UInt32,
            account_name Nullable(FixedString(3))
        ) Engine=Memory";

    let database_url = "tcp://localhost:9000";
    let pool = Pool::new(database_url);

    let mut client = pool.get_handle().await?;
    client.execute(ddl).await?;
    let block = client.query("SELECT * FROM payment").fetch_all().await?;

    for row in block.rows() {
        let id: u32             = row.get("customer_id")?;
        let amount: u32         = row.get("amount")?;
        let name: Option<&str>  = row.get("account_name")?;
        println!("Found payment {}: {} {:?}", id, amount, name);
    }

    Ok(())
}

fn main() {
    async_std::task::block_on(amain()).unwrap();
}

crashes at

The application panicked (crashed).
Message:  blocks should not be empty.
Location: src/libcore/option.rs:1188
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                          (9 post panic frames hidden)
 9: core::option::expect_failed::ha19676642f985d31
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/option.rs:1188
10: core::option::Option<T>::expect::h7f223c049345661a
    at <unknown source file>
11: clickhouse_rs::types::block::Block::concat::hb3631d01190f30c1
    at <unknown source file>
12: clickhouse_rs::types::query_result::QueryResult::fetch_all::{{closure}}::h5cb956ae16817174
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/clickhouse-rs-0.2.0-alpha.3/src/types/query_result/mod.rs:40
      38 │             })
      39 │             .await?;
      40 >         Ok(Block::concat(blocks.as_slice()))
      41 │     }
      42 │
13: <std::future::GenFuture<T> as core::future::future::Future>::poll::h7e4757197a3d6c15
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/future.rs:43
14: std::future::poll_with_tls_context::hcbbebb5adc01600a
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/future.rs:99
15: chrs_empty_chunk_bug::amain::{{closure}}::h0618f8e982313893
    at /Users/ath/Desktop/chrs-empty-chunk-bug/src/main.rs:16
      14 │     let mut client = pool.get_handle().await?;
      15 │     client.execute(ddl).await?;
      16 >     let block = client.query("SELECT * FROM payment").fetch_all().await?;
      17 │
      18 │     for row in block.rows() {
16: <std::future::GenFuture<T> as core::future::future::Future>::poll::hbd6938f851d171e9
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/future.rs:43
17: std::future::poll_with_tls_context::h7d159d26975d6f2c
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/future.rs:99
18: async_std::task::block_on::block_on::{{closure}}::hcf2793df5dc7a7ac
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.4.0/src/task/block_on.rs:69
      67 │         }
      68 │
      69 >         future.await
      70 │     };
      71 │
19: <std::future::GenFuture<T> as core::future::future::Future>::poll::h9514f53f0f36293e
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/future.rs:43
20: async_std::task::block_on::run::{{closure}}::h816ed4f2bfadb79c
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.4.0/src/task/block_on.rs:130
     128 │         let mut step = 0;
     129 │         loop {
     130 >             if let Poll::Ready(t) = future.as_mut().poll(cx) {
     131 │                 // Save the parker for the next invocation of `block`.
     132 │                 cache.set(Some(arc_parker));
21: std::thread::local::LocalKey<T>::try_with::hdf7835f3ad873576
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/local.rs:262
22: std::thread::local::LocalKey<T>::with::hf20ce7f8c58095da
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/local.rs:239
23: async_std::task::block_on::run::hae2023dda2a588bc
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.4.0/src/task/block_on.rs:119
     117 │     pin_utils::pin_mut!(future);
     118 │
     119 >     CACHE.with(|cache| {
     120 │         // Reuse a cached parker or create a new one for this invocation of `block`.
     121 │         let arc_parker: Arc<Parker> = cache.take().unwrap_or_else(|| Arc::new(Parker::new()));
24: async_std::task::block_on::block_on::{{closure}}::h291e29a105c0e435
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.4.0/src/task/block_on.rs:73
      71 │
      72 │     // Run the future as a task.
      73 >     unsafe { Task::set_current(&task, || run(future)) }
      74 │ }
      75 │
25: async_std::task::task::Task::set_current::{{closure}}::hdbfc92a8d92df19a
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.4.0/src/task/task.rs:129
     127 │                 current.set(old_task);
     128 │             }
     129 >             f()
     130 │         })
     131 │     }
26: std::thread::local::LocalKey<T>::try_with::h345ca29fc90da595
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/local.rs:262
27: std::thread::local::LocalKey<T>::with::h313b34c9cb72053e
    at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/local.rs:239
28: async_std::task::task::Task::set_current::hd1b76895f7972498
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.4.0/src/task/task.rs:124
     122 │         F: FnOnce() -> R,
     123 │     {
     124 >         CURRENT.with(|current| {
     125 │             let old_task = current.replace(task);
     126 │             defer! {
29: async_std::task::block_on::block_on::he68527be772e060b
    at /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.4.0/src/task/block_on.rs:73
      71 │
      72 │     // Run the future as a task.
      73 >     unsafe { Task::set_current(&task, || run(future)) }
      74 │ }
      75 │
30: chrs_empty_chunk_bug::main::h411acd94b5968b0e
    at /Users/ath/Desktop/chrs-empty-chunk-bug/src/main.rs:30
      28 │ fn main() {
      29 │     color_backtrace::install();
      30 >     async_std::task::block_on(amain()).unwrap();
      31 │ }
                         (6 runtime init frames hidden)

Readonly param

It would be really helpful if we can get the readonly as one of the parameter in the URL that is being for connection to the clickhouse

Hanging and high cpu usage

First, thanks so much for writing this! I'm using this in one of my projects right now.

I'm using actix-web, and calling out to clickhouse here: https://github.com/hwchen/tesseract/blob/master/tesseract-server/src/handlers/aggregate.rs#L111

I think I'm using it correctly; but I've been able to induce a hang (with 100% of one core usage) pretty reliably by sending a two requests concurrently.

So basically, I start up the server, send 2 requests which call clickhouse simultaneously. One resolves, and the other one hangs, using 100% of one core.

Looking at sudo perf top I get

  99.05%  tesseract                [.] _ZN88_$LT$clickhouse_rs..block..block..ReadBlockFuture$u20$as$u20$futures..future..Future$GT$4poll17h48077bff47d8f67fE

I'm not super familiar with futures, but would be happy to look further if you can give me some guidance.

Or let me know if there's anything else I can do.

streaming rows

Hello again!

I was wondering if you plan to expose an api for streaming rows. I see that you have Fold currently, I'm not sure that will work for my current purposes (streaming output rows through to the user). If it could work, or I missed something in the api, let me know!

FromSql trait is private

To my mind it is better to have public FromSql trait. Sometimes there is a need to write generics like this:

fn get_db_column<T>(column_name: &str, query: &str) -> Vec<T>
where T: Send + FromSql
{
        let mut runtime = tokio::runtime::Runtime::new()
                                   .expect("Unable to create a runtime");
        let pool = Pool::new(get_clickhouse_addr());

        let done = pool
            .get_handle()
            .and_then(move |c| c.query(query).fetch_all())
            .and_then(move |(_, block)| {
                let mut column_vec = vec!();

                for row in block.rows() {
                    let item: T = row.get(column_name)
                ...

Alternative hosts support for load-balancing

Hey, any plans to add client side load balancing? Like in official clickhouse-go client? (alt_hosts option)

If not, when any recommendation on how to implements it properly using multiple connections at application level?

thanks for rust client! We've been thinking about deploying separate go service just for clickhouse before I've found it :)

Possible bug in pool

Hello,

I may have found a bug in the handling of new connections which causes a race condition.

The effect is that I've been able to semi-reliably cause a panic. (Although, I do have to send thousands of concurrent requests at my app over several minutes to cause it to happen).

If this is the bug, I've also created a fix, you can see it in the pull request. I don't know if I'm understanding all the pool logic correctly, so feel free to ask for changes, or just close and fix yourself.
Pull request: #24

I'm currently testing the fix through a long-running load test, but wanted to get the pull request in first to hear your thoughts on it. I'm not sure you have a way to reliably reproduce though.

(The fix only directly addresses the first of these errors, I'm not exactly sure if the others will also be fixed, but I think it might)

Errors:

thread 'arbiter:3c5ae82c-8c17-442d-9b94-4285fed8c742:actix-net-worker-7' panicked at 'index out of bounds: the len is 0 but the index is 0', /rustc/ec194646fef1a467073ad74b8b68f6f202cfce97/src/libcore/slice/mod.rs:2461:14
stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: std::panicking::rust_panic_with_hook
   5: <unknown>
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::panicking::panic_bounds_check
   9: <unknown>
  10: <clickhouse_rs::pool::futures::get_handle::GetHandle as futures::future::Future>::poll
  11: <unknown>
  12: <unknown>
  ...

thread 'arbiter:295230d8-3040-4dbf-8573-9e5c40307fc3:actix-net-worker-1' panicked at 'cannot poll MapErr twice', src/libcore/option.rs:1040:5
stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: std::panicking::rust_panic_with_hook
   5: <unknown>
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::option::expect_failed
   9: <unknown>
  10: <unknown>
  11: <clickhouse_rs::pool::futures::get_handle::GetHandle as futures::future::Future>::poll
...

thread 'arbiter:c9c90e43-7289-480f-8b2e-1a43d4e24518:actix-net-worker-5' panicked at 'cannot poll a chained future twice', /home/hwchen/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.25/src/future/chain.rs:33:28
stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: std::panicking::rust_panic_with_hook
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <clickhouse_rs::pool::futures::get_handle::GetHandle as futures::future::Future>::poll
...

Support for columnar function on insert

Bad title, but basically I would like the ability to wrap a column's values with a function on insert.

Right now the following code would result in a query like INSERT INTO table (addresses) VALUES ("192.168.107.9"), ("192.168.45.8")

let block = Block::new()
        .column("addresses",  vec!["192.168.107.9", "192.168.45.8"]);

I would like the ability to wrap the values in a, for example, toIPv4 function. So the following code would produce a query like INSERT INTO table (addresses) VALUES (toIPv4("192.168.107.9")), (toIPv4("192.168.45.8"))

let block = Block::new()
        .column("addresses",  vec!["192.168.107.9", "192.168.45.8"]);
block.column_function("addresses", "toIPv4");

This could also allow having a column populated by just a function, like having a Timestamp column populated by the now() function

Use backticks for column names in insert block queries

Using block.column(col_name, col_from) for block population and then inserting it to server results in a query with "col_name" as is (without backticks):

INSERT INTO db.table1 (id, ns:exampleAttr1, ns:exampleAttr2, ...) VALUES

which leads to a syntax error due to colon used in column name.

Adding backticks manually to "col_name" results in a query like:

INSERT INTO db.table1 (`id`, `ns:exampleAttr1`, `ns:exampleAttr2`, ...) VALUES

But with another error complaining that there is no column with `id` in a block.

Is it a bug or there is some other way to use column names with spaces and special characters (delimiters)?

clickhouse-rs = "0.2.0-alpha.4"

And thanks a lot for your work!

Tokio v0.2

Hey there,

as I'm sure you already heard, Tokio has recently released the first alpha version of v0.2, finally bringing async/await without combinator madness. Do you have any plans to migrate towards it anytime soon? Do you want to wait for the final?

Construct Decimal from mantissa and exponent

Right now I see no way to construct Decimal without going through float with potential loose of precision. Thus it's not possible to insert monetary data. It would be nice to see interface similar to those in rust_decimal (standard de facto for decimal according to usage stats?), especially Decimal::new.

There is also a fork that defines Decimal::of_no_scale method with similar semantics.

clickhouse-rs does not compile with the latest version of tokio 0.3.5

I'm getting such error:

error[E0277]: `GetHandle` is not a future
  --> src/main.rs:72:22
   |
72 |     let mut client = pool.get_handle().await?;
   |                      ^^^^^^^^^^^^^^^^^^^^^^^ `GetHandle` is not a future
   |
   = help: the trait `Future` is not implemented for `GetHandle`
   = note: required by `poll`

If I get back to tokio 0.1.22 it compiles.

Is it possible to fix this?