copy_to limit size requirement

[ianmcook]

copy_to limit size requirement

[cderv]

Yes I read the README and it is what makes me look into the code and into the doc of impala.

Impala does not include built-in capability for loading local data into these systems.

I tried to find some information on limitations and I found CREATE TABLE documentation that did not mentionned those limitations in size. I could use an SQL query to create a table with many rows using impala.

I wanted to know if it really was something not possible, or just not recommended and thus by design in Implyr. I saw also you wanted to make this limit a configuration. Thus my interrogation.

Also a dbWriteTable with impala odbc driver worked to create a table using impala bigger that was possible using copy_to in implyr.

I understand that impala is not good for writing and may be spark should be use to use PARQUET file for example.

[ianmcook]

Yes, this is explained in the README under the heading Loading Local Data into Impala. Quoting from there:

implyr does not provide tools for loading local data into Impala tables. This is because Impala can query data stored in several different filesystems and storage systems (HDFS, Apache HBase, Apache Kudu, Amazon S3, Microsoft ADLS, and Dell EMC Isilon) and Impala does not include built-in capability for loading local data into these systems.

Some other dplyr backends implement the function copy_to, which copies a local data frame to a remote source. implyr implements copy_to, but it currently only supports very small data frames. It uses the SQL INSERT ... VALUES() technique, which is not suitable for loading large amounts of data.

Also see the suggestions below there in the README.