heavyai / heavyai Goto Github PK

View Code? Open in Web Editor NEW

19.0 19.0 19.0 7.93 MB

Python Data Science package for HeavyDB.

Home Page: https://heavyai.readthedocs.io/en/latest/

License: Apache License 2.0

Makefile 0.03% Shell 2.09% Python 97.88%

database heavyai python

heavyai's Introduction

heavyai

This package enables using common Python data science toolkits with HeavyDB. It brings data frame support on CPU and GPU as well as support for arrow. See the documentation for more.

Quick Install (CPU)

Packages are available on conda-forge and PyPI:

# using conda-forge
conda install -c conda-forge heavyai

# using pip
pip install heavyai

Quick Install (GPU)

We recommend creating a fresh conda 3.8 or 3.9 environment when installing heavyai with GPU capabilities.

To install heavyai for GPU Dataframe support (conda-only):

mamba create -n heavyai-gpu -c rapidsai -c nvidia -c conda-forge -c defaults \
    --no-channel-priority \
    cudf heavyai pyheavydb pytest shapely geopandas pyarrow=*=*cuda

Documentation

Further documentation for heavyai usage is available at: http://heavyai.readthedocs.io/

heavyai's People

Contributors

Stargazers

Watchers

Forkers

jp-harvey muskanmahajan37 guozanhua isabella232 vishalbelsare cuulee mfeldman143 xmnlab mdlenin validereinc tupui moniqueeygelaar crismorris95120 gopinathjcs conversight ajunlonglive guilhermeleobas khlaifiabilel mattgara

heavyai's Issues

Remove <py3.10 restriction

heavyai/environment.yml

Line 10 in d23eb70

- python >=3.7.0, <3.10

heavyai/environment_gpu.yml

Line 10 in d23eb70

- python >=3.7.0, <3.10

Is there a way to disable query cache (both for CPU and GPU)?

Hi there,

I'm benchmarking OmnisciDB via pyomnisci, is there a way to disable query cache (both for CPU and GPU) between runs for a single query (disabling omnisci to use any result of the previously run queries but letting it use the tables loaded in memory)?

Also, is there a way to set the number of CPU threads used during query execution (on CPU)? Is num-executors the correct flag to set?

Thanks so much,
Dong

load_table support for geospatial data

We need to add support in pymapd for geo data formats via load_table - minimally, to be able to pass geopandas dataframes to the load function.

The main thing is to make this performant since we'd want to do this using the columnar load path, ideally.

Return type Connection.get_table_details()

Hey, the get_table_details() function says that it returns a List[tuples] type, but it actually returns a TTableDetails object. Is this intended?

Python API remap dictionary

I am currently working with the Heavyai Python api to access/edit dashboards via databricks.
So far I can connect to my dash environment fine (I have the free version) - I use the following logic to build the connection:

from heavyai import connect
con = connect(user="admin", password="HyperInteractive", host="localhost",
              dbname="heavyai")

I then use the duplicate dashboard function to create a copy of a dashboard but with different data sources:

source_remap = {'oldtablename1': {'name': 'newtablename1'}, 'oldtablename2': {'name': 'newtablename2'}}
newdash = con.duplicate_dashboard(12345, "new dash", source_remap)

Duplicate dashboard creates a new dashboard, however remapping the data sources seems to only remap the first entry in the dict.
If I just do a remap without copying the dashboard, absolutely nothing changes. Is this a known issue?
Are there any possible workarounds to achieve the same result?

Support Columnar Load from Geopandas

This used to work, I believe, and so may be a regression in OmniSci 5.7

To replicate:

Create a geopandas data frame
Create a matching OmniSci table
Use load_table_columnar method
Note error message:

ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column omnisci_geo with type geometry')

Expected Behavior

Should load correctly, as long as table is pre-created with correct DDL

Next release

hi everyone!

I am implementing pyomnisci to ibis-omniscidb and I had this issue:

E       TypeError: load_table() missing 1 required positional argument: 'column_names'

it seems it was already fixed on master. Is there any plan for the next release?

install pymapd and cudf for GPU Dataframe support (conda-only) not working?

I tried the below command after a fresh miniconda installation
(https://repo.anaconda.com/miniconda/Miniconda3-py37_4.11.0-Linux-x86_64.sh)

conda create -n omnisci-gpu -c rapidsai -c nvidia -c conda-forge
-c defaults cudf=0.18 python=3.7 cudatoolkit=11.0 pyomnisci

as mentioned in here

https://github.com/heavyai/pyomnisci/blob/master/README.rst

and not successful with the following conflict:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package numba conflicts for:
cudf -> numba[version='>=0.40|>=0.41|>=0.45.1|>=0.46.0|>=0.48.0|>=0.49.0|>=0.53.1|>=0.54']
cudf -> rmm[version='>=22.2.0,<22.3.0a0'] -> numba[version='>=0.49']

Package libgcc-ng conflicts for:
python=3.8 -> libgcc-ng[version='>=7.3.0|>=7.5.0|>=9.3.0|>=9.4.0']
python=3.8 -> openssl[version='>=3.0.0,<4.0a0'] -> libgcc-ng[version='>=10.3.0|>=4.9|>=7.2.0']

Package _openmp_mutex conflicts for:
python=3.8 -> libgcc-ng[version='>=9.4.0'] -> _openmp_mutex[version='>=4.5']
cudatoolkit -> libgcc-ng[version='>=9.4.0'] -> _openmp_mutex[version='>=4.5']
cudf -> numba[version='>=0.54'] -> _openmp_mutex[version='>=4.5']
pyomnisci -> numba[version='>=0.49'] -> _openmp_mutex[version='>=4.5']

Package libstdcxx-ng conflicts for:
python=3.8 -> libstdcxx-ng[version='>=7.3.0|>=7.5.0|>=9.3.0|>=9.4.0']
python=3.8 -> libffi[version='>=3.2.1,<3.3.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0']

Package cudatoolkit conflicts for:
cudatoolkit
cudf -> cupy[version='>=9.5.0,<11.0.0a0'] -> cudatoolkit[version='10.0|10.0.|10.1|10.1.|10.2|10.2.|11.0|11.0.|11.1|11.1.|>=11.2,<12|9.2|9.2.|>=11.0,<=11.6|>=11.0,<=11.5|>=11.0,<11.2|>=11.2,<12.0a0|11.4|11.4.*|>=10.0.130,<10.1.0a0|>=9.2,<9.3.0a0|>=10.1.168,<10.2.0a0']
cudf -> cudatoolkit[version='>=10.1.243,<10.2.0a0|>=11,<12.0a0|>=11.2.72,<11.3.0a0|>=11.0.221,<11.1.0a0|>=11.4.1,<11.5.0a0|>=11.4.0,<11.5.0a0|>=10.2.89,<10.3.0a0']

Package python conflicts for:
pyomnisci -> python[version='>=3.7|>=3.7,<3.9']
cudf -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0']
pyomnisci -> numba[version='>=0.49'] -> python[version='2.7.|3.5.|3.6.|>=2.7,<2.8.0a0|>=2.7|>=3.10,<3.11.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|>=3.6,<3.7.0a0|>=3.6|>=3.5,<3.6.0a0|3.4.|>=3.7.1,<3.8.0a0']
python=3.8
cudf -> cachetools -> python[version='2.7.|3.4.|3.5.|3.6.|>=3.5|>=3.5,<3.6|>=3.10,<3.11.0a0|>=2.7,<2.8.0a0|>=3.6|>=3.7|>=3.5,<3.6.0a0|>=2.7|>=3.7.1,<3.8.0a0|3.8.|3.9.|3.7.*']

Package numpy conflicts for:
cudf -> numpy
cudf -> cupy[version='>=9.5.0,<11.0.0a0'] -> numpy[version='1.15.*|>=1.11.3,<2.0a0|>=1.14.6,<1.18.0|>=1.14.6,<2.0a0|>=1.15.4,<2.0a0|>=1.16.5,<2.0a0|>=1.16.6,<2.0a0|>=1.17|>=1.18|>=1.21.5,<2.0a0|>=1.18.5,<2.0a0|>=1.19.5,<2.0a0|>=1.21.2,<2.0a0|>=1.21.4,<2.0a0|>=1.17.5,<2.0a0|>=1.19.4,<2.0a0|>=1.19.2,<2.0a0|>=1.18.4,<2.0a0|>=1.18.1,<2.0a0|>=1.20.3,<2.0a0|>=1.20.2,<2.0a0|>=1.16,<2.0a0|>=1.9.0|>=1.16|>=1.15|>=1.16,<1.20.0a0|>=1.13.3,<2.0a0|>=1.12.1,<2.0a0|>=1.14,<1.20.0a0|>=1.9.3,<2.0a0']

Package pyarrow conflicts for:
cudf -> libcudf=0.5.1 -> pyarrow=0.12.0
cudf -> pyarrow[version='0.12.1.|0.14.1.|0.15.0.|0.17.1.|>=1.0.1,<1.0.2.0a0|>=4.0.1,<4.0.2.0a0|>=5.0.0,<5.0.1.0a0',build=*cuda]

Package pandas conflicts for:
cudf -> pandas[version='>=0.23.4|>=0.24.2,<0.25|>=0.25,<0.26|>=1.0,<1.2.0dev0|>=1.0,<1.3.0dev0|>=1.0,<1.4.0dev0|>=1.0,<=1.2.4']
cudf -> pyarrow[version='>=1.0.1,<1.0.2.0a0'] -> pandasThe following specifications were found to be incompatible with your system:

feature:/linux-64::__glibc==2.31=0
feature:|@/linux-64::__glibc==2.31=0
cudatoolkit -> __glibc[version='>=2.17,<3.0.a0']
cudatoolkit -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
cudf -> cuda-python[version='>=11.5,<12.0'] -> __glibc[version='>=2.17|>=2.17,<3.0.a0']

Your installed version is: 2.31

Could you please update the recommended package version in the readme as it not seem to be updated and perhaps not compatible now? I'm using Ubuntu 20.04

Auto-detect optimal arrow transport method

If the user is connecting to a DB instance on the same machine, automatically use the shared memory. If the machine is remote, use arrow wire transport (what we default to now).

Support Columnar Load of WKT from Pandas Dataframes

Current Behavior:

Create dataframe with WKT geo column (of any type point,line or poly)
Create a new OmniSci table with matching schema
Use Load Table Columnar method to upload dataframe to table
Observe error: unsupported type

ArrowInvalid: ('Could not convert POLYGON ((-81.81239976621903 41.49372525646714, -81.81243984996641 41.4937031198906, -81.81243236697307 41.49366718380983, -81.81238480029786 41.49365338429463, -81.81234471656468 41.4936755208384, -81.81235219949251 41.49371145693014, -81.81239976621903 41.49372525646714)) with type Polygon: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column omnisci_geo with type object')

Expected Behavior
Use shapely/libgeos to coerce the text column to supported binary transport form (wkb)

Current versions of heavyai and cudf don't work together

It looks like the current version of heavyai is out of date w/ cudf's API.

After starting the heavyai Docker image, registering my license key as directed, and creating a conda environment as directed by the docs:

conda create -n heavyai-gpu -c rapidsai -c nvidia -c conda-forge -c defaults python cudf cudatoolkit heavyai
...
conda list | grep rapids
cubinlinker               0.2.2            py39h11215e4_0    rapidsai
cudf                      22.12.01        cuda_11_py39_gf700408e68_0    rapidsai
libcudf                   22.12.01        cuda11_gf700408e68_0    rapidsai
librmm                    22.12.00        cuda11_g8aae42d1_0    rapidsai
rmm                       22.12.00        cuda11_py39_g8aae42d1_0    rapidsai
...
heavyai                   1.0                pyhd8ed1ab_0    conda-forge
pyheavydb                 6.1.1              pyhb39878e_1    conda-forge

Then trying to connect

>>> from heavyai import connect
>>> con = connect(user="admin", password="HyperInteractive", host="localhost",
...               dbname="heavyai")
>>> con
Connection(heavydb://admin:***@localhost:6274/heavyai?protocol=binary)
>>> con.select_ipc_gpu("show tables")
Traceback (most recent call last):
  File "/home/nfs/rgelhausen/conda/envs/heavyai-gpu/lib/python3.9/site-packages/heavyai/connection.py", line 317, in select_ipc_gpu
    from cudf.comm.gpuarrow import GpuArrowReader  # noqa
ModuleNotFoundError: No module named 'cudf.comm.gpuarrow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nfs/rgelhausen/conda/envs/heavyai-gpu/lib/python3.9/site-packages/heavyai/connection.py", line 320, in select_ipc_gpu
    raise ImportError(
ImportError: The 'cudf' package is required for `select_ipc_gpu`

CUDF gpu memory management

select_ipc_gpu calls thrift api sql_execute_gdf and optionally calls deallocate_ipc_gpu, or the user should call deallocate_ipc_gpu.

select_ipc_gpu does not pass the device_id to deallocate_ipc_gpu, so sends 0 by default, but does not error if the memory is on device_id=1. Is this correct?

pyomnisci currently depends on pyarrow=3.0.0, but cudf uses 5.0.0. OmniSci may upgrade to 5.0.0 soon.

pyomnisci has this code @pytest.mark.skip(reason="deallocate non-functional in recent distros") from 3 years ago, so this should be re-enabled for testing.

Should select_ipc_gpu use numba.cuda.cudadrv.devices.gpus to implicitly select device for sql_execute_gdf and for the device onto which it inits the DataFrame?

Update tests once HeavyDB is released

Tests need to connect to the default database which is named omnisci with OmniSciDB and heavyai with HeavyDB. Once a docker image is publicly available, we need to update the tests images and default database.

select_ipc can not handle empty query result.

If the result of a select statement is empty, then select_ipc will raise an error:

TTransportException('TSocket read 0 bytes')
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/pymapd/connection.py", line 489, in select_ipc
    tdf = self._client.sql_execute_df(
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/omnisci/thrift/OmniSci.py", line 1798, in sql_execute_df
    return self.recv_sql_execute_df()
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/omnisci/thrift/OmniSci.py", line 1814, in recv_sql_execute_df
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/thrift/protocol/TBinaryProtocol.py", line 134, in readMessageBegin
    sz = self.readI32()
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/thrift/protocol/TBinaryProtocol.py", line 217, in readI32
    buff = self.trans.readAll(4)
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/thrift/transport/TTransport.py", line 164, in read
    self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "/home/coder/.conda/envs/conda_jl/lib/python3.8/site-packages/thrift/transport/TSocket.py", line 142, in read
    raise TTransportException(type=TTransportException.END_OF_FILE,

the bug is in
https://github.com/omnisci/pymapd/blob/7750cec20ee4904ccec1f3828144fd14ada00e04/omnisci/thrift/OmniSci.py#L1814

What about catch this exception, and returns an empty dataframe?

Installation error (GPU)

Hi there,

I've been trying to install pyomnisci (GPU) but it hasn't worked. I have cuda 11.0 installed.

The command I have been using was，

conda create -n omnisci-gpu -c rapidsai -c nvidia -c conda-forge -c defaults cudf=0.18 python=3.7 cudatoolkit=11.0 pyomnisci

The error was,

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
pyomnisci -> numba[version='>=0.49'] -> python[version='2.7.|3.5.|3.6.|>=2.7,<2.8.0a0|>=2.7|>=3.6|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0|>=3.10,<3.11.0a0|>=3.5,<3.6.0a0|3.4.|>=3.7.1,<3.8.0a0']
pyomnisci -> python[version='>=3.7|>=3.7,<3.9']
python=3.7

Package numba conflicts for:
cudf=0.18 -> numba[version='>=0.49.0']
cudf=0.18 -> rmm[version='>=0.18.0,<0.19.0a0'] -> numba[version='>=0.49']

Package libstdcxx-ng conflicts for:
python=3.7 -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=7.5.0|>=9.3.0|>=9.4.0|>=7.2.0']
pyomnisci -> numba[version='>=0.49'] -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=7.5.0|>=9.3.0|>=9.4.0|>=7.2.0']
cudatoolkit=11.0 -> libstdcxx-ng[version='>=7.3.0|>=9.3.0|>=9.4.0']
cudf=0.18 -> cudatoolkit[version='>=11.0.221,<11.1.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=7.5.0|>=9.4.0|>=9.3.0|>=7.2.0']

Package pandas conflicts for:
cudf=0.18 -> pyarrow[version='>=1.0.1,<1.0.2.0a0'] -> pandas
cudf=0.18 -> pandas[version='>=1.0,<1.2.0dev0']

Package numpy conflicts for:
cudf=0.18 -> numpy
cudf=0.18 -> cupy[version='>7.1.0,<9.0.0a0'] -> numpy[version='>=1.11.3,<2.0a0|>=1.14.6,<1.18.0|>=1.14.6,<2.0a0|>=1.15|>=1.16|>=1.9.0|>=1.18.5,<2.0a0|>=1.19.5,<2.0a0|>=1.17.5,<2.0a0|>=1.16.6,<2.0a0|>=1.16.5,<2.0a0|>=1.19.4,<2.0a0|>=1.19.2,<2.0a0|>=1.15.4,<2.0a0|>=1.18.4,<2.0a0|>=1.18.1,<2.0a0|>=1.16,<2.0a0|>=1.16,<1.20.0a0']The following specifications were found to be incompatible with your system:

feature:/linux-64::__glibc==2.27=0
feature:|@/linux-64::__glibc==2.27=0
cudatoolkit=11.0 -> __glibc[version='>=2.17,<3.0.a0']
cudatoolkit=11.0 -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
python=3.7 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.27

p.s. I have also tried some other methods to install cudf and pyomnisci separately but no luck so far. They seem to be incompatible regarding the required versions for pyarrow.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.