Giter Site home page Giter Site logo

dremio_client's Introduction

Dremio client

Documentation Status Updates Codacy Codecov

The un-official python client for Dremio's REST API. This enables both administrators and data scientists to get the most out of Dremio in Python

Features

  • Cross platform support

  • All Pythons between 2.7 - 3.7 supported

  • Full support for Dremio's REST API

  • Optional Support for Dremio's ODBC or experimental Arrow Flight capabilities

  • Rich config file support via confuse yaml config library. Simple to create a client (config stored in a yaml file)

    from dremio_client import init
    client = init() # initialise connectivity to Dremio via config file
    catalog = client.data # fetch catalog
    vds = catalog.space.vds.get() # fetch a specific dataset
    df = vds.query() # query the first 1000 rows of the dataset and return as a DataFrame
    pds = catalog.source.pds.get() # fetch a physical dataset
    pds.metadata_refresh() # refresh metadata on that dataset
  • CLI interface for integration with scripts

    $ dremio_client query --sql 'select * from sys.options'
    {'results':results}
    $ dremio_client refresh-metadata --table 'my.vds.name'
    {'result':'ok'}
  • Catalog autocompletion in Jupyter Notebooks

https://raw.github.com/rymurr/dremio_client/master/docs/images/autocomplete.gif

Status

This is still alpha software and is relatively incomplete. Contributions in the form of Github Issues or Pull requests are welcome. See CONTRIBUTING

TODO

  • see issues

dremio_client's People

Contributors

abtvkt avatar drivard avatar gclarkjr5 avatar insatomcat avatar kishorekumar422 avatar markfjohnson avatar pyup-bot avatar rymurr avatar skadyan avatar tibbersdrivemustang avatar viktorgitdev avatar vintydremio avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

dremio_client's Issues

json posted with requests is string, and so is escaped, make all put/post queries fail

  • Dremio client version: 0.10.1
  • Dremio version: 4.1.8
  • Python version: 3.8.2
  • Operating System: docker/python:3

Description

Trying to use the update_catalog endpoint.

What I Did

root@13290d3e7d28:/work# dremio_client update-catalog -i f03366b6-bf34-46dc-b57e-641c1cc46325 '{"entityType": "dataset", "id": "f03366b6-bf34-46dc-b57e-641c1cc46325", "type": "VIRTUAL_DATASET", "path": ["BDF", "example_bug_round"], "sql": "SELECT * FROM bug_round_test limit 1000", "sqlContext": ["@RE00004T"], "accessControlList": {"groups": [{"id": "PrestaBDF", "permissions": ["READ", "WRITE"]}, {"id": "BDF", "permissions": ["READ", "WRITE"]}], "version": "4"}, "fields": [{"name": "ID ", "type": {"name": "DOUBLE"}}, {"name": "value", "type": {"name": "UNION", "subSchema": [{"type": {"name": "DOUBLE"}}, {"type": {"name": "VARCHAR"}}]}}]}'
{"entityType": "dataset", "id": "f03366b6-bf34-46dc-b57e-641c1cc46325", "type": "VIRTUAL_DATASET", "path": ["BDF", "example_bug_round"], "sql": "SELECT * FROM bug_round_test limit 1000", "sqlContext": ["@RE00004T"], "accessControlList": {"groups": [{"id": "PrestaBDF", "permissions": ["READ", "WRITE"]}, {"id": "BDF", "permissions": ["READ", "WRITE"]}], "version": "4"}, "fields": [{"name": "ID ", "type": {"name": "DOUBLE"}}, {"name": "value", "type": {"name": "UNION", "subSchema": [{"type": {"name": "DOUBLE"}}, {"type": {"name": "VARCHAR"}}]}}]}
Traceback (most recent call last):
  File "/usr/local/bin/dremio_client", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/cli.py", line 374, in update_catalog
    x = _update_catalog(token, base_url, cid, data, ssl_verify=verify)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/endpoints.py", line 411, in update_catalog
    return _put(base_url + "/api/v3/catalog/{}".format(cid), token, json, ssl_verify=ssl_verify)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/endpoints.py", line 60, in _put
    return _check_error(r, details)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/endpoints.py", line 72, in _check_error
    raise DremioBadRequestException("Requested object does not exist on entity " + details, error)
dremio_client.error.DremioBadRequestException: Requested object does not exist on entity : 400 Client Error: Bad Request for url: http://XXXX:81/api/v3/catalog/f03366b6-bf34-46dc-b57e-641c1cc46325

The problem is that the json is encoded when sent to the server:

PUT /api/v3/catalog/f03366b6-bf34-46dc-b57e-641c1cc46325 HTTP/1.1
Host: XXXX:81
User-Agent: python-requests/2.23.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: XXX
content-type: application/json
Content-Length: 641


"{\"entityType\": \"dataset\", \"id\": \"f03366b6-bf34-46dc-b57e-641c1cc46325\", \"type\": \"VIRTUAL_DATASET\", \"path\": [\"BDF\", \"example_bug_round\"], \"sql\": \"SELECT * FROM bug_round_test limit 1000\", \"sqlContext\": [\"@RE00004T\"], \"accessControlList\": {\"groups\": [{\"id\": \"PrestaBDF\", \"permissions\": [\"READ\", \"WRITE\"]}, {\"id\": \"BDF\", \"permissions\": [\"READ\", \"WRITE\"]}], \"version\": \"4\"}, \"fields\": [{\"name\": \"ID \", \"type\": {\"name\": \"DOUBLE\"}}, {\"name\": \"value\", \"type\": {\"name\": \"UNION\", \"subSchema\": [{\"type\": {\"name\": \"DOUBLE\"}}, {\"type\": {\"name\": \"VARCHAR\"}}]}}]}"HTTP/1.1 

400 Bad Request

Method Not Allowed for url Reflections

  • Dremio client version: 0.15.1
  • Dremio version: 17.0.0
  • Python version: 3.7
  • Operating System: MacOS

Description

Trying to get a list of reflections via either the simple or non-simple client options.

What I Did

from dremio_client import init
from dfp.connectors.connector import Connector

dremio_clientdir = config('DREMIO_CLIENTDIR')

client = init(config_dir=dremio_clientdir, simple_client=True)

reflections = client.reflections()

---------------------------------------------------------------------------
DremioException                           Traceback (most recent call last)
<ipython-input-29-3e4ddf05516b> in <module>
----> 1 reflections = client.reflections()

~/anaconda3/envs/dfp/lib/python3.8/site-packages/dremio_client/dremio_simple_client.py in reflections(self, summary)
    111 
    112     def reflections(self, summary=False):
--> 113         return reflections(self._token, self._base_url, summary, ssl_verify=self._ssl_verify)
    114 
    115     def reflection(self, reflectionid):

~/anaconda3/envs/dfp/lib/python3.8/site-packages/dremio_client/model/endpoints.py in reflections(token, base_url, summary, ssl_verify)
    183     :return: result object
    184     """
--> 185     return _get(base_url + "/api/v3/reflection" + ("/summary" if summary else ""), token, ssl_verify=ssl_verify)
    186 
    187 

~/anaconda3/envs/dfp/lib/python3.8/site-packages/dremio_client/model/endpoints.py in _get(url, token, details, ssl_verify)
     44 def _get(url, token, details="", ssl_verify=True):
     45     r = requests.get(url, headers=_get_headers(token), verify=ssl_verify)
---> 46     return _check_error(r, details)
     47 
     48 

~/anaconda3/envs/dfp/lib/python3.8/site-packages/dremio_client/model/endpoints.py in _check_error(r, details)
     88     if code == 404:
     89         raise DremioNotFoundException("No entity exists at " + details, error, r)
---> 90     raise DremioException("unknown error", error)
     91 
     92 

DremioException: unknown error: 405 Client Error: Method Not Allowed for url: http://dremio.nonprod.yumds.com:80/api/v3/reflection

catalog-item seems to substitute "/" for "." characters

  • Dremio client version: 0.15.1
  • Dremio version: 19.3
  • Python version: 3.10.0
  • Operating System: MacOS 11.13.1

Description

While using the tool to retrieve items using the path I noticed that the sample sets are getting . characters substituted for / This means if I wanted to access an item via a path I cant seem to if it has period characters

What I Did

% dremio_client --config . catalog-item 'Samples/samples.dremio.com'
Traceback (most recent call last):
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/bin/dremio_client", line 8, in <module>
    sys.exit(cli())
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/dremio_client/cli.py", line 191, in catalog_item
    x = _catalog_item(token, base_url, cid, [i.replace(".", "/") for i in path] if path else None, ssl_verify=verify,)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/dremio_client/model/endpoints.py", line 111, in catalog_item
    return _get(base_url + "/api/v3/catalog{}".format(endpoint), token, idpath, ssl_verify=ssl_verify)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/dremio_client/model/endpoints.py", line 46, in _get
    return _check_error(r, details)
  File "/Users/mc/Issues/NewYorkLifeIns/115277/dremio-env/lib/python3.10/site-packages/dremio_client/model/endpoints.py", line 89, in _check_error
    raise DremioNotFoundException("No entity exists at " + details, error, r)
dremio_client.error.DremioNotFoundException: No entity exists at , Samples/samples/dremio/com: 404 Client Error: Not Found for url: http://172.25.1.66:9047/api/v3/catalog/by-path/Samples%2Fsamples%2Fdremio%2Fcom

I tried escaping and double quotes but that didnt seem to help

Conversely a regular URL with CURL will work ok so I know the API is ok

curl -s -H 'Content-Type: application/json' -H "Authorization: _dremiou5lr78suu8cmvuut3o08lm7igi" -X GET "172.25.1.66:9047/api/v3/catalog/by-path/Samples/samples.dremio.com/NYC-taxi-trips?pretty" | jq '.id'
"dremio:/Samples/samples.dremio.com/NYC-taxi-trips"

Got Segmentation fault while using flight.query

  • Dremio client version: 0.13.6
  • Dremio version: 11.0.0-202011171636110752-16ab953d
  • Python version: Python 3.7.8
  • Operating System: NAME="Ubuntu" ,VERSION="18.04.5 LTS (Bionic Beaver)"

Description

I executed the following query:
'SELECT ts FROM s3."my-bucket".motibz.velodyne WHERE dir0=\'95fa\' AND dir1=\'drive_id=5e329d63bd724a000126d9b2\' AND ts>=569999927.0 AND ts<=570000927.0 ORDER BY ts'
And I got the following error
Fatal Python error: Segmentation fault
If I add another column it works perfectly:
'SELECT x,ts FROM s3."my-bucket".motibz.velodyne WHERE dir0=\'95fa\' AND dir1=\'drive_id=5e329d63bd724a000126d9b2\' AND ts>=569999927.0 AND ts<=570000927.0 ORDER BY ts'
If I'm executing the "problematic" query throw the dremio UI SQL editor it works perfectly
image

Unfortunately, there is no traceback but core...

The returned data info is:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1716 entries, 0 to 1715
Data columns (total 2 columns):
Column Non-Null Count Dtype
0 x 1716 non-null float64
1 ts 1716 non-null float64
dtypes: float64(2)
memory usage: 26.9 KB

--skip-verify not working with CERTIFICATE_VERIFY_FAILED error

  • Dremio client version:
  • Dremio version: Enterprise Edition 4.1.7-202002220604270180-54f278ec
  • Python version: Python 3.7.6
  • Operating System: macOS Catalina Version 10.15.3

Description

I am trying to use the dremio_client command to issue a SQL command to the Dremio server. However, I am getting a client side certificate verification error even when I use the --skip-verify argument

What I Did

First, I installed dremio-client using the pip3 command like this:

$ pip3 install dremio-client

Then I attempted to run the following command:

$ dremio_client --ssl --skip-verify --hostname dremio.org --port 443 --username <MY USERID> --password '<MY PASSWORD>' query --sql 'select * from sys.options'

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dremio_client/auth/__init__.py", line 41, in auth
    return _existing_token(config_dict)
  File "/usr/local/lib/python3.7/site-packages/dremio_client/auth/__init__.py", line 76, in _existing_token
    raise KeyError
KeyError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 360, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dremio.org', port=443): Max retries exceeded with url: /apiv2/login (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/dremio_client", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dremio_client/cli.py", line 107, in query
    base_url, token = get_base_url_token(args)
  File "/usr/local/lib/python3.7/site-packages/dremio_client/conf/cli_helper.py", line 36, in get_base_url_token
    token = auth(base_url, config)
  File "/usr/local/lib/python3.7/site-packages/dremio_client/auth/__init__.py", line 44, in auth
    token = config_auth(base_url, config_dict)
  File "/usr/local/lib/python3.7/site-packages/dremio_client/auth/config.py", line 52, in login
    return _login(base_url, username, password, timeout, verify)
  File "/usr/local/lib/python3.7/site-packages/dremio_client/auth/basic.py", line 40, in login
    r = requests.post(url, json={"userName": username, "password": password}, timeout=timeout, verify=verify)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='dremio.org', port=443): Max retries exceeded with url: /apiv2/login (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)')))

I also tried placing the configuration in the file: ~/.config/dremio_client/config.yaml but I experienced the same error.

$ cat ~/.config/dremio_client/config.yaml
auth:
    type: basic #  currently only basic is supported
    username: [email protected]
    password: ********
    timeout: 10
hostname: dremio.org
port: 443
ssl: true
skip-verify: true

python module: get item from id ?

Can the python module retrieve a catalog item from its id ?
That's very easy to do with the client, but from the python module I cannot find a way.

For now the only way I found to programmatically get a specific VDS given its ID and path is to start at the root et move throught the path, folder by folder. And this has a limitation given most of my folders have spaces in them, which must be converted to underscores:

"path": ["BDF", "VDS Management"]

In [45]: catalog["BDF"]["VDS Management"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-45-b5d3437247dc> in <module>
----> 1 catalog["BDF"]["VDS Management"]

KeyError: 'VDS Management'

In [46]: catalog["BDF"]["VDS_Management"]
Out[46]: {"entityType": "folder", "id": "652074a1-7c74-4f25-bc82-e64c32d0cdbb", "path": ["BDF", "VDS Management"], "tag": "vjbOHwWjwZo=", "accessControlList": {"users": null, "groups": null, "version": "1"}}

Thank you for your advice.

getting token expired issues in dremio simple client

  • Dremio client version:
  • Dremio version:
  • Python version:
  • Operating System:

Description

I have written a python api using dremio simple cleint and running on ubuntu VM.it was working fine from past 4-5 days but sudenly server responding with Error

What I Did

here  is  python  api  to ftech  dremion dataset  details
import requests
from dremio_client import init
from dremio_client.flight import query
import json
from concurrent.futures import as_completed
from flask_cors import CORS
import pandas as pd
# Load an empty map
from flask import Flask, request, jsonify
app = Flask(__name__)
app.config["DEBUG"] = True
CORS(app)
from waitress import serve
import urllib3

requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
try:
    requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass
MapConfig = ''
@app.route('/api/', methods=['GET','POST'])
def api_get():
    if request.method == 'GET':
        url = 'https://testdremio.test.com/apiv2/login'
        headers  = {"content-type": "application/json","Connection": "keep-alive"}
        data = json.dumps({"userName": "*****", "password": "*****"})
        response = requests.post(url, data=data,headers = headers,verify=False )
        print(response.json())
        url = 'https://testdremio.test.com/apiv2/space/Test_Space/folder/Datasets'
        authkey='_dremio'+response.json()['token']
        print(authkey)
        headers  = {"Accept": "application/json","Authorization": authkey}
        print(headers)
        response = requests.get(url, headers = headers,verify=False )
        print(response.json())
        return response.json()
    elif request.method == 'POST':
        data=None
        global MapConfig
        print('post app')
        requestPayLoad = json.loads(json.dumps(request.json))
        print(requestPayLoad)
        if requestPayLoad['Type'] == 'UpdateMapConfig':
            MapConfig=requestPayLoad['Value']
            data='SUCCESS'
        elif requestPayLoad['Type'] == 'ReadMapConfig':
            data=MapConfig
        else:
            query=requestPayLoad['Value']
            client = init(simple_client=True)
            results = client.query(query)
            data=pd.DataFrame(results)
            data=data.to_json()
        return  data;

    else:
        return 'not OK'
serve(app,host='0.0.0.0', port=5001)



and  i  am  getting  below  error

json.decoder.JSONDecodeError: Extra data: line 1 column 124 (char 123)
ERROR:waitress:Exception while serving /api/
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/waitress/channel.py", line 350, in service
    task.service()
  File "/usr/local/lib/python3.6/dist-packages/waitress/task.py", line 171, in service
    self.execute()
  File "/usr/local/lib/python3.6/dist-packages/waitress/task.py", line 441, in execute
    app_iter = self.channel.server.application(environ, start_response)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in _call_
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "server.py", line 29, in api_get
    client = init(simple_client=True)
  File "/usr/local/lib/python3.6/dist-packages/dremio_client/__init__.py", line 67, in init
    return _connect(config, simple_client)
  File "/usr/local/lib/python3.6/dist-packages/dremio_client/__init__.py", line 72, in _connect
    return SimpleClient(config)
  File "/usr/local/lib/python3.6/dist-packages/dremio_client/dremio_simple_client.py", line 82, in _init_
    self._token = auth(self._base_url, config)
  File "/usr/local/lib/python3.6/dist-packages/dremio_client/auth/__init__.py", line 41, in auth
    return _existing_token(config_dict)
  File "/usr/local/lib/python3.6/dist-packages/dremio_client/auth/__init__.py", line 73, in _existing_token

teporary solution if i delete auth.json file it will start responding .how to solve this token issues so that i dont have to maual delete auth.json

ODBC Issues, Downgrading from Flight --> ODBC --> REST

  • Dremio client version: 0.12.0
    would be helpful to have a --version here:
dremio_client --version
  • Dremio version: 4.2.1-202004111451200819-0c3ecaea
  • Python version: 3.8.2
  • Operating System: MacOS

Description

Hi. I'm trying to connect over ODBC with dremio_client. Always see the following when trying to execute a query.:

WARNING:root:Unable to run query as flight, downgrading to odbc
WARNING:root:Unable to run query as odbc, downgrading to rest

The flight downgrade is not surprising and acceptable for now but the ODBC downgrade is a bit surprising. I think that is tied to this:
https://github.com/rymurr/dremio_client/blob/f892563f64ed07c00b4a1314172d3ea9d324d41c/dremio_client/odbc.py#L31

What I Did

When trying to connect over ODBC via pyodbc driver name: Dremio ODBC Driver i see the following. Note, this is external to dremio_client for the sake of this test...

pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'Dremio ODBC Driver' : file not found (0) (SQLDriverConnect)")

However, changing the driver name to the absolute path of the dylib resolves the issue such as:
/Library/Dremio/ODBC/lib/libdrillodbc_sbu.dylib.
I believe this is why dremio_client is downgrading from odbc --> rest. Is there a way to override the default driver constant defined as _OSX_DRIVER here:
https://github.com/rymurr/dremio_client/blob/f892563f64ed07c00b4a1314172d3ea9d324d41c/dremio_client/odbc.py#L31

delete_catalog may fail if tag contains special characters

  • Dremio client version: v0.14.0
  • Dremio version: v.18.1
  • Python version: 3.7
  • Operating System: Linux

Description

Pulling information about catalog item and then deleting an item using id and tag returned may fail if tag contains special characters
vds_json = client.catalog_item(None, vds_path.split("/"))
client.delete_catalog(vds_json["id"], tag=vds_json["tag"])
In some cases characters in tag will break URL and error similar to this is returned

ERROR - exception:unknown error: 409 Client Error: Conflict for url:

https://dremio-test.expedia.biz:443/api/v3/catalog/5fda0e34-ab57-42b9-a800-5c3fa9169112?tag=GRjH+GRCIhM=

What I Did

It is easy to fix by urlencoding the tag before passing to the function
            vds_json = client.catalog_item(None, vds_path.split("/"))
            encoded_tag = urllib.parse.quote(vds_json["tag"], safe="")
            client.delete_catalog(vds_json["id"], tag=encoded_tag)
            
However, correct fix is probably to encode it if needed in the client library, before passing to Dremio 
https://github.com/rymurr/dremio_client/blob/5c2052d02abec1777bda8f985cebe9c50e040de9/dremio_client/model/endpoints.py#L561

Conda version seems to be stale

  • Dremio client version: 0.10.1
  • Dremio version: 15.0
  • Python version: 3.8.6
  • Operating System: Ubuntu 20.04.1 LTS

Description

I'm trying from dremio_client import flight, I get the following error message:

/opt/conda/lib/python3.8/site-packages/pyarrow/compat.py:24: FutureWarning: pyarrow.compat has been deprecated and will be removed in a future release
  warnings.warn("pyarrow.compat has been deprecated and will be removed in a "

Then, when I try to connect, I get the following error message:

/opt/conda/lib/python3.8/site-packages/dremio_client/flight/__init__.py in connect(*args, **kwargs)
    101 
    102     def connect(*args, **kwargs):
--> 103         raise NotImplementedError("Python Flight bindings require Python 3 and pyarrow > 0.14.0")
    104 
    105     def query(*args, **kwargs):

NotImplementedError: Python Flight bindings require Python 3 and pyarrow > 0.14.0

What I Did

I built the docker image, pulling 0.10.1 from conda, today. I'm not super familiar with conda, so I may be doing something wrong, however this appears to be the latest version number: https://github.com/rymurr/dremio_client/blob/master/conda.recipe/meta.yaml#L2

It appears this is executing code that was changed quite some time ago (See compat error above):
18efb0e

Finally, it looks like an artifact hasn't been published to conda in over a year: https://anaconda.org/rymurr/dremio_client. Is it possible the deployment process is broken?

pyodbc.Error with python module "Can't open lib 'None' : file not found"

  • Dremio client version: 0.10.1
  • Dremio version: 4.1.8
  • Python version: 3.8.2
  • Operating System: docker/python:3

Description

Trying to query a dataset using the python module.
The module falls back to ODBC, but then ODBC fails. I'm not quite sure what the prerequisites are to make odbc work (driver ? config ?)

What I Did

root@e5388f7692fa:/work# cat test.py
from dremio_client import init
client = init() # initialise connectivity to Dremio via config file
print(client)
catalog = client.data # fetch catalog
print(catalog)
vds = catalog.DEMO.test.test2.test.get() # fetch a specific dataset
df = vds.query() # query the first 1000 rows of the dataset and return as a DataFrame
print(df)

root@e5388f7692fa:/work# python test.py
<dremio_client.dremio_client.DremioClient object at 0x7f767ba88d60>
{"id": "root"}
WARNING:root:Unable to run query as flight, downgrading to odbc
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    df = vds.query() # query the first 1000 rows of the dataset and return as a DataFrame
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/data.py", line 747, in query
    return self.sql("select * from {}")
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/data.py", line 750, in sql
    return self._flight_endpoint(sql)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/dremio_client.py", line 131, in query
    return query(
  File "/usr/local/lib/python3.8/site-packages/dremio_client/query.py", line 64, in query
    return _odbc_query(sql, hostname=hostname, port=odbc_port, username=username, password=password)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/odbc.py", line 89, in query
    client = connect(hostname, port, username, password)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/odbc.py", line 65, in connect
    c = pyodbc.connect(
pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'None' : file not found (0) (SQLDriverConnect)")

Use the python module to change an ACL or the sql of a dataset

Dremio client version: 0.10.1
Dremio version: 4.1.8
Python version: 3.8.2
Operating System: docker/python:3

Description

I'm trying to write to the dremio catalog with the python module (not the utility).
For instance changing an ACL or a sql definition.

What I Did

from dremio_client import init
client = init()
catalog = client.data

Then I tried to change the attribute and then commit...

a = catalog.BDF.example_bug_round.get()
b = a.meta
b.sql = "SELECT * FROM bug_round_test limit 3000" 
setattr(a,"meta",b)
a.commit()

But commit seems deactivated (_dirty=False?)

So I don't even know if it's possible right now.

Can you please advise ?
Thank you.

Updates needed for newest Dremio version

  • Dremio client version: 0.14.0
  • Dremio version: 15.0
  • Python version: 3.9.4
  • Operating System: Linux

Description

The dremio_client repo doesn't work with the latest verion of Dremio (15.0), as supported by dremio-hub/arrow-flight-client-examples.

It's probably necessary to update the flight authentication method similarly to how it was done in the dremio-hub example (I'm including a diff of that here as well)
dremio-hub/arrow-flight-client-examples@a08169c

I would really appreciate the help!!

What I Did

What I'm seeing is that the dremio_client still works with older versions of Dremio, but with Dremio 15.0 when I do:

from flight import connect
client = connect(hostname, port, user, pass)

I get the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/homes/jonathanw/.conda/envs/test-pdc/lib/python3.9/site-packages/dremio_client/flight/__init__.py", line 52, in connect
    c.authenticate(HttpDremioClientAuthHandler(username, password if password else ""))
  File "pyarrow/_flight.pyx", line 1155, in pyarrow._flight.FlightClient.authenticate
  File "pyarrow/_flight.pyx", line 66, in pyarrow._flight.check_flight_status
pyarrow._flight.FlightUnauthenticatedError: gRPC returned unauthenticated error, with message: 

multi-user use case, trying to avoid "token sharing"

I'm trying to use dremio_client in an existing django app, where dremio_client must act with the connected user's privilege level.
It mean that if an admin and a non admin user are using the app at the same time, dremio_client must use the admin privilege on dremio with the admin user and the non-admin privilege with the non-admin user.
The authentication in dremio is passed either throught a yaml file (seems inapropriate in this use case) or throught environment variables (which seems more appropriate).
Howerver if I use environment variables I'm not quite sure I understand how dremio_client will manage the "session" (will it store and reuse of the token in a file with the use of the _existing_token method, or if not reauth for each and every request?)
it seems either there's a risk the token will be shared amongst user, or that it will suffer performance problem by reauthenticating at every request. Since this webapp is there to launch batch processing on thousands of VDS, I'm concerned.

Maybe we could add a possibility to init the client with a preexisting token ? I'm willing to work on it but I don't understand the codebase enough to be sure that's the best way to solve my issue.
Let me add that I'm a django total beginner.

Could you share your thoughts on this please ?
Thanks.

sql from command line fails with large result sets

Not sure if this is a problem w/ the python client or dremio but large result sets don't seem very happy.

Few things to try:

  • is there some connection leak in python
  • what is the error on the dremio side
  • is 100 too small as a record batch size

HTML encoding of in endpoints.py makes dremio_client utility fail when using HTTP

  • Dremio client version: current master
  • Dremio version: 4.1.8
  • Python version: 3.8.2
  • Operating System: docker/python:3

Description

Where querying dremio using HTTPS, current code works fine.

However when querying:

  • dremio using HTTP
  • with the dremio_client utility
  • with a path and not a cid

Then, I get this error :

# dremio_client catalog-item BDF.example_bug_round
Traceback (most recent call last):
  File "/usr/local/bin/dremio_client", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/cli.py", line 181, in catalog_item
    x = _catalog_item(token, base_url, cid, [i.replace(".", "/") for i in path] if path else None, ssl_verify=verify,)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/endpoints.py", line 99, in catalog_item
    return _get(base_url + "/api/v3/catalog{}".format(endpoint), token, idpath, ssl_verify=ssl_verify)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/endpoints.py", line 44, in _get
    return _check_error(r, details)
  File "/usr/local/lib/python3.8/site-packages/dremio_client/model/endpoints.py", line 77, in _check_error
    raise DremioNotFoundException("No entity exists at " + details, error)
dremio_client.error.DremioNotFoundException: No entity exists at , BDF/example_bug_round: 404 Client Error: Not Found for url: http://xxxx:81/api/v3/catalog/by-path/BDF%2Fexample_bug_round

What I Did

If I remove the HTML conversion then it solves the problem, and it still works with HTTPS...
I don't know why this conversion was here in the first place.
And I still haven't found why this fails only with HTTP and not HTTPS...

Move off TravisCI

free travis is shutting down. Have to translate the travis job to github actions

Can't fetch physical data sources metadata

  • Dremio client version: 0.14.0
  • Dremio version: 14.0.0
  • Python version: 3.8
  • Operating System: Mac OS 10.15.7 Catalina (Dockerized)

Description

I am trying to fetch physical datasets info via simple client API, but look like it only returns VIRTUAL_DATASETs.

    client = init(simple_client=True)
    catalog_raw_api_data = client.catalog()

    c_ids = map(lambda x: (x['id'], x['path']), catalog_raw_api_data['data'])
    for (c_id, c_path) in c_ids:
        catalog_item = client.catalog_item(c_id, c_path)
        entity_type = catalog_item.get('entityType')
        print(catalog_item)

Alternatively, via DremiClient I can't get anything:

    client = init()
    catalog = client.data
    pds = catalog.source.pds.get()

this renders an error:

Traceback (most recent call last):
  File "/Users/nikkatalnikov/opt/anaconda3/envs/flowtale/lib/python3.8/site-packages/dremio_client/model/data.py", line 484, in __getattr__
    value = dict.__getitem__(self, item)
KeyError: 'source'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nikkatalnikov/Desktop/projetcs/flowtale/flowtale-acc/application/dremio-datalake/dremio-clonner/dremio-exporter.py", line 8, in <module>
    pds = catalog.source.pds.get()
  File "/Users/nikkatalnikov/opt/anaconda3/envs/flowtale/lib/python3.8/site-packages/dremio_client/model/data.py", line 492, in __getattr__
    return dict.__getitem__(self, item)
KeyError: 'source'

What am I doing wrong? Thank you!

Using dremio_client to retrieve ACLs seems broken, at least with dremio 4.1.8

  • Dremio client version: 0.10.1
  • Dremio version: 4.1.8
  • Python version: 3.8.2
  • Operating System: docker/python:3

Description

Trying to audit the Space and Datasets ACL using dremio_client.
Using the REST API with curl it works:

$ curl -k https://XXX/api/v3/catalog/by-path/BDF -H 'Content-Type: application/json' -H "Authorization: _dremiodXXX" 2>/dev/null | jq '.'
{
  "entityType": "space",
  "id": "98bb39ce-50ea-43ed-867b-f80b7a760a85",
  "name": "BDF",
  "tag": "4",
  "children": [
REDACTED
  ],
  "accessControlList": {
    "groups": [
      {
        "id": "Group1",
        "permissions": [
          "READ",
          "WRITE"
        ]
      },
      {
        "id": "BDF",
        "permissions": [
          "READ",
          "WRITE"
        ]
      }
    ],
    "version": "4"
  }
}

using dremio_client, the permission list is always "null/none".

What I Did

In [1]: from dremio_client import init
   ...: client = init()
   ...: catalog = client.data

In [2]: catalog.BDF
Out[2]: {"entityType": "space", "id": "98bb39ce-50ea-43ed-867b-f80b7a760a85", "name": "BDF", "tag": "4", "path": ["BDF"], "accessControlList": {"users": null, "groups": [{"id": "Group1", "permission": null}, {"id": "BDF", "permission": null}], "version": "4"}}

In [3]: catalog.BDF.meta.accessControlList
Out[3]:
{'users': None,
 'groups': [{'id': 'Group1', 'permission': None},
  {'id': 'BDF', 'permission': None}],
 'version': '4'}

how to get schema of the dataset using dremio flight connector

  • Dremio client version:0.13.2
  • Dremio version: 4.6.1-202007220122450047-62e084d0
  • Python version: Python 3.8.3
  • Operating System: ubuntu 20.0

Description

I am trying to ser dremio flight client to query flight data,but data what i recieved from the dremio flight is missing schema ,i need schema to bind on the map ,i able to get same scheam with Data using dremio simple python client

What I Did

i have written below code for python for flight

from flask_cors import CORS
import pandas as pd
#from dremio_client import init
from dremio_client.dremio_client import query
#import pandas as pd
import json
import urllib3
# Load an empty map
from keplergl import KeplerGl
from flask import Flask, request, jsonify
app = Flask(__name__)
app.config["DEBUG"] = True

CORS(app)

@app.route('/api/', methods=['POST', 'GET'])
def api_post():
    if request.method == 'POST':
        print('post app')
        req = request.json
        username = 'visur'
        password = '**************'
        hostname = '************'
        port = '47470'
        sql = '''select * from visurtest_Space.Datasets.FacilityCodeProductionDayRecord'''
        tlsfilename = 'ba04da291ec01ee6.crt'
        output = query(sql =sql, hostname=hostname, port=port, username=username, password=password,tls_root_certs_filename=tlsfilename)
        pandaataframe=pd.DataFrame(output)
        x = json.dumps(json.loads(pandaataframe.to_json(orient='records')), indent=2)
        print(x)
        df1 = output.to_json()
        y = json.loads(df1)
        return x
    else:
        return 'not OK'
app.run(host='0.0.0.0', port=5000

on the above code i am just getting data which i converted to json array,but i need to get data as well as Schema so that i can use in maps

here is the desired output i want to achive

image

REST API to Dataframe

  • Dremio client version: 0.13.6
  • Dremio version: 13
  • Python version: 3.7
  • Operating System: Windows

Description

When using the REST API, the dataframe returned is a list of json objects instead of the expected dataframe.

What I Did

from dremio_client import init
import pandas as pd

query = 'select * from information_schema."tables"'
client = init(simple_client=True)
results = client.query(query, asynchronous=False)
df = pd.DataFrame(results)

Results

 rowCount  ...                                               rows
0       636  ...  [{'TABLE_CATALOG': 'DREMIO', 'TABLE_SCHEMA': '...
1       636  ...  [{'TABLE_CATALOG': 'DREMIO', 'TABLE_SCHEMA': '...
2       636  ...  [{'TABLE_CATALOG': 'DREMIO', 'TABLE_SCHEMA': '...
3       636  ...  [{'TABLE_CATALOG': 'DREMIO', 'TABLE_SCHEMA': '...
4       636  ...  [{'TABLE_CATALOG': 'DREMIO', 'TABLE_SCHEMA': '...
5       636  ...  [{'TABLE_CATALOG': 'DREMIO', 'TABLE_SCHEMA': '...
6       636  ...  [{'TABLE_CATALOG': 'DREMIO', 'TABLE_SCHEMA': '...

Fix

Adding the following code would convert the results from the REST API to a proper dataframe.

new_df = pd.concat([pd.DataFrame(pd.json_normalize(x)) for x in df['rows']], ignore_index=True)

Proper results

TABLE_CATALOG  ... TABLE_TYPE
0          DREMIO  ...      TABLE
1          DREMIO  ...      TABLE
2          DREMIO  ...      TABLE
3          DREMIO  ...      TABLE
4          DREMIO  ...       VIEW
..            ...  ...        ...
631        DREMIO  ...      TABLE
632        DREMIO  ...      TABLE
633        DREMIO  ...      TABLE
634        DREMIO  ...      TABLE
635        DREMIO  ...      TABLE

Need to set "state" to null to be able to modify a source acl

  • Dremio client version: 0.14.0
  • Dremio version: 15.0
  • Python version: 3.6
  • Operating System: centos in docker

Description

When trying to modify the ACL of a SOURCE after creating it, I get the following error :

Requested object does not exist on entity : 400 Client Error: Bad Request for url: https://mathis-integration.rte-france.com:443/api/v3/catalog/d251df8f-1515-4c25-b9d6-4318956742d9

the code is as follows:

#SOURCE CREATION

source = 'SOURCE1'
NewSource='{"entityType": "source", "name": "'+ source + '", "description": "essai", "type": "NAS", "config": {"path": "/mnt/nasdata/test/", "allowCreateDrop": true}}'
d=catalog.add(item=json.loads(NewSource))
catalog[source]._dirty = True
catalog[source].commit()

#SOURCE ACL MODIFICATION

a = catalog.SOURCE1.get()
b = a.meta
b.accessControlList = json.loads('{"users": [{"id": "user1", "permissions": ["READ", "WRITE"]}], "groups": null, "version": null}')
setattr(a,"meta",b)
a._dirty = True
a.commit()

What I Did

The problem can be worked around in two ways. Either re-init the client between the creation and the modification:

client = init()
catalog = client.data

Or (and I found this because I checked the difference between "b" with or without the re-init) by removing the "state" attribute in the SourceMetadata before commiting (note the "b.state = None" line):

a = catalog.SOURCE1.get()
b = a.meta
b.accessControlList = json.loads('{"users": [{"id": "user1", "permissions": ["READ", "WRITE"]}], "groups": null, "version": null}')
b.state = None
setattr(a,"meta",b)
a._dirty = True
a.commit()

Note that this is only needed for sources (and maybe spaces I didn't test) but I don't have to do this to change an ACL on an folder or a VDS.

Maybe it's some dremio specificity that has not been taken into account by dremio_client ?

What do you think ?

Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.