qubole/qds-sdk-r

R extension to execute Hive Commands through Qubole Data Service Python SDK.

qds-sdk-R

R wrapper code for launching Hive, Spark, Presto queries using qds-sdk-py.

Requires

1. QDS Python SDK. To install it, you can either:

   • Install it from PyPI - pip install qds-sdk.

   • Download the package from GitHub and then

     cd /path/to/qds-sdk-py/
     sudo python setup.py install
     

2. rPython. In R,

   install.packages("rPython")
   

Installation:

In R

install.packages("devtools")
devtools::install_github("qubole/qds-sdk-R")

Usage:

In Shell

$ export QDS_API_TOKEN = xxyyzz

In R

library(qds)
results<-qds::quboleHiveCommand(query="show tables")

Or

library(qds)

hive<-HiveCommand$new() 
Or 
hive<-HiveCommand$new(<apitoken>) 
Or
hive<-HiveCommand$new(<apitoken>,<pollinterval>)

For Spark:
spark<-SparkCommand$new()
sparkCmdId<-spark$submit(sql="select * from default_qubole_memetracker limit 10")

Asynchronous

commandId<-hive$submit(query = "show tables")
result<-hive$getresult(commandId)

Cancel

hive$cancel(<commandId>)

Synchronous

result<-hive$run(query = "show tables")

For results

hive$getresult(<commandId>)

For logs

hive$getlog(<commandId>)

To display command object

hive$check(<commandId>)

Optional:

To change the default QDS environment variables,

$ export QDS_API_TOKEN = xxyyzz
$ export QDS_API_URL = https://api.qubole.com/api/
$ export QDS_API_VERSION = v1.2

Notes:

The default api_url is https://api.qubole.com/api/.

The default api_version is v1.2.

Additionally you can also use sampling in Hive. For further details read this page.

qds-sdk-r's Issues

Can't parse result of check(<commandId>)

I would like to check a command's status programmatically. However the result of

presto$check(commandId)

is printed to console and I couldn't assign it to a variable. I've tried capture.output and that didn't work. How would you suggest going about this? Thank you.

Compability with python 3

This package is not compatible with python3
Some of files: https://github.com/qubole/qds-sdk-R/blob/master/exec/r-quboleCommand.py

included old style syntax:
print """You need to install qds-sdk-py from http://github.com/qubole/qds-sdk-py"""
it doesn't work with python3 version

column names don't return from query results

A long string is returned as query results; It will be great to format results into data frame.

AND most importantly, no column names are returned

Getting result of a query in a data frame in R

Hi,

I am trying to get the result of a hive command in a data frame in R using the qds package.

results<-qds::quboleHiveCommand(query="show tables" ,api_token='xxx')

But instead of getting the result of my query in 'results' , i just get the output of my query in the R console.
Is there any way to get the result set of the query in a data frame??

Thanks
Japneesh Singh