vertica/r-dataconnector

Distributed Data Connector R package

r-dataconnector's Issues

Add globbing for reading orc files

It would be nice feature. Will try to investigate into this.

Installation results in "undefined symbol" error

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
g++ (GCC) 6.1.0
c++ (GCC) 6.1.0
GNU Make 4.2
Linux 2.6.32-431.23.3.el6

After cloning the repository with
git clone --recursive https://github.com/vertica/r-dataconnector.git

And then installing with the command
sudo R CMD INSTALL r-dataconnector/dataconnector

I get the following error

/usr/lib64/R/library/dataconnector/libs/dataconnector.so: undefined symbol: _ZN3ddc9ddc_writeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_RSt3mapIS5_N5boost3anyESt4lessIS5_ESaISt4pairIS6_SA_EEE

Parquet support

Any plans to add parquet support?

Install fails with "linker command failed" error

Following the install instructions yields the following error:

Undefined symbols for architecture i386:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[5]: *** [link_app.darwin] Error 1
make[4]: *** [openssl] Error 2
make[3]: *** [build_apps] Error 1
cp: the -H, -L, and -P options may not be specified with the -r option.
make[2]: *** [/Users/tobiasschmidt/repos/r-dataconnector/dataconnector/src/    distributed-data-connector/ddc/third-party/install/lib/libssl.a] Error 1
make[1]: *** [third-party] Error 2
make: *** [bootstrap] Error 2
ERROR: compilation failed for package ‘dataconnector’
* removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/dataconnector’

sessionInfo()

R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS Sierra 10.12.3

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] tools_3.3.2

Installation problems

I have some troubles with installation on both OS X and CentOS 7.
Can somebody point me direction?

1. OS X

cc -DMONOLITH -I.. -I../include -fPIC -fno-common -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch i386 -O3 -fomit-frame-pointer -DL_ENDIAN -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -c -o openssl.o openssl.c
rm -f openssl
shlib_target=; if [ -n "libcrypto.1.0.0.dylib libssl.1.0.0.dylib" ]; then
shlib_target="darwin-shared";
elif [ -n "" ]; then
FIPSLD_CC="cc"; CC=/usr/local/ssl/fips-2.0/bin/fipsld; export CC FIPSLD_CC;
fi;
LIBRARIES="-L.. -lssl -L.. -lcrypto" ;
make -f ../Makefile.shared -e
APPNAME=openssl OBJECTS="openssl.o verify.o asn1pars.o req.o dgst.o dh.o dhparam.o enc.o passwd.o gendh.o errstr.o ca.o pkcs7.o crl2p7.o crl.o rsa.o rsautl.o dsa.o dsaparam.o ec.o ecparam.o x509.o genrsa.o gendsa.o genpkey.o s_server.o s_client.o speed.o s_time.o apps.o s_cb.o s_socket.o app_rand.o version.o sess_id.o ciphers.o nseq.o pkcs12.o pkcs8.o pkey.o pkeyparam.o pkeyutl.o spkac.o smime.o cms.o rand.o engine.o ocsp.o prime.o ts.o srp.o"
LIBDEPS="-Wl,-search_paths_first $LIBRARIES "
link_app.${shlib_target}
( :; LIBDEPS="${LIBDEPS:--Wl,-search_paths_first -L.. -lssl -L.. -lcrypto }"; LDCMD="${LDCMD:-cc}"; LDFLAGS="${LDFLAGS:--fPIC -fno-common -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch i386 -O3 -fomit-frame-pointer -DL_ENDIAN -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM}"; LIBPATH=for x in $LIBDEPS; do echo $x; done | sed -e 's/^ *-L//;t' -e d | uniq; LIBPATH=echo $LIBPATH | sed -e 's/ /:/g'; LD_LIBRARY_PATH=$LIBPATH:$LD_LIBRARY_PATH ${LDCMD} ${LDFLAGS} -o ${APPNAME:=openssl} ${LIBDEPS} )
Undefined symbols for architecture i386:
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[5]: *** [link_app.darwin] Error 1
make[4]: *** [openssl] Error 2
make[3]: *** [build_apps] Error 1
cp: the -H, -L, and -P options may not be specified with the -r option.
make[2]: *** [/Users/dmitryselivanov/projects/r-dataconnector/dataconnector/src/distributed-data-connector/ddc/third-party/install/lib/libssl.a] Error 1
make[1]: *** [third-party] Error 2
make: *** [bootstrap] Error 2
ERROR: compilation failed for package ‘dataconnector’

• removing ‘/Users/dmitryselivanov/Library/R/3.2/library/dataconnector’

1. CentOS

cd gflags-2.1.2/Release; /tmp/Rtmp8CZV4y/R.INSTALL46f07159062b/dataconnector/src/distributed-data-connector/ddc/third-party/install/bin/cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-fPIC" .. && make -j32 && cp -r include/gflags /tmp/Rtmp8CZV4y/R.INSTALL46f07159062b/dataconnector/src/distributed-data-connector/ddc/third-party/install/include && cp lib/libgflags.a /tmp/Rtmp8CZV4y/R.INSTALL46f07159062b/dataconnector/src/distributed-data-connector/ddc/third-party/install/lib
/bin/sh: /tmp/Rtmp8CZV4y/R.INSTALL46f07159062b/dataconnector/src/distributed-data-connector/ddc/third-party/install/bin/cmake: cannot execute binary file
make[2]: *** [/tmp/Rtmp8CZV4y/R.INSTALL46f07159062b/dataconnector/src/distributed-data-connector/ddc/third-party/install/lib/libgflags.a] Error 126
make[2]: Leaving directory /tmp/Rtmp8CZV4y/R.INSTALL46f07159062b/dataconnector/src/distributed-data-connector/ddc/third-party' make[1]: *** [third-party] Error 2 make[1]: Leaving directory/tmp/Rtmp8CZV4y/R.INSTALL46f07159062b/dataconnector/src/distributed-data-connector/ddc'
make: *** [bootstrap] Error 2
ERROR: compilation failed for package ‘dataconnector’

• removing ‘/home/dmitry.selivanov/R/x86_64-redhat-linux-gnu-library/3.2/dataconnector’

Add Amazon S3 support

integer64 support

There is some integer64 support in R - bit64 package. It can make sense to follow data.table::fread logic when handling integer64. Actually it mean we should add integer64 class attribute. See here and here.

orc2dataframe limited to read of max 64mb chunk?

If .orc file is larger than 64mb, orc2dataframe reads only first 64mb and cuts remaining rows whithout any warnings / errors.
I tried to look for such limits in source code, but found nothing. Mb somewhere in third-party libraries.