gabechurch/sparkedatools

The missing SparklyR EDA toolkit (for use in R). Quick, efficient, and easy to use

sparkedatools

The missing SparklyR EDA toolkit (for use in R). Quick, efficient, and easy to use.

Wrap and graph the outputs from the SparkEDA package for Spark.

You will need a few things to get started.

1. Grab the SparkEDA Jar from my website

2. Download this repository to your R home or other location Note: You will need devtools installed in R (as this package is not yet up on CRAN

install.packages("devtools")
library("devtools")

Then

devtools::install_github('GabeChurch/sparkedatools')

3. Edit your SparklyR Configuration (in R)

You need to add the SparkEDA jar for the package to work in R.

conf = spark_config()

This is the important line

#This is the configuration option
conf$'sparklyr.jars.default'= "/system/path/to/sparkeda_2.11-2.07.jar"

sc = spark_connect(master = "yarn-client", config = conf, version = '2.3.2')

The ORDER IS IMPORTANT. Must be BEFORE you have connected and AFTER you have instantiated your spark_config in R.

4. Enjoy being able to visualize and understand your giant data-sets like never before!