kevinwkc/sparkr-tutorials

Code snippets and tutorials for working with SparkR.

sparkr-tutorials

Description: Code snippets and tutorials for working with SparkR.

Spark Social Science Manual

The tutorials included in this repository are geared towards social scientists and policy researchers that want to undertake research using "big data" sets. A manual to accompany these tutorials is linked below. The objective of the manual is to provide social scientists with a brief overview of the distributed computing solution developed by The Urban Institute's Research Programming Team, and of the changes in how researchers manage and analyze data required by this computing environment.

Spark Social Science Manual

Getting Started with SparkR Tutorials

Last Updated: October 3, 2016 (Updated to include Spark Social Science Manual)

In order to begin working with SparkR, users must first:

1. Make sure that SPARK_HOME is set in environment (using Sys.getenv)
2. Load the SparkR library
3. Initiate a sparkR.session

if (nchar(Sys.getenv("SPARK_HOME")) < 1) {
  Sys.setenv(SPARK_HOME = "/home/spark")
}

# Load the SparkR library
library(SparkR)

# Initiate a SparkR session
sparkR.session()

❗ The expressions given above must be evaluated in SparkR before beginning any of the tutorials hosted here. Example data loading is included in each tutorial.

Users can end a SparkR session with the following expression:

sparkR.session.stop()

Note: the data visualization tutorial, linked to below, is currently not updated for SparkR 2.0, but does still function with SparkR 1.6.

Table of Contents:

• SparkR Basics I: From CSV to SparkR DataFrame

  • Load a csv file into SparkR
  • Update a DataFrame with new rows of data
  • Rename DataFrame column(s)
  • Understanding data-types & schema
    • Specifying schema in read.df operation & defining a custom schema
  • Export DF as data file to S3

• SparkR Basics II: Essential DataFrame Operations

  • Aggregating
  • Grouping:
  • Arranging (Ordering) rows in a DataFrame
  • Append a column to a DataFrame
  • Types of SparkR operations
  • DataFrame Persistence:
  • Converting a SparkR DataFrame to a local R data.frame

• Subsetting SparkR DataFrames

  • Subset DataFrame by row
  • Subset DataFrame by column
    • Drop a column from a DF
  • Subset a DF by taking a random sample
    • Collect a random sample as a local data.frame
    • Export DF sample as a single .csv file to S3

• Dealing with Missing Data in SparkR

  • Specify null values when loading data in as a SparkR DataFrame (DF)
  • Conditional expressions on empty DF entries
    • Null and NaN indicator operations
    • Empty string entries:
    • Distribution of missing data across grouped data
  • Drop rows with missing data
    • Null value entries
    • Empty string entries
  • Fill missing data entries
    • Null value entries
    • Empty string entries

• Computing Summary Statistics with SparkR

  • Numerical Data
    • Measures of Location
      • Mean
    • Measures of dispersion
      • Range width & limits
      • Variance & standard deviation
      • Approximate Quantiles
    • Measures of distribution shape
      • Skewness
      • Kurtosis
    • Measures of dependence
      • Covariance & correlation
  • Categorical Data
    • Frequency table
    • Relative frequency table
    • Contingency table

• Merging SparkR DataFrames

  • Join (merge) two DataFrames by column condition(s)
  • Append rows of data to a DataFrame:
    • Append rows when column name lists are equal across DFs
    • Append rows when DF column name lists are not equal

• Data Visualizations in SparkR

  • Bar graph
    • Stacked or proportional bar graph
  • Histogram:
  • Frequency polygon
  • Boxplot:
  • Additional ggplot2.SparkR functionality
  • Functionality gaps between ggplot2 and SparkR extension
  • Bivariate histogram

• Time Series I: Working with the Date Datatype & Resampling a DataFrame

  • Converting a DataFrame column to 'date' dtype
  • Compute relative dates and measures based on a specified unit of time
    • Relative dates
    • Relative measures of time
  • Extract components of a date dtype column as integer values
  • Resample a time series DF to a particular unit of time frequency

• Fitting Generalized Linear Models in SparkR

  • Examine data prior to model fitting & perform data transformations
  • Fit Generalized Linear Model
    • Print model summary
    • General linear model measurements from Gaussian/identity GLM
    • Calculate fitted values from model
    • Compute sum of squared totals and residuals
    • Compute R-squared and adjusted R-squared
    • Fit Gaussian/identity GLM and general linear model in R for comparison
  • Fit other GLM distribution families supported by SparkR
    • Create binary response variable
    • Fit binomial, Gamma and Poisson GLMs in SparkR
  • Linear model diagnostics
    • Fitted v. residual values plot
    • Q-Q normality plot of the standardized residuals

Created by gh-md-toc