haqiqiar/spark-sample

Sample code for big data problems using Apache Spark (Python)

spark-sample

Sample code for big data problems using Apache Spark (Python)

Software Requirements

1. Python 2.7.x
2. Apache Spark
3. winutils.exe (If you are running on Windows)
4. Enthought Canopy

Setup for Windows

Check the details here.

1. Set environment variable SPARK_HOME to Spark's directory (where you extract the Apache Spark). For example: D:\spark. Make sure that your Spark directory structure is like this.
2. Copy winutils.exe to the directory D:\winutils\bin\.
3. Set environment variable HADOOP_HOME to winutils.exe's directory (D:\winutils\bin).