mongodb-spark-demo's Introduction

MongoDB-Spark Demo

Prerequisites

To build the MongoDB-Spark demo application, you'll need to have the following:

Maven
MongoDB-Hadoop Connector
Spark (1.0 or greater, for Hadoop 2.x)

MongoDB-Hadoop

Note: you must build the MongoDB-Hadoop connector from source for your version of Hadoop and install the core JAR file to your local Maven repo. For example:

$ git clone http://github.com/mongodb/mongo-hadoop.git
$ cd mongo-hadoop
$ ./gradlew jar -Phadoop_version='2.4'
$ mvn install:install-file \
    -Dfile=core/build/lib/mongo-hadoop-core-1.2.1-SNAPSHOT-hadoop_2.4.jar \
    -DgroupId=com.mongodb \
    -DartifactId=hadoop \
    -Dversion=1.2.1-SNAPSHOT \
    -Dpackaging=jar

Spark

Refer to the Spark overview to get started.

Building

To the build the MongoDB-Hadoop demo applications use Maven:

$ mvn package

This will build the demo application and place all of the dependencies in target/lib. If instead you want to build a single jar with all of the dependencies, execute the assembly:single Maven goal:

$ mvn compile assembly:single

Running

$ cd your-spark-directory
$ SPARK_JAR=assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop2.4.0.jar \
  HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop \
  bin/spark-submit --master local --class com.mongodb.spark.demo.Recommender /path/to/demo-1.0-SNAPSHOT.jar \
  --jars /path/to/mongo-java-driver-2.12.2.jar,/path/to/hadoop-1.2.1-SNAPSHOT.jar \
  --executor-memory 4G /movielens/ratings.bson /movielens/users.bson \
  /movielens/movies.bson movielens.predictions