Big Data Analysis examples using Hadoop
After cloning run the following command from the checkout directory.
chmod 755 gradlew
then run
./gradlew
and make sure that all the dependencies are pulled down. Assuming that runs you can now build the artifact using the command
./gradlew assemble
If you now check the directory build/libs/
you should see the jar file present.
You now want to copy the data into hdfs. Run the command
hdfs dfs -copyFromLocal data
Now you can run the job. For example running the following
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.mapreduce.AggregateJob data output
The job should and complete. Check the output folder in hdfs to ensure that the results are present.
hadoop fs -ls output
Running the LogWritable example
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.logwritable.LogProcessor /user/guest/logwritable-input /user/guest/logwritable-output 3
Running the Analytics examples
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.analytics.weblog.MsgSizeAggregateMapReduce /user/guest/logwritable-input /user/guest/msgsize-out
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.analytics.weblog.HitCountMapReduce /user/guest/logwritable-input /user/guest/hit-count-out
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.analytics.weblog.FrequencyDistributionMapReduce /user/guest/hit-count-out /user/guest/freq-dist-out
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.analytics.mbox.CountReceivedRepliesMapReduce /user/guest/mbox /user/guest/count-replies-out
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.analytics.mbox.CountSentRepliesMapReduce /user/guest/mbox /user/guest/count-emails-out
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.analytics.mbox.JoinSentReceivedReplies /user/guest/join-input /user/guest/join-output
Running the Inverted Index Example -
hdfs dfs -mkdir -p invertedindex/input
echo "cat sat mat" | hdfs dfs -put - invertedindex/input/1.txt
echo "dog lay mat" | hdfs dfs -put - invertedindex/input/2.txt
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.invertedindex.InvertedIndex invertedindex/input invertedindex/output
hdfs dfs -cat invertedindex/output/part*
Running the CSV example. Upload an example CSV file
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.csv.CSVMapReduce csv/input csv/output
hdfs dfs -cat csv/output/part*
Running the multiple outputs example
mkdir ../libtemp
wget http://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.0/commons-lang3-3.0.jar
mv commons-lang3-3.0.jar ../libtemp
hadoop jar build/libs/data-analysis.jar com.naughtyzombie.dataanalysis.multipleoutputs.MultipleOutputsJob -libjars ../libtemp/commons-lang3-3.0.jar csv/input csv/multi-output
Running the Spark Wordcount example
spark-submit --class com.naughtyzombie.dataanalysis.wordcount.WordCount --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 build/libs/data-analysis.jar /user/guest/join-input /user/guest/spark-wc-output
The VM config area is /etc/hadoop/conf/
DFS Health Monitoring App Hadoop GUI Hadoop GUI SparkUI SparkUI
Connect spark-shell
to the cluster (when logged onto cloudera box)
spark-shell --master yarn-client
Elasticsearch Marvel Marvel
Spark Installation Directory
/usr/lib/spark