sryza/simplesparkapp

Simple Spark Application

Simple Spark Application

A simple Spark application that counts the occurrence of each word in a corpus and then counts the occurrence of each character in the most popular words. Includes the same program implemented in Java and Scala.

To make a jar:

mvn package

To run from a gateway node in a CDH5 cluster:

spark-submit --class com.cloudera.sparkwordcount.SparkWordCount --master local \
  target/sparkwordcount-0.0.1-SNAPSHOT.jar <input file> 2

This will run the application in a single local process. If the cluster is running a Spark standalone cluster manager, you can replace "--master local" with "--master spark://<master host>:<master port>".

If the cluster is running YARN, you can replace "--master local" with "--master yarn".

simplesparkapp's Issues

How to run java main and scala main

From where i need to run this command
spark-submit --class com.cloudera.sparkwordcount.SparkWordCount --master local
target/sparkwordcount-0.0.1-SNAPSHOT.jar 2

Do I able to ran java example also with above command?

Maven build fails with unresolved dependencies

I attempted to run the maven build on a clean clone of the repo, and got the following error. It's possible that there is an issue with my maven configuration, as I am not an expert user.

[ERROR] Failed to execute goal on project sparkwordcount: Could not resolve dependencies for project com.cloudera.sparkwordcount:sparkwordcount:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at org.apache.spark:spark-core_2.10:jar:1.0.0-cdh5.1.0 -> org.eclipse.jetty:jetty-plus:jar:8.1.14.v20131031: Failed to read artifact descriptor for org.eclipse.jetty:jetty-plus:jar:8.1.14.v20131031: Could not find artifact org.eclipse.jetty:jetty-parent:pom:20 in scala-tools.org (http://scala-tools.org/repo-releases) -> [Help 1]
[ERROR] 

problem on cdh5.3.1

Hi,

Could you please help me in making this example work on 2.5.0-cdh5.3.1, i tried using different cloudera repo and dependency in the pom.xml but i m getting all the time:

Exception in thread "main" java.io.IOException: No FileSystem for scheme: dfs

regards,
Dalibor