spark-yarn package doesn't exist in CDH repo

[ngageoint]

spark-yarn package doesn't exist in CDH repo

[chrisbennight]

Is there a separate 'spark-yarn'? I think the regular 'spark-core' package should work (did a quick glance and it seemed like it was build with yarn support)

http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.3/RPMS/noarch/spark-core-1.2.0+cdh5.3.2+369-1.cdh5.3.2.p0.17.el6.noarch.rpm

[pliguori]

Sorry maybe I wasn't clear enough. I'm talking about the maven artifacts that are defined in the mrgeo pom file.
In the cdh532 profile the pom wants to download the following spark-yarn dependency:

org.apache.spark
spark-yarn_2.10
${spark.version}

Where spark.version is<spark.version>1.2.0-cdh5.3.2</spark.version>
But this version does not exist in the cdh artifactory as you can see here:

https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-yarn_2.10/

[ttislerdg]

For some reason, Cloudera has left out the spark_yarn_2.10 jar from its CDH5.3.x repos. They do exist in previous and more recent versions. To get around the problem, I had to download the spark release from Cloudera (located at http://archive.cloudera.com/cdh5/cdh/5/, specifically http://archive.cloudera.com/cdh5/cdh/5/spark-1.2.0-cdh5.3.2.tar.gz for CDH5.3.2). Unpacking the tar contains all the spark jars. Then take the spark_yarn_2.10 and load it into your local maven repo using a typical mvn install:install-file command.

Let me know if this works for you.

[pliguori]

I will try soon. In the meantime I spoke to Cloudera support and they recommended to use spark-network-yarn instead of spark-yarn.

[pliguori]

I can confirm that your trick worked. The workaround suggested by Cloudera doesn'work at all, since spark-network-yarn depends on spark-yarn.
However it seems that Cloudera will redeploy the 1.2 artifact on their repository soon.