java.lang.NoClassDefFoundError

[isarn]

java.lang.NoClassDefFoundError

[erikerlandson]

Looks like it is not finding the companion object TDigest$. I've never tried it that way, but it may be a consequence of how spark loads jar files.

Out of curiosity, does this work for you:

spark-shell --packages "org.isarnproject::isarn-sketches-spark:0.3.1-sp2.3-py3.6"

[erikerlandson]

Also: import org.isarnproject.sketches._

[richardxy]

Thanks. I tried what you asked for

$spark-shell --packages "org.isarnproject:isarn-sketches-spark:0.3.1-sp2.3-py3.6"

Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: org.isarnproject#isarn-sketches-spark;0.3.1-sp2.3-py3.6: not found]

[richardxy]

What's your recommended way of loading the package?

[erikerlandson]

you need two colons after org.isarnproject

[richardxy]

$ spark-shell --packages "org.isarnproject::isarn-sketches-spark:0.3.1-sp2.3-py3.6"
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Provided Maven Coordinates must be in the form 'groupId:artifactId:version'. The coordinate provided is: org.isarnproject::isarn-sketches-spark:0.3.1-sp2.3-py3.6
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.deploy.SparkSubmitUtils$$anonfun$extractMavenCoordinates$1.apply(SparkSubmit.scala:1000)
at org.apache.spark.deploy.SparkSubmitUtils$$anonfun$extractMavenCoordinates$1.apply(SparkSubmit.scala:998)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at org.apache.spark.deploy.SparkSubmitUtils$.extractMavenCoordinates(SparkSubmit.scala:998)
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1220)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:49)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:350)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[erikerlandson]

Oh, right it wants the scala version explicit.

spark-shell --packages "org.isarnproject:isarn-sketches-spark_2.11:0.3.1-sp2.3-py3.6"

I do think you'll also want: import org.isarnproject.sketches._

[richardxy]

Thanks. But...

scala> import org.isarnproject.sketches._
import org.isarnproject.sketches._

scala> val udaf = tdigestUDAF[Double]
:26: error: not found: value tdigestUDAF
val udaf = tdigestUDAF[Double]

[erikerlandson]

ah, I meant "in addition to the other imports you already had"

[richardxy]

Got it. Thank you!

[erikerlandson]

I think originally, you might have just been missing the import org.isarnproject.sketches._, although I'm not sure if the jar file would have included all the right dependencies.

For maven style packages, the "intended" way is to use --packages ..., but if you got the full set of jar files for all deps, then --jars ought to work too

[richardxy]

I'll follow your suggestion of importing packages. I didn't know what's the right package name to use in the first place. Now I know. Thanks!