Incompatible Zeppelin 0.7.2 and Spark 2.2

[anitagraser]

Incompatible Zeppelin 0.7.2 and Spark 2.2

[anitagraser]

I don't know since I haven't tested any newer Zeppelin versions yet. Outside Zeppelin, Spark 2.2 and ORC files seem to play nice.

[djr7m]

@anitagraser if you change the Cloud-Local config file, is there a pair of versions that does currently work for you? So, the problem is just the defaults that we have in place?

[anitagraser]

I did not try changing the config files, but I installed Spark 2.0 and that works. (I had an issue using ORC files with Spark 2.1)

[djr7m]

Spark 0.7 isn't available on the archive http://archive.apache.org/dist/spark/ so I'm going to attempt something like Spark 0.9.2
I also found a bug in cloud-local.sh where we are missing a slash in the Zepplin download URL. Clearly no one has been testing Zepplin recently. Frankly, I wasn't even sure how to test it beyond the standard download and install steps but I'll give it a try.

[djr7m]

Also, I found the Zepplin 0.7.0 release notes and it is supposed to support Spark 2.1 so that should work fine through standard config changes. https://zeppelin.apache.org/releases/zeppelin-release-0.7.0.html

[djr7m]

P/R #66 was tested with this config:

--- a/conf/cloud-local.conf
+++ b/conf/cloud-local.conf
@@ -37,7 +37,7 @@ pkg_zookeeper_ver="3.4.10"
 pkg_kafka_scala_ver="2.11"
 pkg_kafka_ver="0.9.0.1"
 
-pkg_spark_ver="2.2.0"
+pkg_spark_ver="2.1.2"
 # Note pkg_hadoop_ver above
 #  - don't auto-derive this as spark & hadoop major releases aren't lock-step
 #  - use "without-hadoop" to use version without hadoop deps
@@ -110,5 +110,5 @@ fi
 
 # Enable/Disable Zepplin
 # You must change Spark to a compatible version (2.1.x)
-zeppelin_enabled=0
+zeppelin_enabled=1

[anitagraser]

Sorry, I mixed up the Spark and Zeppelin versions this morning. I fixed them in the above comment now. Basically, Spark 2.1 worked fine with Zeppelin except that I couldn't read ORC files.

[djr7m]

Ok, the release notes are not written the same way between Zeppelin versions but I found that 0.7.3 supports Spark 2.2 https://issues.apache.org/jira/browse/ZEPPELIN-2768 so we could/should bump the default Spark and Zeppelin versions in the config.
Will the newest Spark + Zeppelin read your files?

[djr7m]

P/R merged, I'm going to assume this will work for you now. Please open a new issue as needed. Thanks for your interest in the project!