SparkContext giving StackOverflowError

[dfdx]

SparkContext giving StackOverflowError

[exyi]

I ran into the same problem with StackOverflowException and the problem was solved by switching to Java 8. I think there should be a warning when newer Java is used by accident, it was quite hard to find what caused the problems 😅

[dfdx]

The most likely case is using Spark 3, which we didn't test against yet. Can you try it with Spark 2.4?

[dmoliveira]

Good point @dfdx
I changed the version to this one: https://downloads.apache.org/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz

Now I`m getting:

julia> using Spark

julia> Spark.init()
ERROR: ArgumentError: invalid index: nothing of type Nothing
Stacktrace:
 [1] to_index(::Nothing) at ./indices.jl:297
 [2] to_index(::Array{String,1}, ::Nothing) at ./indices.jl:274
 [3] to_indices at ./indices.jl:325 [inlined]
 [4] to_indices at ./indices.jl:322 [inlined]
 [5] getindex at ./abstractarray.jl:980 [inlined]
 [6] load_spark_defaults(::Dict{Any,Any}) at /root/.julia/packages/Spark/3MVGw/src/init.jl:55
 [7] init() at /root/.julia/packages/Spark/3MVGw/src/init.jl:5
 [8] top-level scope at REPL[2]:1

What is your setup?

[dfdx]

I'm using Julia 1.5 and all the default settings, which result in Spark 2.4.7.

How do you set up Spark version? Do you use SPARK_CONF or SPARK_HOME environment variables for this?

[dfdx]

Closing as outdated.