Comments (5)
I ran into the same problem with StackOverflowException and the problem was solved by switching to Java 8. I think there should be a warning when newer Java is used by accident, it was quite hard to find what caused the problems 😅
from spark.jl.
The most likely case is using Spark 3, which we didn't test against yet. Can you try it with Spark 2.4?
from spark.jl.
Good point @dfdx
I changed the version to this one: https://downloads.apache.org/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz
Now I`m getting:
julia> using Spark
julia> Spark.init()
ERROR: ArgumentError: invalid index: nothing of type Nothing
Stacktrace:
[1] to_index(::Nothing) at ./indices.jl:297
[2] to_index(::Array{String,1}, ::Nothing) at ./indices.jl:274
[3] to_indices at ./indices.jl:325 [inlined]
[4] to_indices at ./indices.jl:322 [inlined]
[5] getindex at ./abstractarray.jl:980 [inlined]
[6] load_spark_defaults(::Dict{Any,Any}) at /root/.julia/packages/Spark/3MVGw/src/init.jl:55
[7] init() at /root/.julia/packages/Spark/3MVGw/src/init.jl:5
[8] top-level scope at REPL[2]:1
What is your setup?
from spark.jl.
I'm using Julia 1.5 and all the default settings, which result in Spark 2.4.7.
How do you set up Spark version? Do you use SPARK_CONF or SPARK_HOME environment variables for this?
from spark.jl.
Closing as outdated.
from spark.jl.
Related Issues (20)
- read sequence file from HDFS HOT 8
- Class Not Found org/apache/spark/SparkConf when initialising sparkcontext HOT 2
- StackOverflowError thrown from the UncaughtExceptionHandler in thread "process reaper"
- Nothing works HOT 4
- load_spark_defaults(d::Dict{Any, Any}) causes error HOT 2
- No SparkContext defined HOT 7
- Registration HOT 3
- New version does not work as stated in docs HOT 6
- DataFrames.jl to Spark dataframe HOT 2
- InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") HOT 10
- Serialization error HOT 30
- I'm lost for a title! Will update when I get clarity :-( HOT 7
- Can we abandon RDD API? HOT 5
- segfault in MacOS HOT 4
- TagBot trigger issue HOT 5
- Use SparkSubmit to initialize JVM (maybe?) HOT 4
- sparkjl jar not found HOT 2
- Does Spark.jl support writing data to HDFS? HOT 8
- An config loadding error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark.jl.