Comments (10)
Spark.jl is currently in a very early stage and supports only very basic operations like map
, map_partitions
, reduce
and collect
, and works only with byte arrays. So right now for any serious production or research code I would suggest using PySpark or SparkR depending on your preferences. Nevertheless, you can leave your requests for features that are most important for you so that I could set up priorities.
However, if you are brave enough to try Sparta.jl, here's a short instruction to get you started:
-
Clone the project: from Julia REPL, run
Pkg.clone("https://github.com/dfdx/Sparta.jl")
. -
Build Java part of Sparta.jl: go to
<sparta-root>/jvm/sparta
and runmvn clean package
(assuming you have Maven installed on your system) . -
Write and run sample program, e.g. such one:
using Sparta
sc = SparkContext()
rdd = text_file(sc)
collect(rdd)
Feel free to ask any questions, submit issues or just blame me for not working functionality :)
from spark.jl.
Also note that Spark and HDFS are best suited for working with big data. If you just need heavy computations on relatively small data, consider using Julia's native ClusterManager and all great tools for parallel computing in Julia.
from spark.jl.
Followed the steps mentioned to install. mvn clean package was successful with the following last few lines:
[INFO] Attaching shaded artifact.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21:16 min
[INFO] Finished at: 2015-08-29T00:37:30+05:30
[INFO] Final Memory: 76M/755M
[INFO] ------------------------------------------------------------------------
But sc = SparkContext() fails as in below:
julia> using Sparta
julia> sc = SparkContext()
ERROR: SparkContext not defined
from spark.jl.
Have you pulled latest master? I pushed some fixes including exporting SparkContext
recently, so you could miss it. Just go to ~/.julia/v0.3/Sparta/
and run git pull origin master
.
from spark.jl.
Yes I pulled the latest master and re-complied using "mvn clean package". Apart from Sparta package do I need to install some more packages which Sparta is in-turn using ? I suspect this is the case. Pls see below:
julia> using IJulia
julia> using BinDeps
julia> using Sparta
ERROR: Docile not found
in require at loading.jl:47
in include at boot.jl:245
in include_from_node1 at loading.jl:128
in include at boot.jl:245
in include_from_node1 at loading.jl:128
in reload_path at loading.jl:152
in _require at loading.jl:67
in require at loading.jl:51
while loading C:\Users\gagands.julia\v0.3\Sparta\src\core.jl, in expression starting on line 2
while loading C:\Users\gagands.julia\v0.3\Sparta\src\Sparta.jl, in expression starting on line 15
julia> using Docile
ERROR: Docile not found
in require at loading.jl:47
julia>
from spark.jl.
Correct, Sparta additionally uses Docile
, JavaCall
and Iterators
packages. I pushed fix to master, but for you it should be easier just to add these packages manually:
Pkg.add("Docile")
Pkg.add("JavaCall")
Pkg.add("Iterators")
Since this package is still in a very early stage, most likely you will get many more issues. If you could describe what you are working on, I would try to adjust current code for your immediate needs and maybe develop parts that are most critical for your use case.
from spark.jl.
Hi, Thanks and apologies for a late reply. I was travelling. I am working on implementing Monte Carlo analytics using Julia working on Spark nodes. I wish to use the compute nodes exposed by Spark for Parallel Monte Carlo Analytics. Will clone the latest master of Sparta and do some research on how this can be done. Thanks.
from spark.jl.
I'm closing this issue as README now has instructions for building and running the project.
from spark.jl.
Googling for "Sparta.jl" I only found this. Just to be sure, is that this project renamed? Maybe README should say.
from spark.jl.
Yes, this repository had the name "Sparta.jl" originally, but was renamed to "Spark.jl" before the first registered version. I wonder why you were looking for "Sparta.jl"? That name is really old, so it might be better to update references in the source you used.
from spark.jl.
Related Issues (20)
- read sequence file from HDFS HOT 8
- Class Not Found org/apache/spark/SparkConf when initialising sparkcontext HOT 2
- StackOverflowError thrown from the UncaughtExceptionHandler in thread "process reaper"
- Nothing works HOT 4
- load_spark_defaults(d::Dict{Any, Any}) causes error HOT 2
- No SparkContext defined HOT 7
- Registration HOT 3
- New version does not work as stated in docs HOT 6
- DataFrames.jl to Spark dataframe HOT 2
- InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") HOT 10
- Serialization error HOT 30
- I'm lost for a title! Will update when I get clarity :-( HOT 7
- Can we abandon RDD API? HOT 5
- SparkContext giving StackOverflowError HOT 5
- segfault in MacOS HOT 4
- TagBot trigger issue HOT 5
- Use SparkSubmit to initialize JVM (maybe?) HOT 4
- sparkjl jar not found HOT 2
- Does Spark.jl support writing data to HDFS? HOT 8
- An config loadding error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark.jl.