Comments (5)
So to me, the biggest reason to use Spark.jl is the ability to run Julia UDFs in a distributed fashion. That is, we use Spark to partition the data among nodes, and run a Julia function on the local data in each node. If that is easy and fast, we probably do not care whether the RDD or Dataset features are used.
from spark.jl.
A quick update on UDFs. Although a UDF in Spark can be implemented a single class which evaluates its inputs, for UDFs that call external processes a more sophisticated handling is needed. In particular, PythonUDF
implements Expression
interface and can be included into logical query plan, but when Spark translates the logical plan to a physical one (either during analysis, or optimization - I'm not sure yet), PythonUDF
is replaced with something like ArrowEvalPythonExec
which applies serialized Python functions to the input of type Seq[InternalRow]
.
Translation from a logical to a physical plan is implemented using strategies (instances of Strategy == SparkStrategy
class). For Python, it's PythonEvals
object. As far as I understand, all activated strategies are listed in SparkPlanner.strategies
sequence. At this point I'm not sure if it's possible to alter the list of strategies from the external library, so we may need to add a strategy for Julia directly to Spark source code.
Contributing directly to Apache Spark implies we commit to supporting the high quality of the code now and later, as well as closely following all the changes in new releases. Even without it, messing up with Spark's query plans sounds like a huge time investment. Now I want to step back and reconsider the problems we solve with Spark.jl, and whether there's an easier way to achieve them in the long run.
Thoughts and opinions are welcome.
from spark.jl.
At work we use primarily PySpark which benefits greatly from the DataFrame API, I've used plain RDDs only a couple of times because of that. If Spark.jl supported the DataFrame API, it would be easier for me to try to promote it to my colleagues:)
from spark.jl.
@Drvi Note that the current version of Spark.jl already supports the most popular DataFrame functions, including select
, group_by
, join
, etc. Other functions are usually easy to add by request. The only big issue is the UDFs, for the reasons I described above.
from spark.jl.
RDD API has been discontinued.
from spark.jl.
Related Issues (20)
- read sequence file from HDFS HOT 8
- Class Not Found org/apache/spark/SparkConf when initialising sparkcontext HOT 2
- StackOverflowError thrown from the UncaughtExceptionHandler in thread "process reaper"
- Nothing works HOT 4
- load_spark_defaults(d::Dict{Any, Any}) causes error HOT 2
- No SparkContext defined HOT 7
- Registration HOT 3
- New version does not work as stated in docs HOT 6
- DataFrames.jl to Spark dataframe HOT 2
- InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") HOT 10
- Serialization error HOT 30
- I'm lost for a title! Will update when I get clarity :-( HOT 7
- SparkContext giving StackOverflowError HOT 5
- segfault in MacOS HOT 4
- TagBot trigger issue HOT 5
- Use SparkSubmit to initialize JVM (maybe?) HOT 4
- sparkjl jar not found HOT 2
- Does Spark.jl support writing data to HDFS? HOT 8
- An config loadding error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark.jl.