Comments (2)
Hi @dpittner. Yeah, we have not supported dynamic executor allocation currently. We will add it support recently.
If you use spark on ray
only, it should be easy to add the support. If you want to use spark on ray couple with estimator API, we should add the support of ray object ownership transfer.
from raydp.
close as stale. This has been supported
from raydp.
Related Issues (20)
- Add shim layer for Spark 3.4.x
- Support/Validate Ray 2.4.0 Release
- Performance benchmark on RayDP v.s. Spark HOT 1
- Data ownership transfer tests fail in Ray 2.4.0
- Wanted to configure spark to use mysql as metastore instead of derby hive metastore which is default HOT 5
- enable_hive not working with raydp HOT 24
- Threads hang on getting from object store when calling `to_spark`
- raydp.init_spark fails HOT 3
- Issues with running on Ray Client HOT 3
- raydp start occur null error HOT 1
- Executor lost triggered by JVM OOM can't recover and cause subsequent steps stuck HOT 12
- Support ray 2.3.1 HOT 3
- When submit a script another process start locally in the head node HOT 1
- Add the ability to set the owner of objects when calling _save_spark_df_to_object_store() HOT 1
- Runtime env_vars cannot be passed down to executors HOT 1
- Getting Java gateway process exited before sending its port number during init_spark() HOT 9
- Issue with Running Multiple Spark Jobs Concurrently on Ray Cluster HOT 1
- RayDP Docker Image with Python 3.10 Failing to Build HOT 2
- [stream] does raydp support spark stream data? HOT 1
- Is it possible for raydp to utilize a Spark session created by Apache Livy? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from raydp.