Comments (1)
kira-lin
commented on June 9, 2024
Hi @gbraes,
Thanks for using RayDP! I'm afraid that I don't understand your issue very well. I've never used ray job API before. What's the expected behavior? Also I notice you're using Ray 2.6.0 and RayDP 1.5.0, but RayDP 1.5.0 does not provide support for Ray 2.6.0. Have you tried your script using ray client?
Also, I am not sure if that means that pyspark is running locally in the head node and not in the cluster.
You can visit Ray's dashboard to see if RayDP actors are distributed to all nodes. In addition, you can use placement group to ensure they are distributed, otherwise it might happen that executors are on the same node because it has enough resources
from raydp.
Related Issues (20)
- Threads hang on getting from object store when calling `to_spark`
- raydp.init_spark fails HOT 3
- Issues with running on Ray Client HOT 3
- raydp start occur null error HOT 1
- Executor lost triggered by JVM OOM can't recover and cause subsequent steps stuck HOT 12
- Support ray 2.3.1 HOT 3
- Add the ability to set the owner of objects when calling _save_spark_df_to_object_store() HOT 1
- Runtime env_vars cannot be passed down to executors HOT 1
- Getting Java gateway process exited before sending its port number during init_spark() HOT 9
- Issue with Running Multiple Spark Jobs Concurrently on Ray Cluster HOT 1
- RayDP Docker Image with Python 3.10 Failing to Build HOT 2
- [stream] does raydp support spark stream data? HOT 1
- Is it possible for raydp to utilize a Spark session created by Apache Livy? HOT 1
- Fix OpenSSF issues
- Fix CVE issues
- install raydp==1.6.0 with pip rasied the "More than one SparkShimProvider found" error
- raydp.init_spark fails HOT 1
- How to develop in intellij?
- how to init raydp resource HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from raydp.