Comments (10)
For things like JobStatus, we should aim to have a common implementation across operators. Please see https://github.com/kubeflow/tf-operator/blob/master/pkg/apis/common/v1beta2/common_types.go.
The pytorch operator is a great example for using the common types and libraries.
from mpi-operator.
Yes. As @richardsliu suggested, it would be better if you use common JobStatus. It would be easier to implement at this point. We are aiming to reach a point where all operators have common JobStatus type so that other components can take use of this.
In Pytorch operator, See https://github.com/kubeflow/pytorch-operator/blob/master/pkg/apis/pytorch/v1beta1/types.go#L41
from mpi-operator.
+1 on common job status. I am recently working on kubebench and found it's better to have common job status to orchestrate workflow to avoid extra control on job status. Otherwise, I have to define job finish condition based on different JobStatus based on different DL framework operator.
from mpi-operator.
Thanks @richardsliu @johnugeorge @Jeffwan for the suggestion. I totally agree that we can reuse JobStatus
. Should we extract the common JobStatus out instead of importing from tf-operator? I don't think it makes sense for pytorch-operator and mpi-operator to depend on tf-operator. I am more than happy to help move them out to a common repo as part of the process.
from mpi-operator.
+1 on using the common job status and spec. Looking more: (a) ReplicaStatus in both lists only the number of Active/Running, Succeeded and Failed replicas; how about other states like Pending when some replicas are waiting for resources to be scheduled. (b) BackoffLimit would be a good candidate to move into common RestartPolicy (but is probably a broader change).
from mpi-operator.
@terrytangyuan we have been thinking about it for sometime. It didn't happen because of effort to be put into it. We also have a JobController which was designed to share features across operators. It should be also moved to a common repo. (See https://github.com/kubeflow/tf-operator/tree/master/pkg/common/jobcontroller )
@richardsliu Please add your thoughts too
from mpi-operator.
+1 to separate the common module out.
we also have a project (a DL framework) thinking to reuse the common job controller logic.
from mpi-operator.
Thanks for everyone's feedback! I've created a PR in #95 for the initial v1alpha2 MPIJob API Spec based on everyone's feedback. Please take a look and let me know if there's anything else that needs to be addressed.
Note that I copied the common types from tf-operator
for now since I don't believe mpi-operator
should depend on tf-operator
. We can switch to use a common repo once it's ready. We should continue this discussion. @richardsliu @jlewi Please also add your thoughts on this.
from mpi-operator.
I'll defer to @richardsliu @johnugeorge since they have largely been driving operators these days.
from mpi-operator.
We can take this up after 0.5 release.
from mpi-operator.
Related Issues (20)
- pod priority was assigned to 0 though the priorityclassname of the podgroup had been assigned HOT 2
- Port conficts will occur when multiple pods dispatched to the same node under hostnetwork. HOT 3
- MPI Job fails on EKS with > 2 instances of 128 core per instance HOT 2
- Update stale examples HOT 2
- MPI-Operator run example failed HOT 8
- Cant get mpijob status when pod template is invalid HOT 9
- which is the latest mpi job definition between mpi-operator and training operator HOT 4
- the object has been modified; please apply your changes to the latest version and try again HOT 8
- When when WaitForWorkersReady is enabled in MPI operator, MPI operator and gang scheduler are in a deadlock HOT 4
- Work with DeepSpeed for large scale training HOT 28
- Replace the plain pod workers with Indexed Job HOT 4
- "make generate" command run failed
- The operator still creates the launcher when launcherCreationPolicy is "WaitForWorkersReady" and suspend is "true"
- Running in a subset of namespaces HOT 8
- Wrong host info in discover_hosts.sh
- NCCL tests example HOT 1
- how could mpijob of mpi operator worker get the hostname of launcher HOT 2
- Connection reset HOT 4
- "cleanPodPolicy: All" does not clean up launcher pod HOT 1
- ttlSecondsAfterFinished for MPIJob, not only launcher HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mpi-operator.