MPI Operator v1alpha2 API Design Proposal

[kubeflow]

MPI Operator v1alpha2 API Design Proposal

[richardsliu]

For things like JobStatus, we should aim to have a common implementation across operators. Please see https://github.com/kubeflow/tf-operator/blob/master/pkg/apis/common/v1beta2/common_types.go.

The pytorch operator is a great example for using the common types and libraries.

@johnugeorge

[johnugeorge]

Yes. As @richardsliu suggested, it would be better if you use common JobStatus. It would be easier to implement at this point. We are aiming to reach a point where all operators have common JobStatus type so that other components can take use of this.

In Pytorch operator, See https://github.com/kubeflow/pytorch-operator/blob/master/pkg/apis/pytorch/v1beta1/types.go#L41

[Jeffwan]

+1 on common job status. I am recently working on kubebench and found it's better to have common job status to orchestrate workflow to avoid extra control on job status. Otherwise, I have to define job finish condition based on different JobStatus based on different DL framework operator.

[terrytangyuan]

Thanks @richardsliu @johnugeorge @Jeffwan for the suggestion. I totally agree that we can reuse JobStatus. Should we extract the common JobStatus out instead of importing from tf-operator? I don't think it makes sense for pytorch-operator and mpi-operator to depend on tf-operator. I am more than happy to help move them out to a common repo as part of the process.

[madhukarkm]

+1 on using the common job status and spec. Looking more: (a) ReplicaStatus in both lists only the number of Active/Running, Succeeded and Failed replicas; how about other states like Pending when some replicas are waiting for resources to be scheduled. (b) BackoffLimit would be a good candidate to move into common RestartPolicy (but is probably a broader change).

[johnugeorge]

@terrytangyuan we have been thinking about it for sometime. It didn't happen because of effort to be put into it. We also have a JobController which was designed to share features across operators. It should be also moved to a common repo. (See https://github.com/kubeflow/tf-operator/tree/master/pkg/common/jobcontroller )

@richardsliu Please add your thoughts too

[jian-he]

+1 to separate the common module out.
we also have a project (a DL framework) thinking to reuse the common job controller logic.

[terrytangyuan]

Thanks for everyone's feedback! I've created a PR in #95 for the initial v1alpha2 MPIJob API Spec based on everyone's feedback. Please take a look and let me know if there's anything else that needs to be addressed.

Note that I copied the common types from tf-operator for now since I don't believe mpi-operator should depend on tf-operator. We can switch to use a common repo once it's ready. We should continue this discussion. @richardsliu @jlewi Please also add your thoughts on this.

[jlewi]

I'll defer to @richardsliu @johnugeorge since they have largely been driving operators these days.

[johnugeorge]

We can take this up after 0.5 release.