opendatahub-io/manifests

A repository for Kustomize manifests

manifests's Issues

Plan to upgrade to kubeflow 1.0/1.1?

Hi, are there any plans to resync with Kubeflow main repo and update to 1.0/1.1?

Because my org is planning to try out kubeflow. However, we have an existing OpenShift cluster on AWS. We are considering between spinning up an EKS and use AWS's variant of Kubeflow, vs the effort needed to update this repo to kf1.0 but saves on maintaining another k8s cluster (and also for further plans of moving to a production openshift cluster)?

Any opinions? I am not very familiar with openshift. Would be glad if someone can point to me the main diff for kubeflow in openshift vs k8s? So that I can better evaluate my options? (i.e. what are the key things I might need to change to make it work in openshift?)

I am a contributor to kubeflow and familiar how kf works in a typical kubernetes (i.e. EKS, AKS, ELK, etc. but not familiar with Openshift).

Test KF 1.6 with Servicemesh stack

Test KF1.6 components are running with servicemesh overlay

Jupyter Notebooks KF 1.0

No permissions to list/create notebooks

When I navigate from the central dashboard to try to spawn a notebook (in a profile that was created by kubeflow), I get error messages on the Jupyter launch screen about no permissions to list notebooks and then trying to create a notebook also fails with a similar "no permission to create" message.
Error text is: "User None is not authorized to list kubeflow.org.v1alpha1.poddefaults for namespace: mytestnamespace"

KF 1.0 KFserving

KF 1.0 Pytorch and TF-Job training/serving

• TF Job (training)
• PyTorch (training)

Error while installing with Kubeflow latest instructions on openshift

Following the instructions here: https://www.kubeflow.org/docs/distributions/openshift/install-kubeflow/

I get the following error after executing

while ! kustomize build openshift/example/istio | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

$ while ! kustomize build openshift/example/istio | oc  apply -f -; do echo "Retrying to apply resources"; sleep 10; done
Error: multiple matches for rbac.authorization.k8s.io_v1_Role|~X|leader-election-role:
  [rbac.authorization.k8s.io_v1_Role|kubeflow|notebook-controller-leader-election-role
 rbac.authorization.k8s.io_v1_Role|kubeflow|profiles-leader-election-role
]
error: no objects passed to apply
Retrying to apply resources

openshift version: 4.9
Kustomize version : 3.2.0
kubectl version : 1.20.0 (as i understand it, 1.20+ comes prepackaged with kustomize 4 which is incompatible)

In spawner UI we are not able to change GPU instance

Hello Team,
We have odh installed on openshift 4.11. In spawner UI we are not able to change GPU instance. It show 0 and says "All GPUs are currently in use, try again later." while we still have GPU nodes in the cluster. Can you help here?
Thanks

Add documentation for 0.5.3 JH users and single profile

Create a new notebook image for KF 1.3

Update the tensorflow notebook image for KF 1.3 per request here: kubeflow#1811

Create OCP Stack in KF

Testing done on OCP 4.3/Kub 1.16. Code is in this PR : https://github.com/kubeflow/manifests
Issues:

1. Profiles app is not setting $namespaces as seen in error from validating webhook. Looking at manifest/profiles/base_v3 it seems not to specify a variable $namespace as is specified in base/kustomization.yaml file. DONE-FIXED

WARN[0032] Encountered error applying application profiles:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout859943078": admission webhook "pilot.validation.istio.io" denied the request: configuration is invalid: domain name "$(namespace).svc.cluster.local" invalid (label "$(namespace)" invalid)  filename="kustomize/kustomize.go:266"
WARN[0032] Will retry in 6 seconds.                      filename="kustomize/kustomize.go:267"

2. Cache-server pod crashing on "secret "webhook-server-tls" not found"
3. When creating a notebook server, fsgroup is set to 100. DONE-FIXED
4. When launching a notebook server we get error "no healthy upstream" DONE-FIXED
5. ml-pipeline pod crashes withe logs DONE-FIXED

 config.go:37] Please specify flag PROFILES_KFAM_SERVICE_HOST

This is due to the fact this needs profiles installed, and profiles only works when installed in top kustomization file
6. Getting to Pipeline page from Kubeflow dashboard gives error before: kubeflow/kubeflow#5271
upstream connect error or disconnect/reset before headers. reset reason: connection failure
to solve this set

tls:
      mode: DISABLE

in destination rule for ml-pipeline-ui. DONE-FIXED

• tf-jobs,pytorch,katib,metadata (Vasek)

• profiles, pipelines (Landon)

• Seldon (Juana)

• JH web (Maros)

• Notebook Controller

• istio (Juana)

• cert-manager

Training Operator crashes with RBAC error messages

The training-operator pod goes into a crash loop backoff state with the following error message:

W0207 14:44:50.619030       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:50.619067       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:44:50.619099       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

E0207 14:44:50.619108       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

W0207 14:44:50.620991       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

E0207 14:44:50.621013       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

W0207 14:44:50.624997       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

E0207 14:44:50.625010       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.MPIJob: failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

W0207 14:44:50.625033       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:50.625039       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.Role: failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:44:51.565690       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

E0207 14:44:51.565725       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

W0207 14:44:51.572523       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

E0207 14:44:51.572545       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

W0207 14:44:51.578284       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:51.578322       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.Role: failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:44:51.665981       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

E0207 14:44:51.666004       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.MPIJob: failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

W0207 14:44:51.727345       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:51.727368       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:44:53.301661       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:53.301688       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.Role: failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:44:53.350249       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

E0207 14:44:53.350281       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.MPIJob: failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

W0207 14:44:53.606696       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

E0207 14:44:53.606720       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

W0207 14:44:53.695195       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

E0207 14:44:53.695222       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

W0207 14:44:54.562282       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:54.562308       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:44:57.147413       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

E0207 14:44:57.147436       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

W0207 14:44:57.902870       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

E0207 14:44:57.902893       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.MPIJob: failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

W0207 14:44:58.031891       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

E0207 14:44:58.031913       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

W0207 14:44:58.176967       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:58.176985       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.Role: failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:44:58.875634       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:44:58.875669       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:45:04.961134       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

E0207 14:45:04.961155       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

W0207 14:45:06.304673       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:45:06.304695       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.Role: failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:45:06.406679       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

E0207 14:45:06.406719       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

W0207 14:45:06.522674       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

E0207 14:45:06.522695       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.MPIJob: failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

W0207 14:45:10.320465       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:45:10.320495       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:45:24.376208       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:45:24.376234       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:45:25.495874       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:45:25.495898       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.Role: failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:45:26.169903       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

E0207 14:45:26.169930       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

W0207 14:45:29.954705       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

E0207 14:45:29.954728       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.MPIJob: failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

W0207 14:45:31.217840       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

E0207 14:45:31.217862       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

W0207 14:46:04.904302       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:46:04.904325       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.Role: failed to list *v1.Role: [roles.rbac.authorization.k8s.io](http://roles.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "roles" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:46:05.216038       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

E0207 14:46:05.216058       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "configmaps" in API group "" at the cluster scope

W0207 14:46:05.638956       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

E0207 14:46:05.638977       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.MPIJob: failed to list *v1.MPIJob: [mpijobs.kubeflow.org](http://mpijobs.kubeflow.org/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "mpijobs" in API group "[kubeflow.org](http://kubeflow.org/)" at the cluster scope

W0207 14:46:07.586310       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

E0207 14:46:07.586332       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: [rolebindings.rbac.authorization.k8s.io](http://rolebindings.rbac.authorization.k8s.io/) is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "rolebindings" in API group "[rbac.authorization.k8s.io](http://rbac.authorization.k8s.io/)" at the cluster scope

W0207 14:46:16.687632       1 reflector.go:324] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

E0207 14:46:16.687654       1 reflector.go:138] pkg/mod/[k8s.io/[email protected]/tools/cache/reflector.go:167](http://k8s.io/[email protected]/tools/cache/reflector.go:167): Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeflow:training-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope

The openshift example kustomize renders the following ClusterRole for the training operator:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: training-operator
  name: training-operator
rules:
- apiGroups:
  - kubeflow.org
  resources:
  - tfjobs
  - mxjobs
  - pytorchjobs
  - xgboostjobs
  - tfjobs/status
  - pytorchjobs/status
  - mxjobs/status
  - xgboostjobs/status
  - tfjobs/finalizers
  - mxjobs/finalizers
  - pytorchjobs/finalizers
  - xgboostjobs/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - endpoints
  - events
  - pods/finalizers
  - services/finalizers
  - endpoints/finalizers
  - events/finalizers
  verbs:
  - '*'
- apiGroups:
  - apps
  - extensions
  resources:
  - deployments
  - deployments/finalizers
  verbs:
  - '*'
- apiGroups:
  - scheduling.volcano.sh
  resources:
  - podgroups
  - podgroups/finalizers
  verbs:
  - '*'

The upstream training-operator cluster role renders the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: training-operator
  name: training-operator
rules:
- apiGroups:
  - kubeflow.org
  resources:
  - mpijobs
  - tfjobs
  - mxjobs
  - pytorchjobs
  - xgboostjobs
  - mpijobs/status
  - tfjobs/status
  - pytorchjobs/status
  - mxjobs/status
  - xgboostjobs/status
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - endpoints
  - events
  verbs:
  - '*'
- apiGroups:
  - apps
  - extensions
  resources:
  - deployments
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - pods/exec
  verbs:
  - create
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  - rolebindings
  verbs:
  - create
  - list
  - watch
  - update
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - serviceaccounts
  verbs:
  - create
  - list
  - watch
  - update
- apiGroups:
  - scheduling.volcano.sh
  resources:
  - podgroups
  verbs:
  - '*'

It appears that several new resources have been added to the RBAC which are getting overwritten in the strategic merge here:

manifests/openshift/openshiftstack/application/training-operator/base/kustomization.yaml

Line 10 in 2addde1

- ./configs/addfinalizer-training.yaml

mkdir /argo/outputs: permission denied error

When running example pipeline, e.g. "artifact-passing", get the error:

“child 'artifact-passing-b2j7k-1148074540' failed”
“executor error: mkdir /argo/outputs: permission denied”.

Workaround is:

oc adm policy add-scc-to-user anyuid -z pipeline-runner -n kubeflow

I think this bug is to do with using the k8sapi executor and arg?  It has been fixed in a recent version of argo, which should be fixed in KF 1.0 (although I could not verify this). Please correct me if I'm wrong.

Argo example with empyDir:

# This example demonstrates the ability to pass artifacts
# from one step to the next.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  volumes:
  - name: workdir
    emptyDir: {}
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["sleep 1; cowsay hello world | tee /dir/hello_world.txt"]
      volumeMounts:
      - name: workdir
        mountPath: /dir
    outputs:
      artifacts:
      - name: hello-art
        path: /dir/hello_world.txt

  - name: print-message
    inputs:
      artifacts:
      - name: message
        path: /dir/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /dir/message"]
      volumeMounts:
      - name: workdir
        mountPath: /dir

Notebook-controller is still using our custom v0.7.0 image

The openshift overlay is still pointing to the v0.7.0 custom image. The problem that we were working around has been fixed via our upstream contribution: kubeflow/kubeflow#4782 so ideally, we should not need this overlay at all anymore.

Profiles controller still using custom v0.7 image

We're using an overlay to get our custom image here quay.io/kubeflow/profile-controller:v0.7.0, but we need to sort out if we still need that change for KF 1.0. If it is needed, we would need a new custom image build and update our overlay.

kfctl_openshift.yaml services not create and routed

Hello,

I have deployed successfully
https://github.com/opendatahub-io/manifests/blob/v1.0-branch-openshift/kfdef/kfctl_openshift.yaml
on OpenShift 4.4.20, but none of the associated routes have been created (ie. centraldashboard, katib-ui etc)

Has anyone the same issue with this yam file?

Rgds,
FM

[KF 1.5] Add ServiceMesh stack

Rebase 1.5 branch to add ServiceMesh resources

KF 1.0 Pipelines

Update OCP KF documentation to relect ODH 0.7 and KF 1.0 support

https://github.com/kubeflow/website/pull/2072/files

Update Pytorch operator image

Latest version of Pytorch image should have the fixes introduced by OpenShift overlay

Kubeflow 1.6 Multi-user install on OCP

Hello Team,
is Kubeflow 1.6 multi-user installation supported on RHOCP 4.9+ or is it only available in single-user mode? What we are seeing is that with some flows we wanted to run, e.g. KFP Tekton API is complaining about being launched in single-user mode, so that is where this question comes from.

Adding some clarification about what is supported and what isn't into Kubeflow's OpenShift installation guide would be helpful.

Thanks for all the good work!

Seldon CRD fails to deploy on OpenShift 4.3

OpenShift 4.3 comes with an update to Kubernetes 1.16 which results in some incompatibilities - this time for Seldon CRD to fail to be applied.

It is already known to Kubeflow community - kubeflow#375

@cliveseldon I see your changes in upstream basically resolve to replacing the resources with new version in master, what would you suggest to do here as we would like to keep 0.7 working on OCP 4.3 (i.e. k8s 1.16)?

Question: is there a working Kubeflow 1.0 kfdef for ServiceMesh?

I am looking at this kfdef https://github.com/opendatahub-io/manifests/blob/v1.0-branch-openshift/kfdef/kfctl_openshift_servicemesh.yaml, but apparently this does not work as it misses memberroll application as well as no servicemesh overlay in istio/istio. Also the istio release installed through the service mesh now in OCP already has the clusterrbacconfig deprecated.

So is there a working version of kfdef that actually supports running kubeflow 1.0 with the istio installed through service mesh operator from OCP?

Thanks.

cc @animeshsingh

Is there a plan to tag the KF1.5 release?

Hi All
Consider KF1.5 is done since #84 was closed a couple of days ago.
But I just explored the tags, it seems 'v1.5.0-rc.2' is the latest??
I think it would be nice to have a tag for the KF1.5 release, thanks.

May some of the paths no longer available

The recent update of servicemesh made changes to paths,but it seems this kfdef still points to the old paths.
Is there a plan to update?

https://github.com/opendatahub-io/manifests/blob/a1286a179e672030daad3ac1026c19683a9c132e/openshift/kfdef/kfctl_openshift_v1.5.0_odh.yaml

Broken link in readme

The link from

https://github.com/opendatahub-io/manifests/ to https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#bespoke-configuration is broken.

new url: https://kubernetes-sigs.github.io/kustomize/api-reference/glossary/#bespoke-configuration

KF 1.0 Jupyter Notebook server gives error ```User [email protected] is not authorized to list .v1.persistentvolumeclaims for namespace: anonymous```

This related to KF issue kubeflow/kubeflow#4731 . To reproduce install KF1.0, login with any username and try to launch a notebook server

KF Manifest CI for Openshift Stack- run KF manifest on OCP clusters

ETA 2wks

[KF 1.5] Test KServe on OpenShift 4.8

KServe was added to the manifests starting KF 1.5. This issue involves testing the component on OpenShift to ensure that Kserve can be added to KfDef

Jupyter web app showing our cusotm build TF 0.7.0 image in dropdown list

The openshift overlay is still configuring to use our old notebook image.
Two possibilities:

1. We don't need that overlay anymore (assuming that there's no other config we want to change and that the standard images work...which they should since there was an upstream fix to the permissions problem that we were working around)
2. We need to rebuild those images to include the permission fix and update the openshift overlay for jupyter-web-app

Upgrade Istio to 1.3.1 for KF Serving to work

[KF 1.5] CSI-s3 Storage Integration in kfp-tekton fails to deploy.

KFP-Tekton v1.1.0 added support for csi-S3 storage. https://github.com/kubeflow/kfp-tekton/releases/tag/v1.1.0

However, the kfp-csi-s3- pods fail to deploy on OpenShift 4.8 and throw the following error:

MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /var/data/kubelet/plugins_registry is not a directory

Add KFServing

Enable KF Serving on. Openshift. Estimate 5 days

Migrate Openshift changes from overlays to stacks to conform to Kustomize v3

Per these KF documentations
https://docs.google.com/document/d/1jBayuR5YvhuGcIVAgB1F_q4NrlzUryZPyI9lCdkFTcw/edit#
kubeflow#1062
example: https://github.com/kubeflow/manifests/pull/1246/files

KF 1.0 Metadata db not installing

When installing metadata db with KF 1.0. We get this error

error evaluating kustomization manifest for metadata Error no matches for OriginalId apps_v1_Deployment|~X|db; no matches for CurrentId apps_v1_Deployment|~X|db; failed to find unique target for patch apps_v1_Deployment|db  filename="kustomize/kustomize.go:175"
Error: failed to apply:  (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize:  (kubeflow.error): Code 500 with message: error evaluating kustomization manifest for metadata Error no matches for OriginalId apps_v1_Deployment|~X|db; no matches for CurrentId apps_v1_Deployment|~X|db; failed to find unique target for patch apps_v1_Deployment|db

Release of KF 1.5 on OpenShift

This is tracking issue for any updates to the OpenShift stack for KF 1.5

[KF 1.5] Example PipelineRun failed to execute

Demo Flip coin Pipeline failed with KF 1.5 with Forbidden access error

Possible fix: Update SCCs kubeflow-anyuid-scc.yaml

Question: Multi-user, auth-enabled support?

Working through running Kubeflow on OpenShift and so far so good, but I have a question, is multi-user with auth-enabled supported when running in OpenShift? Thanks, Ryan.

KF 1.0 Seldon

[Pipeline] kf pipeline incorrectly uses `cluster-admin` for one of it's components

In the persistent-agent component of pipelines, the clusterrole cluster-admin is incorrectly being used. This was fixed in upstream recently: commit
Can we get the patch in odh manifests as well?

KF 1.0 istio cluster-local-gateway crashes

We need to add cluster-local-gateway to our kfctl-openshift manifest to enable internal routing that is needed by knative

KF 1.0 Katib

[KF 1.5] Remove references to mpi-job references from KFDef

MpiJob resources are moved to the training-operator component. Remove all references of mpi-job from the OpenShift stack

pods are not getting created while installing opendatahub on k8s minikube

Issue with cert-manager-webhook service selectors

Creation of cert objects such as a certificate or issue are intermittently failing with error messages such as:

Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=10s": dial tcp 10.129.2.30:10250: connect: connection refused

When viewing the cert-manager-webhook service, it appears that the cert-manager-webhook is selecting all three cert-manger pods and not just the cert-manager-webhook pod. This is causing any object creation requests to be round robined to all three pods and only one out of every three object creations succeed.

The original selectors on the service object are as follows:

spec:
  selector:
    app.kubernetes.io/name: webhook
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/component: "webhook"

The original deployment/pod labels are as follows:

spec:
  template:
    metadata:
      labels:
        app: webhook
        app.kubernetes.io/name: webhook
        app.kubernetes.io/instance: cert-manager
        app.kubernetes.io/component: "webhook"
        app.kubernetes.io/version: "v1.5.0"

The kustomization.yaml file located here includes the following:

commonLabels:
  app.kubernetes.io/component: cert-manager
  app.kubernetes.io/name: cert-manager
  kustomize.component: cert-manager

The service after being rendered through kustomize appears as follows:

spec:
  selector:
    app.kubernetes.io/component: cert-manager
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/name: cert-manager
    kustomize.component: cert-manager

The deployment after being redned through kustomize appears as follows:

spec:
  template:
    metadata:
      labels:
        app: webhook
        app.kubernetes.io/component: cert-manager
        app.kubernetes.io/instance: cert-manager
        app.kubernetes.io/name: cert-manager
        app.kubernetes.io/version: v1.5.0
        kustomize.component: cert-manager

The inclusion of the common labels causes the original name and component to be overwritten for all pods in this kustomization file making it unable to correctly select the right pod.

Kubectl apply doesn't work with several CRDs in Kubeflow

Kubeflow applications are successfully created when deployed through Open Data Hub Operator. However, several CRDs failed on update. This results in failed reconciliation.

CRDs of the following components fail to update:

• Cert-manager 1.5
• Knative
• Training Operator

This failure is observed after moving to v1 of API customresourcedefinition.apiextensions.k8s.io .

Authentication while using kubeflow

Hi,
I am new to Openshift and Kubeflow. I have installed the Kubeflow on Openshift using the manifiest in the Kfdef folder. I am wondering if there is any way to add authentication to restrict the access to istio-ingress gateway and allow multiple users to use the kubeflow.

Test KF 1.6 on OpenShift 4.9

Test Kubeflow components on OpenShift as part of distribution testing.

Update https://github.com/opendatahub-io/manifests/tree/v1.6-branch-openshift/openshift for any OpenShift specific changes.

SPIKE Investigate ODH + KF 1.0 Kubeflow mixing

• Verify TF Job
• Create additional KFDef
• Create a documentation PR for odh.io
• Verify Pytorch

Continuous rebasing of manifest repo with upstream

We need to have a weekly plan to rebase with upstream KF manifest branches: master, 0.7, 1.0 etc. Investigate an automation for this