Can not use GPUMounter on k8s

[pokerfacesad]

Can not use GPUMounter on k8s

[Crazybean-lwb]

do not have slave pod in my namespace: gpu-pool

[pokerfaceSad]

@liuweibin6566396837
Thanks for your issue.
Show more relevant logs of gpu-mounter-worker(/etc/GPUMounter/log/GPUMounter-worker.log) plz.

[pokerfaceSad]

It seems like that you edit the k8s version in this issue.
What's your k8s version?
In current version, GPUMounter has a known bug on k8s v1.20+ mentioned in #19 (comment).

[Crazybean-lwb]

It seems like that you edit the k8s version in this issue. What's your k8s version? In current version, GPUMounter has a known bug on k8s v1.20+ mentioned in #19 (comment).

thanks for your reply. I have fixed the problem earlier（just make sure env:
- name: NVIDIA_VISIBLE_DEVICES
value: "none"
the problem will be solved.
）

[Crazybean-lwb]

It seems like that you edit the k8s version in this issue. What's your k8s version? In current version, GPUMounter has a known bug on k8s v1.20+ mentioned in #19 (comment).

Now, I met a new bug in cluster
k8s 1.20.11
docker 20.10.10

bug: when I request addgpu, it return "Add GPU Success", however no slaver pod in gpu-pool.
I found some unusual log in worker's pod, show some logs follows:

2022-03-15T13:12:43.240Z INFO collector/collector.go:136 GPU status update successfully
2022-03-15T13:12:46.402Z INFO allocator/allocator.go:59 Creating GPU Slave Pod: base-0-slave-pod-595282 for Owner Pod: base-0
2022-03-15T13:12:46.403Z INFO allocator/allocator.go:238 Checking Pods: base-0-slave-pod-595282 state
2022-03-15T13:12:50.450Z INFO allocator/allocator.go:252 Not Found....
2022-03-15T13:12:50.450Z INFO allocator/allocator.go:277 Pods: base-0-slave-pod-595282 are running
2022-03-15T13:12:50.450Z INFO allocator/allocator.go:84 Successfully create Slave Pod: base-0-slave-pod-595282, for Owner Pod: base-0
2022-03-15T13:12:50.450Z INFO collector/collector.go:91 Updating GPU status
2022-03-15T13:12:50.452Z DEBUG collector/collector.go:130 GPU: /dev/nvidia0 allocated to Pod: xiaoxuan-fbdd-0 in Namespace shixiaoxuan
2022-03-15T13:12:50.452Z DEBUG collector/collector.go:130 GPU: /dev/nvidia1 allocated to Pod: zwbgpu-pytorch-1-6-0 in Namespace zhouwenbiao
2022-03-15T13:12:50.452Z DEBUG collector/collector.go:130 GPU: /dev/nvidia7 allocated to Pod: admet-predict-0 in Namespace liqinze
2022-03-15T13:12:50.452Z DEBUG collector/collector.go:130 GPU: /dev/nvidia5 allocated to Pod: xiaoxuan-test-d2m-0 in Namespace shixiaoxuan
2022-03-15T13:12:50.452Z DEBUG collector/collector.go:130 GPU: /dev/nvidia2 allocated to Pod: bf-dev-2-0 in Namespace baifang
2022-03-15T13:12:50.452Z DEBUG collector/collector.go:130 GPU: /dev/nvidia3 allocated to Pod: bf-dev-2-0 in Namespace baifang
2022-03-15T13:12:50.452Z DEBUG collector/collector.go:130 GPU: /dev/nvidia6 allocated to Pod: minisomdimsanbai-0 in Namespace yangdeai
2022-03-15T13:12:50.452Z INFO collector/collector.go:136 GPU status update successfully
2022-03-15T13:12:50.452Z INFO gpu-mount/server.go:97 Successfully mount all GPU to Pod: base-0 in Namespace: liuweibin

[pokerfaceSad]

Thx for your report.
It seems that you have the unfixed issue mentioned in #19.
GPUMounter can not work well in k8s v1.20+ in current version.