GPUMounter-worker error in k8s v1.23.1

[pokerfacesad]

GPUMounter-worker error in k8s v1.23.1

[cool9203]

ok, this bug is sloved.

use environment and version:

• OS : ubuntu 20.04.1
• k8s version : v1.23.1
• docker-client version : 19.03.13
• docker-server version : 20.10.12
• CRI: docker
• cgroup driver : systemd

i use nvidia k8s-device-plugin,
and i setting "/etc/docker/daemon.json" contant:

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  }
}

---

in code of "/pkg/util/util.go", always pass "cgroupfs" to cgroupDriver when call function GetCgroupName, then will error.
so this bug not k8s version problem!
and pod id have _ in k8s v1.23.1, so don't check _ characters in function NewCgroupName.

so need detect for what use cgroup method of now.
but i'm rookie for golang, so need more time to coding, i will send PR in few day later.

and need edit title? if need will can direct edit.

[pokerfaceSad]

Thanks for your feedback. I will try to fix it.
PRs are also very welcomed!

[pokerfaceSad]

@cool9203 Happy Spring Festival!

Thanks for your efforts. Sorry for waiting so long time.

• The checking of _ is to handle the systemd cgroup driver. But if _ can be involved in pod id, it may be complex to handle. Can you show me some k8s document descriptions about _ in pod id?
  

  GPUMounter/pkg/util/cgroup/cgroup.go

  Lines 33 to 40 in 7036133

  	for _, component := range components {
  	// Forbit using "_" in internal names. When remapping internal
  	// names to systemd cgroup driver, we want to remap "-" => "_",
  	// so we forbid "_" so that we can always reverse the mapping.
  	if strings.Contains(component, "/") || strings.Contains(component, "_") {
  	panic(fmt.Errorf("invalid character in component [%q] of CgroupName", component))
  	}
  	}

• Pass constant cgroupfs is really a bug! It should be configurable.
  

  GPUMounter/pkg/util/util.go

  Line 25 in 7036133

  cgroupPath, err := cgroup.GetCgroupName("cgroupfs", pod, containerID)

[cool9203]

@pokerfaceSad Happy Spring Festival!!
thanks for your reply.

• GPUMounter/pkg/util/cgroup/cgroup.go

  Lines 33 to 40 in 7036133

  	for _, component := range components {
  	// Forbit using "_" in internal names. When remapping internal
  	// names to systemd cgroup driver, we want to remap "-" => "_",
  	// so we forbid "_" so that we can always reverse the mapping.
  	if strings.Contains(component, "/") || strings.Contains(component, "_") {
  	panic(fmt.Errorf("invalid character in component [%q] of CgroupName", component))
  	}
  	}

  
  you right, today i test done, this is can run.
  this is not necessary edit.
  this edit is my test in the beginning.
  my bug is pass cgroupfs in

  GPUMounter/pkg/util/util.go

  Line 25 in 7036133

  cgroupPath, err := cgroup.GetCgroupName("cgroupfs", pod, containerID)

[cool9203]

i got another problem.

• GPUMounter/pkg/server/gpu-mount/server.go

  Lines 124 to 135 in 7036133

  	removeGPUs, err := gpuMountImpl.GetRemoveGPU(targetPod, request.Uuids)
  	if err != nil {
  	Logger.Error("Failed to get remove gpu of Pod: ", targetPod.Name)
  	Logger.Error(err)
  	return nil, err
  	}
  	if len(removeGPUs) == 0 {
  	Logger.Error("Invalid UUIDs: ", request.Uuids)
  	return &gpu_mount.RemoveGPUResponse{
  	RemoveGpuResult: gpu_mount.RemoveGPUResponse_GPUNotFound,
  	}, nil
  	}

  
  in call RemoveGPU, some times get error Invalid UUIDs.
  i track this error, found this is slave pod status is terminating, than pod will delete.
  example:

  kubectl get pod --all-namespaces
  
  NAMESPACE     NAME                                       READY   STATUS        RESTARTS        AGE
  gpu-pool      test460c04d4-slave-pod-bca118              1/1     Terminating   0               30s
  

  GPUMounter/pkg/util/gpu/collector/collector.go

  Line 90 in 7036133

  func (gpuCollector *GPUCollector) UpdateGPUStatus() error {

  
  so updategpu will not found any slave pod.
  then will not get any GPUresource in mounted gpu pod.
  this error maybe only in k8s v1.23.1? or other version occur too?

[cool9203]

#19 (comment)
maybe i solved this.

https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/
seem like from k8s v1.20+, owner pod and slave pod be need in same namespace.
if owner pod and slave pod not in same namespace, slave pod status will is Terminating.
so slave pod namespace need set to same as owner pod namespace.
but i need testing more, i will report testing result.

---

update

my test result:

kubectl get pod -n gpu-pool

NAME                    READY   STATUS    RESTARTS   AGE
test                    1/1     Running   0          3m12s
test-slave-pod-d34ea2   1/1     Running   0          19s

pod/test.yaml

apiVersion: v1
kind: Pod
metadata:
  name: test
  namespace: gpu-pool
  labels:
    app: test
spec:
  containers:
  - name: test
    image: [docker-image]
    resources:
      requests:
        memory: "1024M"
        cpu: "1"
    env:
      - name: NVIDIA_VISIBLE_DEVICES
        value: "none"

kubectl describe pod test-slave-pod-d34ea2 -n gpu-pool

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  4s    default-scheduler  Successfully assigned gpu-pool/test-slave-pod-290964 to rtxws
  Normal  Pulling    3s    kubelet            Pulling image "alpine:latest"
  Normal  Pulled     1s    kubelet            Successfully pulled image "alpine:latest" in 2.563965249s
  Normal  Created    1s    kubelet            Created container gpu-container
  Normal  Started    1s    kubelet            Started container gpu-container

owner pod and slave pod not in same namespace pod event:

Events:
  Type     Reason                    Age   From                          Message
  ----     ------                    ----  ----                          -------
  Normal   Scheduled                 4s    default-scheduler             Successfully assigned gpu-pool/test460c04d4-slave-pod-22d29a to rtxws
  Warning  OwnerRefInvalidNamespace  5s    garbage-collector-controller  ownerRef [v1/Pod, namespace: gpu-pool, name: test460c04d4, uid: a55bc88b-60d1-460f-a7c7-4072fe6a9a2c] does not exist in namespace "gpu-pool"
  Normal   Pulling                   4s    kubelet                       Pulling image "alpine:latest"
  Normal   Pulled                    1s    kubelet                       Successfully pulled image "alpine:latest" in 2.568386225s
  Normal   Created                   1s    kubelet                       Created container gpu-container
  Normal   Started                   1s    kubelet                       Started container gpu-container
  Normal   Killing                   0s    kubelet                       Stopping container gpu-container

can see in this test, pod/test.yaml namespace change to gpu-pool.
now, slave pod status is running, not is terminating.
and check pod event, will get pod is running, not is stopping.
i have test for idle 15 minutes, slave pod will be running, not delete.
and can see to not in same namespace event log, show to does not exist in namespace gpu-pool.
so in k8s v1.20+, slave pod and owner pod must of same namespace.
if not same, slave pod status will be terminating.
and call RemoveGPU service, will show Invalid UUIDs error.

maybe don't use gpu-pool namespace in k8s v1.20+.
slave pod always use owner pod namespace, not use gpu-pool.
this is good idea or not? please give me advice, thanks!!

[pokerfaceSad]

@cool9203
Thank you for revealing this!
The reason why slave pod can't be created in owner pod namspace is #3.
Maybe need some modifications to adpat k8s v1.20+.

[pokerfaceSad]

@cool9203
The bug of constant cgroup driver has been fixed in 163ef7b.
cgroup driver can be set in /deploy/gpu-mounter-workers.yaml by environment variable CGROUP_DRIVER.

[cool9203]

@pokerfaceSad
sorry, i reply late.

@cool9203 The bug of constant cgroup driver has been fixed in 163ef7b. cgroup driver can be set in /deploy/gpu-mounter-workers.yaml by environment variable CGROUP_DRIVER.

thanks your fixed, pass a environment variable in worker.yaml is good idea!

---

@cool9203 Thank you for revealing this! The reason why slave pod can't be created in owner pod namspace is #3. Maybe need some modifications to adpat k8s v1.20+.

i show one solve method in #19 (comment)
in this solve, owner pod and slave pod must be same namespace, like gpu-pool, default, kube-system or other namespace.
and i not set any resource quota.
so like this solve showed, i think owner and slave pod must be same namespace in k8s v1.20+.
what do you think?

[pokerfaceSad]

@cool9203
In fact, slave pods were created in owner pod namespace before a378e39.

However, in a multi-tenant cluster scenario, cluster administrator may use resourse quota feature to limit the resource usage of users.

If GPUMounter create the slave pods in owner pod namespaces, slave pods will consume the resource quota of the user.