Warning UnexpectedAdmissionErro

[volcano-sh]

Warning UnexpectedAdmissionErro

[WingkaiHo]

If you have use nvidia-smi to hide gpu, index will be change, number of device will be decrease. Such as

nvidia-smi drain -p <id> -m 0

id of gpu devices never delete in scheduler when number gpu decrease
volcano-sh/volcano#2215

[GLL550C]

@Jeffwan

[Thor-wl]

/bug
This bug has also been mentioned in the Volcano wechat group.

[Thor-wl]

/cc @peiniliu @william-wang Can you help for that?

[Thor-wl]

Can you give the reproduce steps and environment? Let me take a try.

[WingkaiHo]

If you have use nvidia-smi to hide gpu, index will be change, number of device will be decrease. Such as

nvidia-smi drain -p <id> -m 0

Or some gpu is damage，it disable by drive, such as

but is still in path dev path

When use nvidia-smi to hide gpu, for example is hide /dev/nvidia0, device plugin init

        deviceByIndex := map[uint]string{}
	for i := uint(0); i < n; i++ {
		d, err := nvml.NewDevice(i)
		check(err)
		var id uint
		_, err = fmt.Sscanf(d.Path, "/dev/nvidia%d", &id)
		check(err)
		deviceByIndex[id] = d.UUID
		// TODO: Do we assume all cards are of same capacity
	}

so device plugint do not have index 0 gpu, but scheduler start gpu index is 0, so if will fail to Warning UnexpectedAdmissionError 79s kubelet Allocate failed due to rpc error: code = Unknown desc = failed to find gpu id, which is unexpected error

[631068264]

same issue volcano-sh/volcano#2701 use Volcano 1.7 and https://github.com/volcano-sh/devices/blob/release-1.0/volcano-device-plugin.yml

I did not use

nvidia-smi drain -p <id> -m 0