ListAndWatch failed when managing large memory GPU such as NVIDIA Telas V100

[volcano-sh]

ListAndWatch failed when managing large memory GPU such as NVIDIA Telas V100

[Thor-wl]

Thanks for your report and debug. The debug is meaningful and we will fix it as soon as possible.

[zzr93]

100MB per block may work fine. Inference services usually cost hundreds to thousands MB memory(train services usually cost much more than this scale), so we actually do not care memory fragments which are less than 100MB.

IC. I'll take this issue to the weekly meeting for discussion. Are you glad to share your ideas in the meeting?

My pleasure, I will be in.

[jasonliu747]

See you 15:00.

Awww that's sweet.🥺

[Thor-wl]

Request more voice about how much should be considered as a block(default is 1M) which is suitable for all specified GPU cards.

[zzr93]

100MB per block may work fine. Inference services usually cost hundreds to thousands MB memory(train services usually cost much more than this scale), so we actually do not care memory fragments which are less than 100MB.

[Thor-wl]

100MB per block may work fine. Inference services usually cost hundreds to thousands MB memory(train services usually cost much more than this scale), so we actually do not care memory fragments which are less than 100MB.

IC. I'll take this issue to the weekly meeting for discussion. Are you glad to share your ideas in the meeting?

[Thor-wl]

See you 15:00.

[lakerhu999]

Is this issue resolved at present？

[Thor-wl]

Is this issue resolved at present？

Not yet. We are considering for a graceful way to make the fix without modifing the gRPC directly.

[lakerhu999]

Any update for this issue?

[Thor-wl]

Any update for this issue?

Not yet now. I'm sorry for developing another feature recently. Will fix that ASAP.

[lakerhu999]

It's still a bug in our product as same as this issue, if fixed, please close this issue.

[Thor-wl]

It's still a bug in our product as same as this issue, if fixed, please close this issue.

OK, it's still on the way. I'll close the issue after the bug is fixed.

[pauky]

How is this going?

[shinytang6]

#22 may resolve this issue

[XueleiQiao]

我们这个最新的镜像公网上有发布吗？@shinytang6