How to reliably free GPU memory after tensorflow/keras inference?

[bioimage-io]

How to reliably free GPU memory after tensorflow/keras inference?

[esgomezm]

I'm a bit lost. Is this something to discuss during the hackathon?

[FynnBe]

I'm a bit lost. Is this something to discuss during the hackathon?

I think so. I'll collect some insights here and if anyone has experience on this please share here so we can have a more informed discussion during the hackathon.

What I found out today about tenforflow: (edit: takes potato out of mouth.. ehem, tensorflow)

For tensorflow it looks a bit like an unsolved problem (e.g. tensorflow/tensorflow#36465 (comment))
Some people take desperate measures that we definitely can't: kill all nvidia processes on gpu
💥 tensorflow/tensorflow#36465 (comment)

Theoretically tf.Session is the solution:
https://github.com/tensorflow/docs/blob/r1.14/site/en/api_docs/python/tf/Session.md#tfsession
There were problems with that related to nvidia drivers (old issue): tensorflow/tensorflow#1727 (comment)

I'll give tf.Session a try and share the code snippet here so we can try on several machienes, etc...

One way to circumvent all of that is to use multiprocessing (would work for any framework):

• e.g. with shared numpy array to get the data back from the inference process: https://stackoverflow.com/a/7908612/17492107 @oeway is there not a way to make this integrated mp approach work on a triton server? no additional software would be needed, we could encapsulate it in the ModelAdapter base (within bioimageio.core's internals)