Memory leak in asyncio operation

[govindamagrawal]

Memory leak in asyncio operation

[t-vi]

Hi,

oh oh. This should not happen, thank you for reporting!

What I would do as a first step is to look at process_input, specifically at the end

dlwpt-code/p3ch15/request_batching_jit_server.py

Line 59 in 323de27

return our_task["output"]

Do the following:

  output = our_task["output"]
  for k in our_task.keys():
     del our_task[k]
  return output

(or somesuch).
Most likely, there is a circular reference somewhere, and this would help break it.
You could also call gc.collect() now and then, but the first thing I'd try is to empty the task dictionary to avoid references that keep objects from being deallocated.

Can you please check if that improves things for you?

Best regards

Thomas

[govindamagrawal]

@t-vi thanks for replying. Actually I found out that it was due to large queue size I had given, as a result, it was storing all those tensors and memory was getting full.
Thanks again.

[t-vi]

😅 So now it's gone or just not as bad?