Nearly 0% GPU utilization for a large majority of time during inference

[google-deepmind]

Nearly 0% GPU utilization for a large majority of time during inference

[cdoersch]

You're seeing compile time, not runtime. Every time the size of the input tensors changes (different resolution, different number of frames, different number of query points), the entire inference graph needs to be recompiled, which takes on the order of minutes.

The solution is typically to make sure you're always running with the same size input. You can pad the query points with zeros if necessary without changing the output for the other points.

[hasnainv]

JAX also prints out this in the command line.

[hasnainv]

Thanks @cdoersch , that's correct. Most of the that was recompilation time. I'm now padding my data and keeping the dimensions same for every run.