What exactly does "Optimal" mean?

[nvidia]

What exactly does "Optimal" mean?

[contentis]

TensorRT sometimes is more art than science. It could be that 2048 is so large that, at that point, it doesn't make a difference. For 1024 you can definitely see a performance delta between the opt=512 and opt=1024 engine.

From your experience is 1.0x it/s slow?

[contentis]

Optimal is the config TensorRT will optimize for, whereas min and max only define the range in which it needs to be operable.
E.g. an engine compiled with optimal width/height set to 512, will perform best when running it at 512x512.

[carnivar10]

Optimal is the config TensorRT will optimize for, whereas min and max only define the range in which it needs to be operable. E.g. an engine compiled with optimal width/height set to 512, will perform best when running it at 512x512.

Thanks for the answer!! So if I start my image at 512 and grow it to 2048, do I need 3 engines with Optimal set to 512, 1024, and 2048? how would you recommend setting Optimal?

[contentis]

Technically, you could do an engine with min=512, opt=1024, max=2048 that covers all cases. But performance will probably be suboptimal at 512 and 2048. How large the performance impact is going to be is hard to estimate. In that case, I'd recommend exporting multiple engines, using your most commonly used resolutions for optimal.

[carnivar10]

I tried changing the Optimal Width and Height to 512, 1024, and 2048, but it doesn't seem to make much of a difference (I did this by pressing Force Rebuild on TensorRT Export and then creating a new Engine)

[carnivar10]

The difference in speed between using TensorRT and not using TensorRT is clear, but it doesn't seem to be an optimal value (at low resolutions, the images are generated very quickly, so it may be that can't tell)