Use OnnxRuntime IO Binding to improve GPU inference performance

[abelriboulot]

Use OnnxRuntime IO Binding to improve GPU inference performance

[sam-writer]

The link at the bottom of the issue is dead, I think the appropriate link is now ONNX Runtime IOBinding.

@tianleiwu did you ever successfully do this?

[shiqingzhangCSU]

你好! 请问现在GPU上进行T5的推理，是不是onnxruntime和pytorch的速度都差不多啊？在decode的过程中，每次decode的结果past value都很大，用onnxruntime推理怎么减少IO呢？
Hello! Now that the inference of T5 on the GPU is similar, is the speed of onnxruntime and pytorch similar? In the process of decode, the past value of each decode result is large, how to reduce IO with onnxruntime reasoning?@tianleiwu

[tianleiwu]

@shiqingzhangCSU,
To reduce I/O, it need design of special CUDA kernels (and also integrates with BeamSearch operator) to deal with past state. In Onnx Runtime, @wangyems is working on optimizations for T5. It is very close to finish.

You can try out current optimizations (although optimizations are still on-going):
https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/convert_generation.py