ORT backend always returns tensor on CPU

[triton-inference-server]

ORT backend always returns tensor on CPU

[askhade]

@Slyne This will be available in triton 22.02 release.

[CoderHam]

@Tabrizian @tanmayv25 can you look into the same

[askhade]

This issue explains the current limitation and why output is always on CPU : triton-inference-server/server#3364

[askhade]

Once triton-inference-server/server#3364 is merged we will enable output binding to gpus in ort backend.

[Slyne]

Any update ?
@askhade
@deadeyegoodwin

[Slyne]

@askhade Thank you for informing!

[vu0607]

@askhade
I serving encoder-decoder model (TrOCR) on Triton onnx backend. I meet a problem:
First, I call and get output from encoder model in server. Afterthat, because output on GPU, I need to transfer to CPU for converting to numpy before call output from decoder model on server. It make botteneck. Hope you can help me with issue. Thanks a lot.