Dynamic shape for input like NLP

[milad1378yz]

Dynamic shape for input like NLP

[diyessi]

I thought of dynamic shapes as like C++ vectors; there's allocated space and something separate that indicates how much of that space you are actually using. You might cache particular compiled combinations of max sizes for inputs. If you are a server, dynamic batch size can help with latency since you would only need to compute for as many samples as you had ready. You could also imagine kernels that could make use of knowing actual sample lengths to reduce the computation in those transformer GEMMs (some hardware skips 0 arithmetic and some hardware does a tile of GEMM in the same amount of time as a partial tile, so it wouldn't matter for them). Whatever Intel did with ngraph along those lines would be in the OpenVINO repo.