This is a comparison of a deformabel attention kernel written in TVM's TVMScript to the pytorch and CUDA kernels in https://github.com/fundamentalvision/Deformable-DETR.
You need to install:
- TVM: https://github.com/apache/tvm
- PyTorch
- The deformable attention cuda kernels from
https://github.com/fundamentalvision/Deformable-DETR. Run
python3 setup.py build install
in themodels/ops
directory.
python3 kernel.py "llvm -mcpu=skylake"
will run the cpu kernels targeted for
an Intel Skylake cpu. YOu can determine what cpu you have by running llc --version
and looking for "Host CPU".
python3 kernel.py cuda
will run the gpu kernels.
# On AMD 5900X cpu
> python3 kernel.py "llvm -mcpu=znver3"
BATCH SIZE 1
TVMScript CPU: 0.33ms
PyTorch: 35.48ms
Speedup: 106.78x
BATCH SIZE 4
TVMScript CPU: 4.08ms
PyTorch: 128.69ms
Speedup: 31.53x
BATCH SIZE 8
TVMScript CPU: 11.30ms
PyTorch: 232.38ms
Speedup: 20.56x
# on NVIDIA Geforce RTX 3070
> python3 kernel.py "nvidia/geforce-rtx-3070"
BATCH SIZE 1
TVMScript GPU: 0.085ms
Handwritten CUDA: 0.127ms
Speedup: 1.49x
BATCH SIZE 4
TVMScript GPU: 0.333ms
Handwritten CUDA: 0.444ms
Speedup: 1.33x
BATCH SIZE 8
TVMScript GPU: 0.669ms
Handwritten CUDA: 0.870ms
Speedup: 1.30x