TensorRT 8 Support

[Michelvl92]

TensorRT 8 Support

[philipp-schmidt]

TensorRT 8 changed the way plugins work. I did not have time to look into it yet. Contributions are very welcomed!

[philipp-schmidt]

Newest v1.4.0 release comes with support for TRT 8. Tested with NGC TensorRT 21.10 and Triton 21.10 with 470 driver. Performance gain compared to previous 1.3.0 release is at about 19.5%.

Did not have the time to run and check the python client, so feel free to report any issues with that.

[philipp-schmidt]

Usually the computational part of the plugin stays the same and only the interface around it changes, so it should not be too big of a change.

[revanb88]

I am also facing same issue, could you please help me. I used tensorrt:21.08 got this error

/yolov4-triton-tensorrt/layers/yololayer.h(59): error: member function declared with "override" does not override a base class member

/yolov4-triton-tensorrt/layers/yololayer.h(108): error: function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)"
/usr/include/x86_64-linux-gnu/NvInferRuntimeCommon.h(836): here is inaccessible

/yolov4-triton-tensorrt/layers/yololayer.h(59): warning: function "nvinfer1::IPluginV2::enqueue(int32_t, const void *const *, void *const *, void *, cudaStream_t)" is hidden by "nvinfer1::YoloLayerPlugin::enqueue" -- virtual function override intended?

/yolov4-triton-tensorrt/layers/yololayer.h(89): warning: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::YoloLayerPlugin::configurePlugin" -- virtual function override intended?

/yolov4-triton-tensorrt/layers/yololayer.h(91): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::detachFromContext" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::detachFromContext"

/yolov4-triton-tensorrt/layers/yololayer.h(89): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::configurePlugin" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::PluginTensorDesc *, int32_t, const nvinfer1::PluginTensorDesc *, int32_t)"

/yolov4-triton-tensorrt/layers/yololayer.h(87): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::attachToContext" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::attachToContext"

/yolov4-triton-tensorrt/layers/yololayer.h(85): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::canBroadcastInputAcrossBatch" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::canBroadcastInputAcrossBatch"

/yolov4-triton-tensorrt/layers/yololayer.h(83): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::isOutputBroadcastAcrossBatch" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::isOutputBroadcastAcrossBatch"

/yolov4-triton-tensorrt/layers/yololayer.h(81): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDataType" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::getOutputDataType"

/yolov4-triton-tensorrt/layers/yololayer.h(79): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(77): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::setPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginV2::setPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::clone"

/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2::clone"

/yolov4-triton-tensorrt/layers/yololayer.h(73): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::destroy" is incompatible with that of overridden function "nvinfer1::IPluginV2::destroy"

/yolov4-triton-tensorrt/layers/yololayer.h(71): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginVersion"

/yolov4-triton-tensorrt/layers/yololayer.h(69): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginType" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginType"

/yolov4-triton-tensorrt/layers/yololayer.h(65): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::supportsFormatCombination" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::supportsFormatCombination"

/yolov4-triton-tensorrt/layers/yololayer.h(63): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::serialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::serialize"

/yolov4-triton-tensorrt/layers/yololayer.h(61): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getSerializationSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getSerializationSize"

/yolov4-triton-tensorrt/layers/yololayer.h(57): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getWorkspaceSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getWorkspaceSize"

/yolov4-triton-tensorrt/layers/yololayer.h(55): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::terminate" is incompatible with that of overridden function "nvinfer1::IPluginV2::terminate"

/yolov4-triton-tensorrt/layers/yololayer.h(53): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::initialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::initialize"

/yolov4-triton-tensorrt/layers/yololayer.h(51): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDimensions" is incompatible with that of overridden function "nvinfer1::IPluginV2::getOutputDimensions"

/yolov4-triton-tensorrt/layers/yololayer.h(46): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getNbOutputs" is incompatible with that of overridden function "nvinfer1::IPluginV2::getNbOutputs"

/yolov4-triton-tensorrt/layers/yololayer.h(133): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(128): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::setPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginCreator::setPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(126): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::deserializePlugin" is incompatible with that of overridden function "nvinfer1::IPluginCreator::deserializePlugin"

/yolov4-triton-tensorrt/layers/yololayer.h(124): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::createPlugin" is incompatible with that of overridden function "nvinfer1::IPluginCreator::createPlugin"

/yolov4-triton-tensorrt/layers/yololayer.h(122): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getFieldNames" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getFieldNames"

/yolov4-triton-tensorrt/layers/yololayer.h(120): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginVersion"

/yolov4-triton-tensorrt/layers/yololayer.h(118): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginName" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginName"

/yolov4-triton-tensorrt/layers/yololayer.cu(173): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/yolov4-triton-tensorrt/layers/yololayer.cu(412): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/yolov4-triton-tensorrt/layers/yololayer.cu(419): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

33 errors detected in the compilation of "/yolov4-triton-tensorrt/layers/yololayer.cu".
CMake Error at layerplugin_generated_yololayer.cu.o.cmake:279 (message):
Error generating file
/yolov4-triton-tensorrt/build/CMakeFiles/layerplugin.dir/layers/./layerplugin_generated_yololayer.cu.o

make[2]: *** [CMakeFiles/layerplugin.dir/build.make:65: CMakeFiles/layerplugin.dir/layers/layerplugin_generated_yololayer.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:109: CMakeFiles/layerplugin.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

[chull434]

yeah ran into the same issue when I update to jetson jetpack 4.6 from 4.5 and tensorRT upgraded to 8.0.1

[chull434]

I followed this pr's changes and got it to at least build now wang-xinyu/tensorrtx#540

Also note need to update the docker image to a TRT8 one, such as nvcr.io/nvidia/tensorrt:21.09-py3

[revanb88]

Thanks a lot. I will go through it.

…

On Wed, Nov 3, 2021 at 9:00 PM chull434 ***@***.***> wrote: I followed this pr's changes and got it to at least build now wang-xinyu/tensorrtx#540 <wang-xinyu/tensorrtx#540> Also note need to update the docker image to a TRT8 one, such as nvcr.io/nvidia/tensorrt:21.09-py3 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWDAXTIJB4DDQHIBALD76KLUKFIRHANCNFSM5F5DGZ3Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Thanks and Regards Dr. Revanasiddappa M Biradar Data Scientist Bengaluru

[ohlr]

@revanb88, @chull434 - would you like to make this a PR?

[ohlr]

@Michelvl92 did you find a way to support TRT8?

[jenfredwell]

I've also ended up here with the same errors trying to use the 21.08 version. Curious about the status?

[philipp-schmidt]

No support yet. We will update this to TensorRT 8 at some point, no roadmap though. PR from someone else will probably be faster. Should only be a matter of applying necessary changes from the patch notes from nvidia to the new plugin interface.