rizhaocai/pytorch_onnx_tensorrt

A tutorial about how to build a TensorRT Engine from a PyTorch Model with the help of ONNX

• 👋 Hi, I’m @RizhaoCai, a PhD student in ROSE Lab@LNTU

• 🌱 I’m currently doing research on face anti-spoofing and domain generalization.

• 👀 Beyond, I’m interested in Basketball, Guitar, Writing, and reading history books. Meanwhile, as a research staff of ROSE Lab, I have been managing and leading projects with several industrial partenrs, which are about using object detection, recognition, image generation and simulation. I am also seeking oppotunities of using AI in industry.

• Reach me if you are interested:
  📫 [email protected]
  🏠 https://rizhaocai.github.io

pytorch_onnx_tensorrt's Issues

AdaptiveAvgPool2d not supported by tensorRT OnnxParser

Env:
ubuntu 16.04
Python 3.6
Pytorch 1.3.0
tensorRT:7.0.11

Detail:
the adaptiveavgpool2d operation is not supported by onnxparser, so it could lead to the failure of transformation of onnx to tensorrt.
you could use
for error in range(parser.num_errors):
print(parser.get_error(error))
after the parser read onnx model

Does onnx convert tensorRT need a data set for calibration?

I saw some quantization tutorials stating that a small part of the training data set is required for quantitative calibration to determine the range of activation values for activation and weights. Why is there no such part in the code you provided, or is it unnecessary? Thank you

（use arcface model） AttributeError: 'NoneType' object has no attribute 'serialize'

show:Completed parsing of ONNX file
but when

f.write(engine.serialize())

AttributeError: 'NoneType' object has no attribute 'serialize'

I guess my onnx model maybe have some problem,I am trying check the pth2onnx..

error: Failed to parse ONNX model.

Hello,thank you for your work. I get a error when run the demo,but i just use the model_128.onnx and did not make any
changes.
What is the reason and how to solve it ？

error:
Please check if the ONNX model is compatible '
AssertionError: Failed to parse ONNX model.

AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Connected to pydev debugger (build 181.5540.34)
Loading ONNX file from path ./models/onnx/model.onnx...
Beginning ONNX file parsing
WARNING: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
Successfully casted down to INT32.
Completed parsing of ONNX file
Building an engine from file ./models/onnx/model.onnx; this may take a while...
[TensorRT] ERROR: Network must have at least one output
Failed to create the engine

AssertionError: Failed to parse ONNX model. Please check if the ONNX model is compatible

trtexec --onnx=/home/guohao02/PyTorch_ONNX_TensorRT/model_128.onnx --explicitBatch

[10/16/2021-13:56:29] [I] === Model Options ===
[10/16/2021-13:56:29] [I] Format: ONNX
[10/16/2021-13:56:29] [I] Model: /home/guohao02/PyTorch_ONNX_TensorRT/model_128.onnx
[10/16/2021-13:56:29] [I] Output:
[10/16/2021-13:56:29] [I] === Build Options ===
[10/16/2021-13:56:29] [I] Max batch: explicit
[10/16/2021-13:56:29] [I] Workspace: 16 MB
[10/16/2021-13:56:29] [I] minTiming: 1
[10/16/2021-13:56:29] [I] avgTiming: 8
[10/16/2021-13:56:29] [I] Precision: FP32
[10/16/2021-13:56:29] [I] Calibration: 
[10/16/2021-13:56:29] [I] Safe mode: Disabled
[10/16/2021-13:56:29] [I] Save engine: 
[10/16/2021-13:56:29] [I] Load engine: 
[10/16/2021-13:56:29] [I] Inputs format: fp32:CHW
[10/16/2021-13:56:29] [I] Outputs format: fp32:CHW
[10/16/2021-13:56:29] [I] Input build shapes: model
[10/16/2021-13:56:29] [I] === System Options ===
[10/16/2021-13:56:29] [I] Device: 0
[10/16/2021-13:56:29] [I] DLACore: 
[10/16/2021-13:56:29] [I] Plugins:
[10/16/2021-13:56:29] [I] === Inference Options ===
[10/16/2021-13:56:29] [I] Batch: Explicit
[10/16/2021-13:56:29] [I] Iterations: 10 (200 ms warm up)
[10/16/2021-13:56:29] [I] Duration: 10s
[10/16/2021-13:56:29] [I] Sleep time: 0ms
[10/16/2021-13:56:29] [I] Streams: 1
[10/16/2021-13:56:29] [I] Spin-wait: Disabled
[10/16/2021-13:56:29] [I] Multithreading: Enabled
[10/16/2021-13:56:29] [I] CUDA Graph: Disabled
[10/16/2021-13:56:29] [I] Skip inference: Disabled
[10/16/2021-13:56:29] [I] === Reporting Options ===
[10/16/2021-13:56:29] [I] Verbose: Disabled
[10/16/2021-13:56:29] [I] Averages: 10 inferences
[10/16/2021-13:56:29] [I] Percentile: 99
[10/16/2021-13:56:29] [I] Dump output: Disabled
[10/16/2021-13:56:29] [I] Profile: Disabled
[10/16/2021-13:56:29] [I] Export timing to JSON file: 
[10/16/2021-13:56:29] [I] Export profile to JSON file: 
[10/16/2021-13:56:29] [I] 
----------------------------------------------------------------
Input filename:   /home/guohao02/PyTorch_ONNX_TensorRT/model_128.onnx
ONNX IR version:  0.0.6
Opset version:    9
Producer name:    pytorch
Producer version: 1.9
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
WARNING: ONNX model has a newer ir_version (0.0.6) than this parser was built against (0.0.3).
While parsing node number 0 [Conv]:
ERROR: ModelImporter.cpp:296 In function importModel:
[5] Assertion failed: tensors.count(input_name)
[10/16/2021-13:56:29] [E] Failed to parse onnx file
[10/16/2021-13:56:29] [E] Parsing model failed
[10/16/2021-13:56:29] [E] Engine could not be created
&&&& FAILED TensorRT.trtexec # trtexec --onnx=/home/guohao02/PyTorch_ONNX_TensorRT/model_128.onnx --explicitBatch

where is the c++ code for onnx and trt?

代码与 tensorrt 8.4 不兼容

我正在使用 tensorrt 8.4的环境运行代码，报错
'tensorrt.tensorrt.Builder' object has no attribute 'build_cuda_engine'，int8_mod, fp16_mode, int8_mode

在新的tensorrt中，使用config 接管了 builder的各种设置。

Run batch size with inputs[1].host

Hi, everyone
I define batch_size_max = 4 and input size onnx= 2 to run batch_size = 2 for model trt
When i convert model that Error show "list index out of range" in inputs[1].host
That mean inputs only have one elements.
How to fix when i want to run batch size > 1

Int8 implementation

Hi! Any idea or resources on how to implement the Int8? The documentation from Nvidia is too minimal to get how it is supposed to work. They mention creating two objects, but who knows from which classes and how.

error:create_execution_context

AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Hi, I want to make sure whether there are any differences between engine generation by your python code and by trtexec?

And I also want to know how to quantize it in int8 model rather than fp32 by trtexec, if this is familiar to you.

problems occured when executing "python trt_int8_demo.py "

Building an engine from file model_128.onnx; this may take a while...
Traceback (most recent call last):
File "trt_int8_demo.py", line 138, in
main()
File "trt_int8_demo.py", line 92, in main
engine_int8 = trt_helper.get_engine(batch_size,onnx_model_path,engine_model_path, fp16_mode=False, int8_mode=True, calibration_stream=calibration_stream, save_engine=True)
File "/data/zhangyl/PyTorch_ONNX_TensorRT/helpers/trt_helper.py", line 95, in get_engine
return build_engine(max_batch_size, save_engine)
File "/data/zhangyl/PyTorch_ONNX_TensorRT/helpers/trt_helper.py", line 76, in build_engine
engine = builder.build_cuda_engine(network)
TypeError: read_calibration_cache() missing 1 required positional argument: 'length'