marcoslucianops/deepstream-yolo-seg

NVIDIA DeepStream SDK 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 implementation for YOLO-Segmentation models

deepstream-yolo-seg's Issues

Error in ONNX export file

While tweaking are model I found an error in utils/export_yoloV8_seg.py. In line 71, the IOU threshold and confidence threshold are passed to the NMS.apply function in the wrong order. In our case, this causes the network to have very few detections when a high IOU threshold is set.

memory leaky

This segmentation has a slight memory leak. Has anyone else encountered a similar situation? Do multiple input sources need to be used for reasoning to more clearly see the increase in memory usage.

Low FPS with jetson ORIN

Dear,

I have followed the instructions for yolo segmentation but i have a problem with orin
With the different deepstream docker version, i have only 3 FPS with a model size is 1088 1920

Convert model pt to onnx is error

use wget https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s-seg.pt download the model.

python3 export_yoloV8_seg.py -w yolov8s-seg.pt --opset 12

output:
Starting: yolov8s-seg.pt
Opening YOLOv8-Seg model

Ultralytics YOLOv8.0.195 🚀 Python-3.8.16 torch-2.0.0+cu118 CPU (12th Gen Intel Core(TM) i9-12900K)
YOLOv8s-seg summary (fused): 195 layers, 11810560 parameters, 0 gradients, 42.6 GFLOPs

Creating labels.txt file

Exporting the model to ONNX
============= Diagnostic Run torch.onnx.export version 2.0.0+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
File "/soft/anaconda3/envs/ultralytics/lib/python3.8/site-packages/torch/onnx/utils.py", line 1636, in _export
_C._check_onnx_proto(proto)
RuntimeError: Unrecognized attribute: coordinate_transformation_mode for operator RoiAlign

==> Context: Bad node spec for node. Name: /1/RoiAlign OpType: RoiAlign

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "export_yoloV8_seg.py", line 213, in
sys.exit(main(args))
File "export_yoloV8_seg.py", line 178, in main
torch.onnx.export(model, onnx_input_im, onnx_output_file, verbose=False, opset_version=args.opset,
File "/soft/anaconda3/envs/ultralytics/lib/python3.8/site-packages/torch/onnx/utils.py", line 506, in export
_export(
File "/soft/anaconda3/envs/ultralytics/lib/python3.8/site-packages/torch/onnx/utils.py", line 1638, in _export
raise errors.CheckerError(e) from e
torch.onnx.errors.CheckerError: Unrecognized attribute: coordinate_transformation_mode for operator RoiAlign

==> Context: Bad node spec for node. Name: /1/RoiAlign OpType: RoiAlign

--simplify in export_yoloV8_seg.py is creating a model of size 160x160

YOLO segmentation model not working on jetpack 4.6.4

Hardware Platform (Jetson / GPU) = Jetson nano
DeepStream Version = 6.0.1
JetPack Version (valid for Jetson only) = 4.6.4
TensorRT Version = 8.2.1.8-1+cuda10.2
Python version = 3.6.9

I have followed this link and perform all the steps
https://github.com/marcoslucianops/DeepStream-Yolo-Seg/blob/master/docs/YOLOv8_Seg.md 1

I have converted my custom yolov8-seg model into onnx using

python3 export_yoloV8.py -w yolov8s.pt --simplify

While running this command

deepstream-app -c deepstream_app_config.txt

i am getting this output:
#####################################################################################################

root@ubuntu:/opt/nvidia/DeepStream-Yolo-Seg# deepstream-app -c deepstream_app_config.txt

Using winsys: x11
ERROR: Deserialize engine failed because file path: /opt/nvidia/DeepStream-Yolo-Seg/S_Kerbhit_YoloV8.onnx_b1_gpu0_fp32.engine open error
0:00:06.595090589 6736 0x7f1c001f80 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/DeepStream-Yolo-Seg/S_Kerbhit_YoloV8.onnx_b1_gpu0_fp32.engine failed
0:00:06.596617444 6736 0x7f1c001f80 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/DeepStream-Yolo-Seg/S_Kerbhit_YoloV8.onnx_b1_gpu0_fp32.engine failed, try rebuild
0:00:06.596735936 6736 0x7f1c001f80 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: Tensor DataType is determined at build time for tensors not marked as input or output.
ERROR: [TRT]: ModelImporter.cpp:773: While parsing node number 290 [RoiAlign -> "/1/RoiAlign_output_0"]:
ERROR: [TRT]: ModelImporter.cpp:774: --- Begin node ---
ERROR: [TRT]: ModelImporter.cpp:775: input: "/0/model.22/proto/cv3/act/Mul_output_0"
input: "/1/Gather_5_output_0"
input: "/1/Gather_1_output_0"
output: "/1/RoiAlign_output_0"
name: "/1/RoiAlign"
op_type: "RoiAlign"
attribute {
name: "coordinate_transformation_mode"
s: "half_pixel"
type: STRING
}
attribute {
name: "mode"
s: "avg"
type: STRING
}
attribute {
name: "output_height"
i: 160
type: INT
}
attribute {
name: "output_width"
i: 160
type: INT
}
attribute {
name: "sampling_ratio"
i: 0
type: INT
}
attribute {
name: "spatial_scale"
f: 0.25
type: FLOAT
}

ERROR: [TRT]: ModelImporter.cpp:776: --- End node ---
ERROR: [TRT]: ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4870 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
ERROR: Failed to parse onnx file
ERROR: failed to build network since parsing model errors.
ERROR: failed to build network.
0:00:08.599046925 6736 0x7f1c001f80 ERROR nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 1]: build engine file failed
0:00:08.600637322 6736 0x7f1c001f80 ERROR nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2020> [UID = 1]: build backend context failed
0:00:08.600739668 6736 0x7f1c001f80 ERROR nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1257> [UID = 1]: generate backend failed, check config file settings
0:00:08.601245199 6736 0x7f1c001f80 WARN nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:08.601298586 6736 0x7f1c001f80 WARN nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Config file path: /opt/nvidia/DeepStream-Yolo-Seg/config_infer_primary_yoloV5_seg.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
** ERROR: main:707: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to create NvDsInferContext instance
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(841): gst_nvinfer_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie:
Config file path: /opt/nvidia/DeepStream-Yolo-Seg/config_infer_primary_yoloV5_seg.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
App run failed

#####################################################################################################
Please find attached the onnx graph.

Can not get the deepstream-app! How to deal with the problem?

@ubuntu:~/DeepStream-Yolo-Seg$ sudo CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo_seg
make: Entering directory '/home/sfhs/DeepStream-Yolo-Seg/nvdsinfer_custom_impl_Yolo_seg'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/sfhs/DeepStream-Yolo-Seg/nvdsinfer_custom_impl_Yolo_seg'

Segmentation fault (core dumped)

1> run:
~/work/yolo_deepstream/DeepStream-Yolo-Seg$ deepstream-app -c deepstream_app_config.txt

2> error logs:
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1487 Deserialize engine failed because file path: /home/work/yolo_deepstream/DeepStream-Yolo-Seg/yolov8s-seg.onnx_b1_gpu0_fp32.engine open error
0:00:02.666741042 13265 0x55aeb942c120 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1976> [UID = 1]: deserialize engine from file :/home/work/yolo_deepstream/DeepStream-Yolo-Seg/yolov8s-seg.onnx_b1_gpu0_fp32.engine failed
0:00:02.667629067 13265 0x55aeb942c120 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2081> [UID = 1]: deserialize backend context from engine from file :/home/work/yolo_deepstream/DeepStream-Yolo-Seg/yolov8s-seg.onnx_b1_gpu0_fp32.engine failed, try rebuild
0:00:02.667649670 13265 0x55aeb942c120 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2002> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
0:00:58.957847518 13265 0x55aeb942c120 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2034> [UID = 1]: serialize cuda engine to file: /home/work/yolo_deepstream/DeepStream-Yolo-Seg/yolov8s-seg.onnx_b1_gpu0_fp32.engine successfully
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT images 3x640x640
1 OUTPUT kFLOAT output1 32x160x160
2 OUTPUT kFLOAT output0 116x8400

0:00:58.974206457 13265 0x55aeb942c120 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/titanx/work/yolo_deepstream/DeepStream-Yolo-Seg/config_infer_primary_yoloV8_seg.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg)
**PERF: 0.00 (0.00)
** INFO: <bus_callback:239>: Pipeline ready

** INFO: <bus_callback:225>: Pipeline running

Segmentation fault (core dumped)

Output masks retrieved with mask_params.get_mask_array() very inaccurate

I am using a self-developed Python app to build up a GStreamer pipeline and to insert a probe to interpret the metadata from the YOLOv8 segmentation model. The app is derived from your Python example in the YOLOv8 pose repository (https://github.com/marcoslucianops/DeepStream-Yolo-Pose/blob/master/deepstream.py)

As in your pose example, I am using obj_meta.mask_params.get_mask_array(). In case of the segmentation model, this delivers the segmentation mask as an array of 160x160 float values. If it is scaled to the bounding box of the detected object and converted to a gray level mask (e.g. 0 for every tensor value below 0.5 and 255 for every tensor value >= 0.5), it can be used for drawing/masking operations. The problem is, that this mask is quite inaccurate as shown in these examples:

When I use the default deepstream-app with the configuration file given in your respository (only modified to use my sample video), I can see much more accurate masks and bounding boxes, as shown in these examples:

As the model is in both cases the same YOLOv8-seg, I ask myself, which part of the meta data the default deepstream-app uses for drawing the masks and bounding boxes.

Looking at a discussion at Ultralytics (ultralytics/ultralytics#2953 (comment)) it looks like the segementation model originally outputs 32 "prototype" masks which have to be multiplied with the given mask weights and summed together in order to get the final mask.

So it looks like, that the mask I get with obj_meta.mask_params.get_mask_array() is only one of these prototype masks but not the final one. The question is, how to get the final one which can be seen when using the original deepstream-app. Similar question for the bounding boxes, which I retrieve in the probe using obj_meta.rect_params and which also look less accurate than those displayed by the deepstream-app.

Segmentation fault (core dumped)

Hello. Thanks for your great work
I take the following steps:

1. Clone the repo and go to the directory
2. I run this command CUDA_VER=11.7 make -C nvdsinfer_custom_impl_Yolo_seg (beacuase i use deepstream 6.1.1)
3. and run deepstream app (deepstream-app -c deepstream_app_config.txt)
   but get following error:
   

When batch>2, can not conver onnx to trt

deepstream=6.2
Jetson NX ori

[09/15/2023-09:21:15] [E] Error[4]: [shapeCompiler.cpp::evaluateShapeChecks::1180] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: IShuffleLayer /1/Reshape_12: reshaping failed for tensor: /1/Expand_1_output_0 Reshape would change volume.)
[09/15/2023-09:21:15] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[09/15/2023-09:21:15] [E] Engine could not be created from network
[09/15/2023-09:21:15] [E] Building engine failed
[09/15/2023-09:21:15] [E] Failed to create engine from model or file.
[09/15/2023-09:21:15] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=yolov8s-seg.onnx --minShapes=input:1x3x640x640 --optShapes=input:4x3x640x640 --maxShapes=input:4x3x640x640 --saveEngine=yolov8s-seg.engine --workspace=1024

Can't build the model engine file for batch size > 2

I exported a YOLOv8-Seg into an ONNX model using the utils from this repo, setting --dynamic option when exporting. I also tried manually specifying the --batch option, but the results were the same:

I have 4 streams from different cameras, and I set batch-size=4 on the model configuration.

However, when running the DeepStream pipeline this is the error I'm seeing. The problem persists with batch-size=3 or batch-size=5.

ERROR: [TRT]: 4: [shapeCompiler.cpp::evaluateShapeChecks::1180] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: IShuffleLayer /1/Reshape_12: reshaping failed for tensor: /1/Expand_1_output_0 Reshape would change volume.)
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1124 Build engine failed from config file
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:816 failed to build trt engine.

I'm wondering if this is internal to the YOLOv8-Seg model, or this is happening due to the export from the utils in this repo. What are the different steps from this repo and the export command from ultralytics?

Division by 0 error if there's no detections in the frame

I managed to create a yolov8s-seg.onnx_b1_gpu0_fp16.engine model. It works fine when the video starts and there's detections, but if no detection is found in the first frame, then there's "Division by 0 " errors, is there anyway to avoid this ?
Full output is below ..

0:00:03.545107844 48470 0x55eeac34e330 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/home/rama/Documents/code-repo/codecommit/edgeai-deepstream-dev_weight_estimation/edgeai-deepstream/models/yolov8s-seg.onnx_b1_gpu0_fp16.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 5
0 INPUT kFLOAT input 3x640x640
1 OUTPUT kFLOAT boxes 100x4
2 OUTPUT kFLOAT scores 100x1
3 OUTPUT kFLOAT classes 100x1
4 OUTPUT kFLOAT masks 100x160x160

0:00:03.644677843 48470 0x55eeac34e330 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /home/rama/Documents/code-repo/codecommit/edgeai-deepstream-dev_weight_estimation/edgeai-deepstream/models/yolov8s-seg.onnx_b1_gpu0_fp16.engine
0:00:03.649166370 48470 0x55eeac34e330 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/rama/Documents/code-repo/github/DeepStream-Yolo-Seg/config_infer_primary_yoloV8_seg_coco.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:239>: Pipeline ready

(deepstream-app:48470): GStreamer-WARNING **: 13:05:06.930: (../gst/gstinfo.c:556):gst_debug_log_valist: runtime check failed: (object == NULL || G_IS_OBJECT (object))
** INFO: <bus_callback:225>: Pipeline running

ERROR: [TRT]: 1: [runner.cpp::shapeChangeHelper::621] Error Code 1: Myelin (Division by 0 detected in the shape graph. Tensor (Divisor) "sp__mye3" is equal to 0.; )
ERROR: nvdsinfer_backend.cpp:506 Failed to enqueue trt inference batch
ERROR: nvdsinfer_context_impl.cpp:1650 Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:00:03.971140577 48470 0x55eeacb92800 WARN nvinfer gstnvinfer.cpp:1388:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
ERROR from primary_gie: Failed to queue input batch for inferencing
Debug info: gstnvinfer.cpp(1388): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
Quitting
ERROR: [TRT]: 1: [runner.cpp::shapeChangeHelper::621] Error Code 1: Myelin (Division by 0 detected in the shape graph. Tensor (Divisor) "sp__mye3" is equal to 0.; )
ERROR: nvdsinfer_backend.cpp:506 Failed to enqueue trt inference batch
ERROR: nvdsinfer_context_impl.cpp:1650 Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:00:03.981255708 48470 0x55eeacb92800 WARN nvinfer gstnvinfer.cpp:1388:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
nvstreammux: Successfully handled EOS for source_id=0
ERROR: [TRT]: 1: [runner.cpp::shapeChangeHelper::621] Error Code 1: Myelin (Division by 0 detected in the shape graph. Tensor (Divisor) "sp__mye3" is equal to 0.; )
ERROR: nvdsinfer_backend.cpp:506 Failed to enqueue trt inference batch
ERROR: nvdsinfer_context_impl.cpp:1650 Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:00:03.994894500 48470 0x55eeacb92800 WARN nvinfer gstnvinfer.cpp:1388:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
ERROR: [TRT]: 1: [runner.cpp::shapeChangeHelper::621] Error Code 1: Myelin (Division by 0 detected in the shape graph. Tensor (Divisor) "sp__mye3" is equal to 0.; )
ERROR: nvdsinfer_backend.cpp:506 Failed to enqueue trt inference batch
ERROR: nvdsinfer_context_impl.cpp:1650 Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:00:04.021695221 48470 0x55eeacb92800 WARN nvinfer gstnvinfer.cpp:1388:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
ERROR: [TRT]: 1: [runner.cpp::shapeChangeHelper::621] Error Code 1: Myelin (Division by 0 detected in the shape graph. Tensor (Divisor) "sp__mye3" is equal to 0.; )
ERROR: nvdsinfer_backend.cpp:506 Failed to enqueue trt inference batch
ERROR: nvdsinfer_context_impl.cpp:1650 Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:00:04.035448646 48470 0x55eeacb92800 WARN nvinfer gstnvinfer.cpp:1388:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
ERROR from primary_gie: Failed to queue input batch for inferencing
Debug info: gstnvinfer.cpp(1388): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR from primary_gie: Failed to queue input batch for inferencing
Debug info: gstnvinfer.cpp(1388): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR from primary_gie: Failed to queue input batch for inferencing
Debug info: gstnvinfer.cpp(1388): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR from primary_gie: Failed to queue input batch for inferencing
Debug info: gstnvinfer.cpp(1388): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
App run failed

Segmentation fault (core dumped)

Clone the repo and go to the directory
I run this command CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo_seg (beacuase i use deepstream 6.0)
and run deepstream app (deepstream-app -c deepstream_app_config.txt)
but get following error:

Missing RoiAlign TRT plugin in TRT 8.4.1

Thanks for publishing your instance segmentation implementation, greatly appreciated!

My setup:

• Device: Jetson AGX Xavier
• Jetpack 5.0.2
• TRT: 8.4.1
• Docker image: nvcr.io/nvidia/deepstream-l4t:6.1.1-samples
• Model: yolov8

Generating a TRT engine on a Jetson AGX Xavier with deepstream 6.1.1 causes an error due to the RoiAlign plugin not being implemented until TRT 8.5.1. Due to a custom L4T kernel, I cannot switch to Jetpack 5.1.

Have you manage to get it to work on Jetpack 5.0.2? If not, would you have any pointers on a workaround? I was looking into grabbing NVIDIAs RoiAlign and adding it as a custom plugin to TRT 8.4.1, however I am unsure whether I have everything I need for that with just this snippet.

The full error trace:

:00:01.242304077    48     0x120b0b60 INFO                 nvinfer gstnvinfer.cpp:646:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: Tensor DataType is determined at build time for tensors not marked as input or output.
ERROR: [TRT]: ModelImporter.cpp:773: While parsing node number 340 [RoiAlign -> "/1/RoiAlign_output_0"]:
ERROR: [TRT]: ModelImporter.cpp:774: --- Begin node ---
ERROR: [TRT]: ModelImporter.cpp:775: input: "/0/model.22/proto/cv3/act/Mul_output_0"
input: "/1/Reshape_1_output_0"
input: "/1/Gather_5_output_0"
output: "/1/RoiAlign_output_0"
name: "/1/RoiAlign"
op_type: "RoiAlign"
attribute {
  name: "coordinate_transformation_mode"
  s: "half_pixel"
  type: STRING
}
attribute {
  name: "mode"
  s: "avg"
  type: STRING
}
attribute {
  name: "output_height"
  i: 160
  type: INT
}
attribute {
  name: "output_width"
  i: 160
  type: INT
}
attribute {
  name: "sampling_ratio"
  i: 0
  type: INT
}
attribute {
  name: "spatial_scale"
  f: 0.25
  type: FLOAT
}

ERROR: [TRT]: ModelImporter.cpp:776: --- End node ---
ERROR: [TRT]: ModelImporter.cpp:778: ERROR: builtin_op_importers.cpp:4890 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
ERROR: Failed to parse onnx file
ERROR: failed to build network since parsing model errors.
ERROR: failed to build network.
0:00:09.646492322    48     0x120b0b60 ERROR                nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1943> [UID = 1]: build engine file failed
0:00:09.848985441    48     0x120b0b60 ERROR                nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2029> [UID = 1]: build backend context failed
0:00:09.849393816    48     0x120b0b60 ERROR                nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1266> [UID = 1]: generate backend failed, check config file settings
0:00:09.852251066    48     0x120b0b60 WARN                 nvinfer gstnvinfer.cpp:846:gst_nvinfer_start:<primary-inference> error: Failed to create NvDsInferContext instance
0:00:09.852458566    48     0x120b0b60 WARN                 nvinfer gstnvinfer.cpp:846:gst_nvinfer_start:<primary-inference> error: Config file path: yolo_luffing_test.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
Error: gst-resource-error-quark: Failed to create NvDsInferContext instance (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(846): gst_nvinfer_start (): /GstPipeline:pipeline0/GstNvInfer:primary-inference:
Config file path: yolo_test.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED

When batch>2 or dynamic, can not conver onnx to trt

It seems to me like the max detections parameter in the line final_dets = batched_dets.new_zeros((b, self.max_det, i)) is causing the issue, as this is set to 100 statically regardless of batch size, causing the reshape to [.., 100, ...]. You can modify the forward pass of the DeepstreamOutput class in export_yoloV8_seg.py to multiply the max detections with the batch size, specifically, if you add to the beginning of the forward pass:

dynamic_max_det = x[0].shape[0] * self.max_det

And then modify line 98 to:

final_dets = batched_dets.new_zeros((b, dynamic_max_det, i))

While this works and the exported ONNX model can be used to successfully build an engine with a dynamic batch size, this does decimate my throughput when used on a single UDP stream input for some reason. Going from ~15FPS on batch size = 1 with an AGX Xavier on Jetpack 5.0.2 to ~3FPS on batch size = 2 and 4. Might be an issue with my modification or perhaps an issue with how I'm applying batching in this context. Would be curious to see if this causes similar issues for others.

I've tried this and it still doesn't work.

pt2onnx run:
python export_yoloV8_seg.py -w yolov8s-seg.pt --dynamic --simplify
and onnx2trt run:
trtexec --onnx=./onnx_models/yolov8s-seg.onnx --saveEngine=./engine_models/yolov8s-seg.engine --fp16 --minShapes=input:1x3x640x640 --optShapes=input:4x3x640x640 --maxShapes=input:4x3x640x640
The error message is still reported, and the error message is as follows:

[02/20/2024-10:50:42] [W] [TRT] /1/Reshape_12: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 1 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:42] [W] [TRT] Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
[02/20/2024-10:50:43] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +10, now: CPU 690, GPU 948 (MiB)
[02/20/2024-10:50:43] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +11, GPU +9, now: CPU 701, GPU 957 (MiB)
[02/20/2024-10:50:43] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[02/20/2024-10:50:43] [E] Error[4]: [shapeCompiler.cpp::evaluateShapeChecks::1180] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: IShuffleLayer /1/Reshape_12: reshaping failed for tensor: /1/Expand_1_output_0 Reshape would change volume.)
[02/20/2024-10:50:43] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[02/20/2024-10:50:43] [E] Engine could not be created from network
[02/20/2024-10:50:43] [E] Building engine failed
[02/20/2024-10:50:43] [E] Failed to create engine from model or file.
[02/20/2024-10:50:43] [E] Engine set up failed```