English | Zh-CN
Deploy Yolov5 object detection service in Triton. The following optimizations have been made:
Pipelines are implemented through Ensemble
and BLS
respectively. The infer module in Pipelines is based on the optimized TensorRT Engine in item 1 above, and the Postprocess module is implemented through Python Backend. The workflow refers to [How to deploy Triton Pipelines](./docs/pipelines_EN.md#3-How to deploy Triton Pipelines).
- CPU: 4cores 16GB
- GPU: Nvidia Tesla T4
- Cuda: 11.6
- TritonServer: 2.20.0
- TensorRT: 8.2.3
- Yolov5: v6.1
Triton Inference Server is deployed on one machine. Perf_analyzer is used to call the gRPC interface on another machine to compare the performance of BLS Pipelines
, Ensemble Pipelines
, and BatchedNMS
under gradually increasing concurrency.
-
python generate_input.py --input_images <image_path> ----output_file <real_data>.json
-
Test with real data
perf_analyzer -m <triton_model_name> -b 8 --input-data <real_data>.json --concurrency-range 1:10 --measurement-interval 10000 -u <triton server endpoint> -i gRPC -f <triton_model_name>.csv
The data shows that BatchedNMS
performs better overall, converging to optimal performance faster under high concurrency, and achieving higher throughput at lower latency. Ensemble Pipelines
and BLS Pipelines
perform better at lower concurrency, but performance degrades more as concurrency increases.
Six metrics are selected for comparison. Each metric is processed and normalized to the 0-1 interval. The original meaning of each metric is as follows:
- Server Queue: Data waiting time in Triton queue
- Server Compute Input: Triton input tensor processing time
- Server Compute Infer: Triton inference execution time
- Server Compute Output: Triton output tensor processing time
- latency: 90th percentile end-to-end latency
- throughput: throughput