How to use Forward to infer the pytorch version of bert, are there any relevant demo?

[Fan9]

How to use Forward to infer the pytorch version of bert, are there any relevant demo?

[yuanzexi]

Bert-Pytorch demo is updated here. https://github.com/Tencent/Forward/tree/master/demo/bert

[Fan9]

@yuanzexi Bug when running test_torch_bert.py("bert-base-uncased" downloaded from https://huggingface.co/bert-base-cased)，There are three errors messages.

 [ERROR] 2021-04-08 16:22:09,532 trt_logger.cpp(64): [TRT] 29 [Cast]: could not find any supported formats consistent with input/output data types
[ERROR] 2021-04-08 16:22:09,533 trt_logger.cpp(64): [TRT] ../builder/cudnnBuilderGraphNodes.cpp (872) - Misc Error in reportPluginError: 0 (could not find any supported formats consistent with input/output data types)

 [ERROR] 2021-04-08 16:22:09,550 trt_logger.cpp(64): [TRT] ../builder/cudnnBuilderGraphNodes.cpp (872) - Misc Error in reportPluginError: 0 (could not find any supported formats consistent with input/output data types)
 [ERROR] 2021-04-08 16:22:09,550 trt_fwd_builder.cpp(121): buildEngineWithConfig error

[ERROR] 2021-04-08 16:52:12,292 torch_engine.cpp(219): Build forward engine failed
Traceback (most recent call last):
  File "test_torch_bert.py", line 42, in <module>
    TestForward(jit_path)
  File "test_torch_bert.py", line 29, in TestForward
    engine.save(engine_path)
AttributeError: 'NoneType' object has no attribute 'save'

Environment

TensorRT Version: 7.2.1.6
NVIDIA GPU: GTX1080TI
NVIDIA Driver Version: 450.80.02
CUDA Version: 11.0
CUDNN Version: 8.0.4
Operating System: centos 7.5
Python Version (if applicable): 3.6.13
PyTorch Version (if applicable): 1.7.1
transformers: 4.0.0

[yuanzexi]

Torch BERT models are only supported for Torch 1.3.1 API. Because BERT models exported by Torch 1.7.1 are different from that exported by Torch 1.3.1. You could use the demo on Torch 1.3.1. Thanks for your feedback. We will update soon for supporting Torch 1.7.1 BERT.

[yuanzexi]

@Fan9 The problem has been resolved in the newest master branch. You could try it again.

[Fan9]

I have successfully run BERT-Torch on torch=1.3.1 and torch=1.7.1 . Thanks for your help. Nice work!

[Fan9]

hi, @yuanzexi ,Have you tested the efficiency improvement of tensort? In my experiment, the speed of bert-trt is greatly affected by the parameters of batch_size and max_length. In my code, builder.set_mode('float32'), bert-model:=BertForSequenceClassification：

• batch_size=1, max_length=52, Bert-trt is three times faster than Bert-pytorch.
• batch_size=1, max_length=108, double times faster.
• batch_size=32, max_length=52/108，it takes the same time as Bert-pytorch.

Is there any problem here?

Environment

TensorRT Version: 7.2.1.6
NVIDIA GPU: GTX1080TI
NVIDIA Driver Version: 450.80.02
CUDA Version: 11.0
CUDNN Version: 8.0.4
Operating System: centos 7.5
Python Version (if applicable): 3.6.13
PyTorch Version (if applicable): 1.7.1
transformers: 4.0.0

[yuanzexi]

@Fan9 Thanks for your feedback. We had tested the performance of models in tensorrt, but we didn't try these configuration and environments. We'll try to profile this BERT model to figure out this problem.