yh646492956/bert-cpp-inference

Deploying Deep Learning Models in C++: BERT Language Model

This repository show the code to deploy a deep learning model serialized and running in C++ backend.

For the sake of comparison, the same model was loaded using PyTorch JIT and ONNX Runtime.

The C++ server can be used as a code template to deploy another models.

Results

The test consist of make 100 async calls to the server. The client measure the time taken to get the response of the 100 requests. To reduce variability it is performed 10x, i.e, 1000 requests.

• The results was obtained in a Macbook Pro with a i7 6C-12T CPU.
• The CPU fans was set the max to reduce any possible thermal-throttling.
• Before start each test the temps was lowered down around 50 C.

Model Details

• Model: BERT Large uncased pretrained from hugginface.
• Input: Text with [MASK] token to be predicted
• Output: Tensor in shape (seq_len, 30522)

Python server details

FastAPI web-app server running by hypercorn with 6 workers (adjust this by the number of threads you have).

• server-python/app_pt.py: Load the model using PyTorch load_model(), the standard way of load a PyTorch model.
• server-python/app_jit.py: Load the serialized model using PyTorch JIT runtime.
• server-python/app_onnx.py: Load the serialized model using ONNX runtime.

C++ server details

To deploy the model in C++ it was used the same serialized model used in JIT runtime. The model is loaded using libtorch C++ API. To deploy as a http-server, it is used crow. The server is started in multithreaded mode.

• server-cpp/server.cpp: Code to generate the c++ application that load the model and start the http server on port 8000.

Deploy a model in C++ is not as straightforward as it is in Python. Check the libs folder for more details about the codes used.

Client details

Client uses async and multi-thread to perform as many requests as possible in parallel. See the test_api.py

TO-DO

• Experiment on CUDA device.
• Experiment another models like BERT.
• Exprimento quantized models.

References

• hugginface
• PyTorch Tutorials
• PyTorch C++
• Crow C++ microframework for web
• Serving PyTorch Models in C++
• Deep Learning in Production
• Book: Hands-On Machine Learning with C++: Build, train, and deploy end-to-end machine learning and deep learning pipelines