What should I do to run bert with lightseq?

[bytedance]

What should I do to run bert with lightseq?

[Taka152]

@itsucks We have implemented BERT in the model/encoder.cu.cc, but because of the diversity of downstream tasks, we didn't provide a full example. However, it can be done by the following procedures.

1. Write an encoder server based on server/transformer_server.cu.cc to get the last layer's output.
2. Use the output to compute the final result for your task.

This can be fast enough because most computing time is in Transformer blocks if your downstream task is not too heavy(like two FFN layers).

And also we are implementing a python wrapper to replace TRTIS, which provides flexibility to use LightSeq in your deployment code, it will be released as soon as possible.