Do you have export model detail example code?

[bytedance]

Do you have export model detail example code?

[SCAUapc]

I have succeefully run the demo and start the server, include use client to invoke the server... Plz give me some idea

[neopro12]

It looks like you have made very good progress. To support models trained from any deep learning framework, We defined our own model format using Google Protocol Buffers. Due to the various versions of training codes, it is somewhat useless to provide a export example directly. We will consider providing one in the next release. Now, you can get some help about how to put weights into Protocol Buffers from this link https://developers.google.com/protocol-buffers/docs/pythontutorial.

[SCAUapc]

It looks like you have made very good progress. To support models trained from any deep learning framework, We defined our own model format using Google Protocol Buffers. Due to the various versions of training codes, it is somewhat useless to provide a export example directly. We will consider providing one in the next release. Now, you can get some help about how to put weights into Protocol Buffers from this link https://developers.google.com/protocol-buffers/docs/pythontutorial.

Thanks you very much ! Another question is how can server my finetune Bert model? I have viewed this issue #12 . If i trained a bert model.
Must i need to rebulit the source to replace trtserver? And which way can i get libbert.so.fp32 like libgptlm.so.fp32? At last....Final question is the proto3... I don't know how to define proto3 to store bert parameters( such as the filed name )... Can you give me some idea? Very thanks for response!

[neopro12]

Currently we have not provided an end-to-end bert server/libbert.so in https://github.com/bytedance/lightseq/tree/master/server. But you can imitate the existing server and easily implement a bert server based on https://github.com/bytedance/lightseq/blob/master/model/encoder.cu.cc.
Based on transormer proto in https://github.com/bytedance/lightseq/blob/master/proto/transformer.proto, deleting the decoding weights will be a suitable bert proto. The definition of bert proto is free, as long as you implement the proto parsing process like https://github.com/bytedance/lightseq/blob/master/proto/transformer_weight.cu.cc
Before the end of this year, we will release a bert based on tvm, you can easily implement a very efficient bert with python, welcome to try it at that time

[SCAUapc]

@neopro12 Thank you! I am very anticipated your work ! I am sorry i forgot to reply your answer !