bert train

[nlpodyssey]

bert train

[matteo-grella]

Hey @HaoRan98, don't worry I am here to help :)

When you talk about training your own BERT model, what are you referring to? Do you want to train a language model from scratch or do you want to fine-tune a pre-trained BERT model on a particular task?

[HaoRan98]

I want to train a language model from scratch,just like chinese.

[matteo-grella]

Quick answer
Unfortunately, spaGO does not support BERT's language model training at the moment.

Long answer
I wrote the code for training a BERT language model in spaGO. You can find it here, and I'll write a tutorial with an example soon.

However, that could only be used for educational purposes, to understand how such a language model works, but not to train one seriously.

I have to tell you that creating a language model from scratch is so expensive that you wouldn't be able to use the CPU only. This is the main reason why spaGO is not ready for that yet. This is one of those cases where we need, at least for the moment, to leave GO and move to another framework that natively supports optimized GPU-based computing.

Take a look at this beautiful tutorial (for Python):
How to train a new language model from scratch using Transformers and Tokenizers

If you use the HugginFace Transformers library, you can go back to spaGO to perform inference or fine-tuning on downstream tasks once you've finished training your model.

You can find some more details about this use case on the Wiki at this page.

Have I been helpful to you?

[HaoRan98]

Thank you very much. I will take more try.