Pre-training FlauBERT with Google Colab?

[getalp]

Pre-training FlauBERT with Google Colab?

[schwabdidier]

definitely, it depends on corpus size. You will find exact values in the article but for 71 Go of text (see https://arxiv.org/pdf/1912.05372.pdf) :

FlauBERTBASE is trained on 32 GPUs Nvidia V100 in 410 hours ( = 546 days) and FlauBERTLARGE is trained on 128 GPUs in 390 hours (=2080 days). It is too huge for google collab

If it is a specialized corpus, consider starting from one Flaubert model and continue learning with your own corpus for instance as describe in this article (https://arxiv.org/pdf/1901.08746.pdf). Even in this case, I am not sure it will be possible as your corpus is certainly too big.