BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
python 3
tensorflow 1.14
0, edit the input and output file name in create_pretraining_data.py
and run_pretraining_gpu.py
1, run create_pretraining_data.py
2, run run_pretraining_gpu.py
Edit n_gpus
in run_pretraining_gpu.py
batch_size is the batch_size per GPU, not the global_batch_size
In sample_text.txt
, sentence is end by \n, paragraph is splitted by empty line.
Quora question pairs English dataset,
Official BERT: ACC 91.2, AUC 96.9
This BERT with pretrain loss 2.05: ACC 90.1, AUC 96.3