cmd23333/bert-tensorflow2.x

A tensorflow 2.x BERT implementation using League of Legends myth data(Chinese).

bert-tensorflow2.x's Issues

Key bert/embeddings/word_embeddings not found in checkpoint

老哥，我用你的代码训练出来的ckpt模型，拿到bert4keras上面跑，结果报了这个错
Key bert/embeddings/word_embeddings not found in checkpoint
请问是什么问题呢

hi, finally found your code, may I share some suggestion?

First of all, your code is clean and functional! I am glad that after reading your code, the reader will definitely learn a lot. I like it, so may I share some my suggestion?
1, your mlm task has only 11% acc, I think it is because your learning rate is too high, you should try 2e-5 as the google officially suggested
2, the mlm_predict=tf.matmul(x,self.embedding.token_embedding.embeddings, transpose_b=True) code may caused some warning of "variable is not traceable". the correct code should be implemented by senting a sequence of 0 to vocab_size to the embedding layer, the output is exactly the self.embedding.token_embedding.embeddings

after doing that i think you will get 99% of mlm task accuracy as me( i got 95+% acc just after 2 epoch), and the warning will go off~~

BTW, did you graduat from tju? the same to me, Good Luck to you my friend!

怎么做预测呢

token_embedding的读取

请问训练完成后的 token_embedding 要怎么读取？

麻烦问下Num_Transformer_Layers是表示什么意思呢