Biobert custom vocab

[naver]

Biobert custom vocab

[jhyuklee]

Hi @LivC182,

1. When you compare NER/QA Results you compare the bert-large vs bert-base architectures. So how can I be sure that the improved results are due to vocab and not arhitecture (bert-large is, well larger, 340M parameters vs 110M for the base model).

We didn't compare BERT-large and BERT-base in our paper. There are comparisons among BERT-base and variants of BioBERT-base. For BioBERT-large, which we only tested on few datasets (link), the improvement over BioBERT-base might be from the vocab or the architecture. We didn't perform an extensive ablation study.

2. If you have a different vocabulary, I assume you first retrain the original BERT and then domain specific pre-train it on pubmed abstracts. My question is do you only change the vocabulary or also the tokenizer used in the original BERT? Also is this vocab specifically designed for biomedical texts, or a more robust version of the original one (meaning it can be still applied to general texts)

Vocab for BioBERT-large is specifically designed for BioBERT-large. We used SentencePiece by Google.