Using BioBERT in bert-as-service

[naver]

Using BioBERT in bert-as-service

[Ayuei]

I've been using it with bert-as-service myself! The only thing you need to do is change the checkpoint names from biobert.ckpt to bert.cpkt after all BioBERT was designed to maximize compatibility with the original BERT models.

[alexferrari88]

I've been using it with bert-as-service myself! The only thing you need to do is change the checkpoint names from biobert.ckpt to bert.cpkt after all BioBERT was designed to maximize compatibility with the original BERT models.

Thank you for your answer. I did that indeed but unfortunately, when I run this code:

bert-serving-start -model_dir=C:\ML\BERT\uncased_L-24_H-1024_A-16 -tuned_model_dir=C:\ML\BERT\BioBERT\pubmed_pmc_470k -ckpt_name=biobert_model.ckpt -max_seq_len=NONE

I get this:

Traceback (most recent call last):
File "c:\program files\python37\lib\runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "c:\program files\python37\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Program Files\Python37\Scripts\bert-serving-start.exe_main.py", line 9, in
File "c:\program files\python37\lib\site-packages\bert_serving\server\cli_init.py", line 4, in main
with BertServer(get_run_args()) as server:
File "c:\program files\python37\lib\site-packages\bert_serving\server_init_.py", line 70, in init
self.graph_path, self.bert_config = pool.apply(optimize_graph, (self.args,))
TypeError: cannot unpack non-iterable NoneType object

[alexferrari88]

I also changed the names as you suggested but still I get the same error...
This is the content of my model folder:

Directory of C:\ML\BERT\BioBERT\pubmed_pmc_470k

30/04/2019 08:38

.
30/04/2019 08:38 ..
23/01/2019 08:00 212 ._bert_config.json
21/01/2019 04:21 1.307.195.224 bert.ckpt.data-00000-of-00001
21/01/2019 04:04 24.074 bert.ckpt.index
21/01/2019 04:05 52.248.151 bert.ckpt.meta
23/01/2019 08:00 313 bert_config.json
23/01/2019 08:00 213.450 vocab.txt

[Ayuei]

Interesting, I might've done it wrong myself. I just point the model directory directly to the BioBERT folder, but keep in mind that BioBERT is slightly different from BERT since it doesn't have the same Adam optimizer, which is why your command is not likely working (and getting that Nonetype error). Apart from modifying bert-as-service's source code, I recommend just running this instead:

bert-serving-start -model_dir=C:\ML\BERT\BioBERT\pubmed_pmc_470k -max_seq_len=NONE

Seeing that we're not really doing any training, I don't think there should be an issue. Also, I rename the checkpoint just because bert-as-service hardcodes bert.cpkt.

[alexferrari88]

it works now but then I wonder...would this still be fine-tuning? In this case we are loading directly the BioBERT model but not on top of BERT large...
I'm totally newbie in this so sorry if this sounds like total bs :D

[Ayuei]

That's all good, I'm somewhat of a newbie myself. But BioBERT are pre-trained models rather than fine-tuned, you can fine-tune with some task (and a smaller dataset) like Named Entity Recognition and then use it from there. So I assume it's safe to use them like this because they don't have task-specific layers added to them.

I assume the reason why there is a separate option for fine-tuning is because there is a task-specific layer that is added when doing fine-tuning and the way bert-as-service works is by pooling on the final layers of the original bert model which will change if you fine-tune because more layers are added. So what the checkpoint is for is to change the shared layer weights of the bert model to be used with the bert-as-service.

Hopefully, someone else with more experience can chime in, but this is what I think is happening.

Though I would be careful using bert-as-service to encode sentences, as from my experience the past few weeks, the distances between vectors is incredibly small and don't make much sense. I would recommend using the regular old word embeddings techniques like CBOW as it might already be sufficient for what you want to do.

[alexferrari88]

thank you for your comment @Ayuei !