Use a smaller model to speed up the prediction time

[ndenStanford]

Use a smaller model to speed up the prediction time

[Giovani-Merlin]

I have created a repository for data generation and training of bi-encoder models (so far, only for entity-linking) based on the BLINK model. In it, you can choose which bert base model to use to make your evaluation faster :). As I remember using a bert-mini I could get R@64 of 84% on zeshel dataset.*

But no cross-encoder was implemented, so, you can make only faster the bi-encoder part.

[mitchelldehaven]

@Giovani-Merlin Is that repo still active? The link is now dead.

[abhinavkulkarni]

If you have your own training data, it's not hard to modify the code slightly to use the latest and smaller HuggingFace BERT models (such as BERT mini or google/bert_uncased_L-8_H-512_A-8) for training biencoder.

You'll need to change how the base model is loaded (use HuggingFace's AutoModel and AutoTokenizer classes) and how the tokenized input is fed to the model (input_ids, token_type_ids and attention_mask).

I tried training the Zeshel model after making these changes and training seemed to go on fine.