Add documentation how to use loaded adapters without prediction head

[arueckle]

Add documentation how to use loaded adapters without prediction head

[arueckle]

Yep, referring to adapter_names. We are in the process of refactoring/simplifying this part such that setting active adapters will also work with models that are not based on BertModelWithHeads (see #44), that's why extending the doc also takes a bit longer. We'll have an update soon.

I think adapter composition might be the more accurate term as you can compose different adapters by stacking and/or fusing them, such as in:

logits = self.masked_bert_model(
                input_ids=tokens_tensor.to(self._model_device),
                token_type_ids=segments_tensor.to(self._model_device),
                attention_mask=attention_mask_tensor.to(self._model_device),
                adapter_names=[[NAME_OF_ADAPTER1], [NAME_OF_ADAPTER2]]   # Adapter Stacking
            )

[sosuperic]

Is this referring to / related to https://github.com/Adapter-Hub/adapter-transformers/blob/master/src/transformers/modeling_bert.py#L679? Or is adapter composition something else?

If I'm understanding correctly, if you're using a BertModelWithHeads (i.e. you trained a Task Adapter with AdapterType.text_task), then you can just call set_active_adapters. However, set_active_adapters is only defined for BertModelHeadsMixin.

In my case, I trained a Language Adapter, where I continued pre-training BERT on my own dataset. This model doesn't have prediction heads (i.e. I can't do the above), so I had to do the following in my forward pass. This took me some time to find as well -- I don't think it's in the docs, and I had to search through the code.

self.masked_bert_model = BertForMaskedLM.from_pretrained(bert_model_name)
adapter_name = self.masked_bert_model.load_adapter(ADAPTER_DIR)

...

logits = self.masked_bert_model(
                input_ids=tokens_tensor.to(self._model_device),
                token_type_ids=segments_tensor.to(self._model_device),
                attention_mask=attention_mask_tensor.to(self._model_device),
                adapter_names=[NAME_OF_ADAPTER]   # MUST PASS THIS IN
            )