Token classification: TypeError: 'NoneType' object is not subscriptable

[stefan-it]

Token classification: TypeError: 'NoneType' object is not subscriptable

[JoPfeiff]

Hey can you try setting "language": None. language is only used when stacking a task adapter (such as NER) on a pre-trained language adapter (such es en). You would need to additionally load a language adapter such as en/wiki@ukp to use this flag with e.g. bert-base-multilingual-cased.
Thanks for pointing this out, this is a bug and we will fix it!

[stefan-it]

Hi @JoPfeiff ,

thanks for that hint - it's working now! I made some experiments with the WNUT-17 dataset (not on CoNLL-2003 as originally stated in the issue) to compare against a fully fine-tuned model:

Model	Run 1	Run 2	Run 3	Run 4	Run 5	Avg.
Fully fine-tuned	(60.60) / 48.65	(58.58) / 47.79	(58.24) / 48.48	(61.10) / 48.06	(58.47) / 48.34	(59.40) / 48.26
Adapter (Pfeiffer)	(57.81) / 46.36	(58.31) / 45.89	(58.31) / 46.78	(55.78) / 46.51	(57.75) / 46.85	(57.59) / 46.48

There's currently a performance diff of 1.78%, so do you have any hints for boosting performance 🤔 Thanks ❤️

[JoPfeiff]

We have found that due to the newly introduced parameters, adapters require a higher learning rate. A default lr=0.0001 seems to work well in most cases. Also, depending on the dataset size you might need to train longer than usual.
We usually trained for up to 30 epochs and stored the best model based on the results on the dev set.

You can also try out the houlsby architecture or increase the bottleneck_size (currently set with reduction_factor).

We will soon add a small guide on how to train adapters where these types of tipps will be included :)

[JoPfeiff]