Adding German to MTEB

[achibb]

Adding German to MTEB

[rasdani]

Hey there :)... currently working on implementing a retrieval benchmark based on GermanQuAD. Will publish a PR soon and keep you updated here. If you want to chat about it/join the discussion, here's a link to the DiscoResearch discord: https://discord.gg/FBvnqsDS

[rasdani]

Working on it here:
https://github.com/DiscoResearch/mteb/tree/germanquad-retrieval

Here are first results for intfloat/multilingual-e5-small on the test split of deepset/germanquad.

INFO:root:MRR@1: 0.8720
INFO:root:MRR@3: 0.9091
INFO:root:MRR@5: 0.9130
INFO:root:MRR@10: 0.9139
INFO:root:MRR@100: 0.9149
INFO:root:MRR@1000: 0.9149

Are the scores on the actual HF leaderboard multiplied by 100 or why are they in the range 0-100? 🤔

And I couldn't find the code for the actual HF space. Is it not open source?

[Muennighoff]

Great work! For running the evaluation, they have a section on their HF hub page: https://huggingface.co/intfloat/multilingual-e5-large#mteb-benchmark-evaluation

Therefore only one matching context can be retrieved from the corpus and MRR would be the best metric to score this, correct?

I think you can still use nDCG but MRR is fine too

[KennethEnevoldsen]

Given the merged PR it seems like this issue is resolved? Though we might still be missing a german tab on the leaderboard. If that is the case we can create a separate issue on this.

[achibb]

Yes will close this issue then as lots of good development. Thanks all!

Will open a new one here for the German tab and see if / how I can support !

[Muennighoff]

Yeah that'd be great. I'd be happy to add an Overall German tab once we have ~30 datasets (https://huggingface.co/spaces/mteb/leaderboard)

Note that for Clustering there are some German datasets already thanks to @slvnwhrl who may also be interested in helping out with this effort.

We should aim to minimize MT and use as many human-written datasets as possible I think.
Some datasets from here #174 may also be available in German

[slvnwhrl]

Hi,

@Muennighoff thanks for including me. I'd be happy to help. And I agree, there should be enough German open source datasets out there, at least for some of the tasks. To give som suggestions:

Classification:

• SCARE - The Sentiment Corpus of App Reviews with Fine-grained Annotations in German
• Germeval Task 2017: Shared Task on Aspect-based Sentiment in Social Media Customer Feedback
• GermEval2021 - Toxic, Engaging, & Fact-Claiming Comments
• GermEval 2019 Task 1 -- Shared task on hierarchical classification of blurbs (note that the current German clustering benchmark uses the same data)
• Ten Thousand German News Articles Dataset (note that the current German clustering benchmark uses the same data)

Reranking:

• GermanQuAD and GermanDPR

These are some German datasets that come to my mind at the moment. I am sure there are more, although, for some tasks it might be harder to find good datasets. I also haven't checked all of the licenses of the listed datasets. This repository could also be of help: German-NLP

[Muennighoff]

Great! Yes they are multiplied by 100 to be from 0-100 in order to make it more readable :)
Everything is open-source - Do you mean this code https://huggingface.co/spaces/mteb/leaderboard/blob/main/app.py ?

[rasdani]

Ah yes thank you! :) I knew about the "Files" tab in spaces but somehow overlooked it this time 😅

I'm currently testing with intfloat/multilingual-e5-small.

As you might know these need to be prompted in a specific way for full capability:
https://huggingface.co/intfloat/multilingual-e5-large#usage

I can't find a corresponding model class in mtebscripts although the e5 embeddings are on the leaderboard.

And how do I contribute a benchmarks specifically?
Everything runs fine with my run_mteb_german.py.

As I understand the README and the other scripts, that's enough and you take care of running it for different embeddings?

If so the only things I see left to do are:

• clean/finish up my fork a bit
• host the GermanQuAD dataset in BEIR format on HF instead of generating it locally

I will create and link a draft PR in a minute, so you can compare the changes more easily ;)

[rasdani]

Draft PR: #197

[rasdani]

one more note: deepset/germanquad has only one relevant context per question.
Therefore only one matching context can be retrieved from the corpus and MRR would be the best metric to score this, correct?

[malteos]

FYI there is this German fork already https://github.com/jina-ai/mteb-de?ref=jina-ai-gmbh.ghost.io

[Muennighoff]

FYI there is this German fork already https://github.com/jina-ai/mteb-de?ref=jina-ai-gmbh.ghost.io

Nice, do you plan on opening a PR? Would be to help 🙌