ludwigstumpp/llm-leaderboard

A joint community effort to create one central leaderboard for LLMs.

👋 Hi, I'm Ludwig

My ultimate goal is to spend my 80,000 working hours with a lasting effect on our world. I believe that technology is there to solve problems, so let's start with engineering today, for a better planet.

Currently on a mission to build the largest AI Ecosystem in Europe together with appliedAI and IPAI at Heilbronn.

I believe in a more enlightened world where there are less prejudices against new perspectives and upcoming technology, so I do my best to contribute to a world where access to high quality educational content is not restricted to the priviledged only. Doing so by contributing to Open-Source projects.

🤙Want to read more?

Too keep things DRY I have compiled an overview about my current projects at my website. Happy to see you there!

llm-leaderboard's Issues

LLMZoo

A source that might be of interest to this project:
https://github.com/FreedomIntelligence/LLMZoo

Where do you find the evaluation result for Chinchilla-70b of MMLU?

I cannot find any relevant evaluation on the linked paper...

Weighted Average On Page Load label:enhancement

It would be nice if the list was auto-sorted to a weighted average across all rankings on page first load. I would suggest using Trueskill - which can rank players (or models) even if they have not played each other.

There evaluation results are from 10-shots rather than 0-shot.

As is seen from https://github.com/mosaicml/llm-foundry/tree/main/scripts/eval

Request: benchmark dromedary 65b variant

HF repo

Request: Benchmark

Hi,

great work again.

Are there any possibility to include any of these model's score on the benchmark if its available?

Anthropic's Claude models: https://www.anthropic.com/product
Cohere's LLM: https://docs.cohere.com/docs/introduction-to-large-language-models
a21's Jurassic models: https://docs.ai21.com/docs/jurassic-2-models

Thanks

Where is the benchmark code itself?

I mean code to get actual scores, for example for MMLU tests.

Make the table reader friendly and exportable to a new page

Please make the table reader friendly by fixing the header row and fixing the first column (with the benchmark name) so that these stay in place when the table is scrolled. I think the CSS attribute position with value fixed or sticky can be used.

Furthermore please make it possible that the table can be opened alone on a new web page so that it can be printed (and exported to pdf) in a reader friendly way.

What are the metrics for those benchmark?

Except for pass@1 and elo rating, do other benchmarks only use accuracy for evaluation?
Yeah, I think there are big issues since most evaluation results are using their respective metrics...
For example, https://www.mosaicml.com/blog/mpt-7b use InContextLearningMultipleChoiceAccuracy rather than exact match, which is used in HELM.

@LudwigStumpp

Add more property columns to model rank tables.

A column for censored yes/no

A column for model size in GB (0.4 for 400MB)

A column if the AI mentions it's an AI. (Some models, though i can only think of ChatGPT at the moment, have a really annoying habit telling the user they are and AI and that they are sorry for things they as an AI can't do. Which i regard as noise. This may be service-code using the model and not really part of the model.. in that case this column would not be useful)

A column for the license. Some can be used freely some can be used free up to revenue of a certain $ amount, etc.