ludwigstumpp/llm-leaderboard

A joint community effort to create one central leaderboard for LLMs.

llm-leaderboard's Issues

LLMZoo

A source that might be of interest to this project:
https://github.com/FreedomIntelligence/LLMZoo

Add more property columns to model rank tables.

A column for censored yes/no

A column for model size in GB (0.4 for 400MB)

A column if the AI mentions it's an AI. (Some models, though i can only think of ChatGPT at the moment, have a really annoying habit telling the user they are and AI and that they are sorry for things they as an AI can't do. Which i regard as noise. This may be service-code using the model and not really part of the model.. in that case this column would not be useful)

A column for the license. Some can be used freely some can be used free up to revenue of a certain $ amount, etc.

What are the metrics for those benchmark?

Except for pass@1 and elo rating, do other benchmarks only use accuracy for evaluation?
Yeah, I think there are big issues since most evaluation results are using their respective metrics...
For example, https://www.mosaicml.com/blog/mpt-7b use InContextLearningMultipleChoiceAccuracy rather than exact match, which is used in HELM.

@LudwigStumpp

Where is the benchmark code itself?

I mean code to get actual scores, for example for MMLU tests.

Request: Benchmark

Hi,

great work again.

Are there any possibility to include any of these model's score on the benchmark if its available?

Anthropic's Claude models: https://www.anthropic.com/product
Cohere's LLM: https://docs.cohere.com/docs/introduction-to-large-language-models
a21's Jurassic models: https://docs.ai21.com/docs/jurassic-2-models

Thanks

Request: benchmark dromedary 65b variant

HF repo

The evaluation results are retrieved from 10-shots rather than 0-shot.

As is seen from https://github.com/mosaicml/llm-foundry/tree/main/scripts/eval

Weighted Average On Page Load label:enhancement

It would be nice if the list was auto-sorted to a weighted average across all rankings on page first load. I would suggest using Trueskill - which can rank players (or models) even if they have not played each other.

Make the table reader friendly and exportable to a new page

Please make the table reader friendly by fixing the header row and fixing the first column (with the benchmark name) so that these stay in place when the table is scrolled. I think the CSS attribute position with value fixed or sticky can be used.

Furthermore please make it possible that the table can be opened alone on a new web page so that it can be printed (and exported to pdf) in a reader friendly way.

Where do you find the evaluation result for Chinchilla-70b of MMLU?

I cannot find any relevant evaluation on the linked paper...