singularitysociety/llm-benchmark

LLM benchmark

According to the leaked memo from Google titled Google "We Have No Moat, And Neither Does OpenAI", it is apparent that the open source LLMs will play very important role.

It means we need a set of "standard tests" (benchmarks) to evaluate various LLMs. While there are already some efforts by other researchers (such as "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems"), this is our attempt to compile a set of questions, which help us to identify the strength and weakness of various LLMs.

List of questions

List

Usage

• Update question list text file in text direcory.
• Update JSON file.
  • ruby scripts/ToJson.rb
• Update Question markdown file.
  • ruby scripts/JsonToMD.rb
• Run Gpt
  • GPT_API_KEY=sk-xxxx npx ts-node scripts/gpt.ts

Contribute

• Fork repository.
• Update text/{en,ja}.txt .
• Create pull request.