braintrustdata/braintrust-sdk

Braintrust SDK

Braintrust is a platform for evaluating and shipping AI products. To learn more about Braintrust or sign up for free, visit our website or check out the docs.

This repository contains the Python and Javascript SDKs for Braintrust. The SDKs include utilities to:

• Log experiments and datasets to Braintrust
• Run evaluations (via the Eval framework)
• Manage an on-premises installation of Braintrust (Python)

Quickstart: Typescript

First, install the Braintrust SDK:

npm install braintrust autoevals

or

yarn add braintrust autoevals

Then, create a file named tutorial.eval.ts with the following code:

import { Eval } from "braintrust";
import { LevenshteinScorer } from "autoevals";

Eval("Say Hi Bot", {
  data: () => {
    return [
      {
        input: "Foo",
        expected: "Hi Foo",
      },
      {
        input: "Bar",
        expected: "Hello Bar",
      },
    ]; // Replace with your eval dataset
  },
  task: (input) => {
    return "Hi " + input; // Replace with your LLM call
  },
  scores: [LevenshteinScorer],
});

Then, run the following command:

BRAINTRUST_API_KEY=<YOUR_API_KEY> \
    npx braintrust eval tutorial.eval.ts

Quickstart: Python

Install the library with pip.

pip install braintrust

Then, create a file named eval_tutorial.py with the following code:

from braintrust import Eval
from autoevals import LevenshteinScorer

Eval(
  "Say Hi Bot",
  data=lambda: [
      {
          "input": "Foo",
          "expected": "Hi Foo",
      },
      {
          "input": "Bar",
          "expected": "Hello Bar",
      },
  ],  # Replace with your eval dataset
  task=lambda input: "Hi " + input,  # Replace with your LLM call
  scores=[LevenshteinScorer],
)

Then, run the following command:

BRAINTRUST_API_KEY=<YOUR_API_KEY> \
  braintrust eval eval_tutorial.py

Documentation

For more information, check out the docs:

• Typescript
• Python

braintrust-sdk's Issues

ScoreSummary should actually be MetricSummary

braintrust-sdk/py/src/braintrust/logger.py

Line 2742 in 9a74696

metrics: Dict[str, ScoreSummary]

TYPO : Your main README's title has a typo Brainrust SDK -> Braintrust SDK

I was deeply impressed by what Braintrust is aiming to do! I do agree that we need a nicely integrated interface that can evaluate the performance of AI models, especially LLMs.

Hope you guys keep up the good work and add more amazing features. Thanks!