archie-mckenzie/spellsynth

A system for fine-tuning powerful LLM translators

Spellsynth

Currently under construction. Please check back later.

Introduction

Spellsynth is system for creating synthetic datasets to fine-tune powerful LLM translators.

In addition to code and instructions, this repo contains two Spellsynth fine-tuning datasets, each of about 20,000,000 tokens:

• spellsynth-el
• kaleidograph

The datasets in data/final are freely licensed. They are intended as fine-tuning data for state-of-the-art LLMs like Mistral's mixtral-8x7b, Meta's llama models, or OpenAI's gpt-4.

In the interests of reproducibility, this research has used only open weights models and publicly available data.

Prerequisites

• Python >3.11.6 installed
• pip installed

Setup

Create a Virtual Environment

python3 -m venv myenv
source myenv/bin/activate  # On Windows, use `myenv\Scripts\activate`

Install Dependencies

pip install -r requirements.txt

Environment Variables

Create a .env file in the project root directory and add the following:

TOGETHER_API_KEY='[YOUR API KEY]'

You only need API keys for the platforms you are using. Any platform that can serve language models like mixtral-8x7b-instruct (or equivalent) and embedding models like m2-bert-80M-2k-retrieval (or equivalent) will likely work. I have used Together.ai.

Data

Sourcing

Specification

Example