Currently under construction. Please check back later.
Spellsynth is system for creating synthetic datasets to fine-tune powerful LLM translators.
In addition to code and instructions, this repo contains two Spellsynth fine-tuning datasets, each of about 20,000,000 tokens:
spellsynth-el
kaleidograph
The datasets in data/final
are freely licensed. They are intended as fine-tuning data for state-of-the-art LLMs like Mistral's mixtral-8x7b
, Meta's llama
models, or OpenAI's gpt-4
.
In the interests of reproducibility, this research has used only open weights models and publicly available data.
- Python >3.11.6 installed
- pip installed
python3 -m venv myenv
source myenv/bin/activate # On Windows, use `myenv\Scripts\activate`
pip install -r requirements.txt
Create a .env
file in the project root directory and add the following:
TOGETHER_API_KEY='[YOUR API KEY]'
You only need API keys for the platforms you are using. Any platform that can serve language models like mixtral-8x7b-instruct
(or equivalent) and embedding models like m2-bert-80M-2k-retrieval
(or equivalent) will likely work. I have used Together.ai.