mikeybellissimo/lora-mpt

A repo for finetuning MPT using LoRA. It is currently configured to work with the Alpaca dataset from Stanford but can easily be adapted to use another.

license
apache-2.0

LoRA-MPT

A repo to make it so that you can easily fine tune MPT-7B using LoRA. This uses the alpaca dataset but can easily be adapted to use another.

Setup

Ensure you are using either Linux or WSL for Windows. Mac might work as well but I don't have one to test it, but Bitsandbytes won't definitely won't run for Windows.

To use as a library in another project/directory simply clone the repo, navigate to the folder and run:

pip install -e ./

or if you want to build a project within this directly just do a git clone and run and (if needed) modify the files in the src folder.

Multi-Gpu Note

For all of the following commands, substitute deepspeed for python to use with multi-gpu systems. Make sure you have deepspeed installed for this (a simple "pip install deepspeed" will normally accomplish this.)

Fine Tuning

To fine tune the MPT-7B model on the Alpaca dataset from Stanford using LoRA use the following command:

python src/finetune.py --base_model 'mosaicml/mpt-7b-instruct' --data_path 'yahma/alpaca-cleaned' --output_dir './lora-mpt' --lora_target_modules '[Wqkv]'

The hyperparameters can be tweaked using the following flags as well:

python src/finetune.py \
    --base_model 'mosaicml/mpt-7b-instruct' \
    --data_path 'yahma/alpaca-cleaned' \
    --output_dir './lora-mpt' \
    --batch_size 128 \
    --micro_batch_size 4 \
    --num_epochs 3 \
    --learning_rate 1e-4 \
    --cutoff_len 512 \
    --val_set_size 2000 \
    --lora_r 8 \
    --lora_alpha 16 \
    --lora_dropout 0.05 \
    --lora_target_modules '[Wqkv]' \
    --train_on_inputs \
    --group_by_length
    --use_gradient_checkpointing True \ 
    --load_in_8bit True

To speed up training at the expense of GPU memory run with --use_gradient_checkpointing False.

load_in_8bit defaults to True so to disable it just run the command with load_in_8bit set to False.

Inference

A Gradio Interface was also created which can be used to run the inference of the model, once fine-tuned, using:

python src/generate.py --load_8bit --base_model 'mosaicml/mpt-7b-instruct' --lora_weights 'lora-mpt'

Evaluation

For the eval library to work there are a few extra steps.

You need to install this repo from source so cd in this repo's directory on your computer and run:

pip install -e ./

Then, git clone my fork of EleutherAI's Evaluation Harness at https://github.com/mikeybellissimo/lm-evaluation-harness/tree/master#language-model-evaluation-harness and follow the instructions to download the library (Pretty much just clone it, cd into it and "pip install -e .").

Once that's done, switch back into MPT-Lora Directory and run:

python src/eval.py --model mpt-causal --model_args pretrained=mosaicml/mpt-7b-instruct,trust_remote_code=True,load_in_8bit=True,peft=lora-mpt --tasks hellaswag 

To evaluate on the hellaswag task, for example, using the LoRA weights defined in lora-mpt. You can change the task to whatever one you'd like.

MosaicML Platform

If you're interested in training and deploying your own MPT or LLMs on the MosaicML Platform, sign up here.

Note: I left this in as a thank you to MosaicML for open-sourcing their model.

Attributions

I would like to thank the wonderful people down at MosaicML for releasing this model to the public. I believe that the future impacts of AI will be much better if its development is democratized.

@online{MosaicML2023Introducing,
    author    = {MosaicML NLP Team},
    title     = {Introducing MPT-7B: A New Standard for Open-Source, 
    ly Usable LLMs},
    year      = {2023},
    url       = {www.mosaicml.com/blog/mpt-7b},
    note      = {Accessed: 2023-03-28}, % change this date
    urldate   = {2023-03-28} % change this date
}

This repo also adapted/built on top of code from Lee Han Chung https://github.com/leehanchung/mpt-lora which was adapted from tloen's repo for training LLaMA on Alpaca using LoRA https://github.com/tloen/alpaca-lora so thank you to them as well.

lora-mpt's Issues

Target modules [Wqkv] not found in the base model.

After running this command python src/finetune.py --base_model 'mosaicml/mpt-7b-instruct' --data_path 'yahma/alpaca-cleaned' --output_dir './lora-mpt' --lora_target_modules '[Wqkv]', i got this error : Target modules [Wqkv] not found in the base model. Please check the target modules and try again.

1. How can i resolve this ?
2. And during the process, the folder "templates/alpaca.json" is missing too.

Problem about generate

Thanks for the repo!

I used your code to trained the model. But when I tried to do the inference, the output is just the copy of the input. Have you met this problem before?

lora model size is always ~400bytes

I have tried multiple attempts with this I always get the adapter_model.bin as 400 bytes.. looks like its not training or saving the model.
train data is small ~2000 json lines in dolly prompt, response format.

python src/finetune.py
--base_model 'mosaicml/mpt-7b-instruct'
--data_path 'dataset/train_data.json'
--output_dir './lora-mpt'
--batch_size 256
--micro_batch_size 4
--num_epochs 100
--learning_rate 3e-5
--cutoff_len 1024
--val_set_size 200
--lora_r 4
--lora_alpha 8
--lora_dropout 0.05
--lora_target_modules '[Wqkv]'
--train_on_inputs False
--group_by_length False
--use_gradient_checkpointing True
--load_in_8bit False
--needs_prompt_generation False

Finetuning on openassis

I want to fine tune this on openassistant dataset as well. I have already fine tuned it on the Alpaca. Do I need to write a script to do so? Or better way is to mix the data...

Thanks for your help!

train/train_loss is always 0.0

I tried LoRA tuning mpt-7b and mpt-7b-instruct. I can get summary like this:
wandb: Run summary:
wandb: eval/loss nan
wandb: eval/runtime 37.2157
wandb: eval/samples_per_second 53.741
wandb: eval/steps_per_second 1.693
wandb: train/epoch 0.6
wandb: train/global_step 234
wandb: train/total_flos 2.9642010569107046e+17
wandb: train/train_loss 0.0
wandb: train/train_runtime 1465.8832
wandb: train/train_samples_per_second 20.367
wandb: train/train_steps_per_second 0.16

But the train/loss is always 0 and eval/loss is always nan. Also, when I load the model using generate.py, it always generates "&". I have tried both yahma/alpaca-cleaned and a manually created simple dataset.

Does it actually save the LoRA weights?

I ran finetune.py and finished a training of 77 steps. It did generate a folder with both adapter_config.json and adapter_model.bin. However, adapter_model.bin is just a file with 443 bytes content.

trainable params: 4194304 || all params: 6653480960 || trainable%: 0.0630392425441013
{'train_runtime': 110.8465, 'train_samples_per_second': 89.313, 'train_steps_per_second': 0.695, 'train_loss': 0.0, 'epoch': 0.99}

Any insights?

Story

Does this work for MPT story as well?

Evaluation

Hi again, I have fined tuned the Instruct model on a new dataset and now i want to evalaute the performance of the model. I followed the instruction and https://github.com/mikeybellissimo/lm-evaluation-harness successfully passed.

However when I run the script %sh python src/eval.py --model mpt-causal --model_args pretrained=mosaicml/mpt-7b-instruct,trust_remote_code=True,load_in_8bit=True,peft=lora-mpt --tasks hellaswag it raises

`Traceback (most recent call last):
File "/Workspace/Repos/LoRA-MPT/src/eval.py", line 8, in
from lm_eval import tasks, evaluator, utils
ModuleNotFoundError: No module named 'lm_eval'

Just as a note when I get list of the libraries with pip freeze, it gives me this # Editable install with no version control (lm-eval==0.3.0)
-e /Workspace/Repos/lm-evaluation-harness

Just putting the last line of the successful installation in case it helps:
Installing collected packages: lm-eval Attempting uninstall: lm-eval Found existing installation: lm_eval 0.3.0 Uninstalling lm_eval-0.3.0: Successfully uninstalled lm_eval-0.3.0 Running setup.py develop for lm-eval Successfully installed lm-eval

Appreciate any input on how to fix this.

Unsupervised training possible?

Hey, thanks for the repo!

Any ideas on how to use it to train with unlabelled data for causal language modelling? I want to adapt the foundation model to my domain first, before I do instruction fine-tuning.