Repo to train / inference different models with GPT4ALL dataset.
This repo is very much a WIP
Currently the following are available:
- hacked together training code for GPT-J and potentially GPT-NeoX
- inference code for GPT-J
- Model LoRA weights available here: M-Chimiste/GPT-J-6B-GPT4ALL-SAMP-LORA-v1 (Current model is only trained on 50k samples of the GPT4ALL dataset as the full dataset would have taken over 5 days to train on my 2x 3090s. Hopefully we can do this when I next go on vacation).
TODO:
- Figure out how to merge lora weights with base model to get an all encompassing checkpoint.