dahoas/reward-modeling

Update 4/21

This repo is being deprecated for CarperAI's autocrit (see this pr)

reward-modeling

This is a research repository for training and evaluating reward models. Code is also included to train supervised fine-tuned base models.

Training Example

Running bash scripts/train_rm/run_gptj.sh will train a gptj reward model using train config configs/rm_configs/gptj.yaml by default.

Note: To do eval on gptj please install transformers repo from source.

Loading Models

Loading models is a bit convoluted so I attach an example here. The reward models are not implemented as HF models and so cannot simply be loaded via a .from_pretrained(MODEL_NAME) call.

Get model weights from hf: wget https://huggingface.co/Dahoas/pythia-6b-rm-synthetic/blob/main/hf_ckpt.pt

      import torch
      from utils import make_rm
      # save_model is used to determine whether a reference to the base model is saved in the RM wrapper (this is necessary to use HF's Activation Checkpointing code)
      save_model = False
      rm = make_rm("EleutherAI/gpt-j-6B", "causal", "EleutherAI/gpt-neox-20b", save_model)
      rm.load_state_dict(torch.load(PATH_TO_CKPT), strict=True)

reward-modeling's Issues

Wrong submit

Get Error, when run "./run_gptj.sh"

Hi Dahoas, thanks your release!
When run "./run_gptj.sh", Get Error:

state dict key error while loading pretrained rm

Thank you for sharing this work!

I found following README scripts to load pretrained rm weights leads to key errors.
I tried both Dahoas/pythia-6b-rm-synthetic and Dahoas/gptj-rm-static.
First one has both missing keys and unexpected keys; Second one has only unexpected keys.
So I guess I should be using Dahoas/gptj-rm-static.

This may be because Dahoas/gptj-rm-static weight has duplicate keys starting with "model" and without "model" flag.

If this was intended, can you explain how I can load this weight properly? Thanks!

Could you make a Google Colab? I would like to try training using the reward modeling strategy, but my coding abilities are limited.