yangrui2015/ric

Code for the ICML 2024 paper "Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment"

Hi there 👋

• 🏫 I'm a grad student at Department of Computer Science and Engineering, HKUST.
• 🔬 mainly study Machine Learning and Reinforcment Learning.

ric's Issues

ValueError: too many dimensions 'str' when running eval_rewarded_soups.py

I meet error

File "/.../python3.10/site-packages/transformers/tokenization_utils_base.py", line 748, in convert_to_tensors
   tensor = as_tensor(value)
File "/.../python3.10/site-packages/transformers/tokenization_utils_base.py", line 720, in as_tensor
    return torch.tensor(value)
ValueError: too many dimensions 'str'

when running eval_rewarded_soups.py. And this issue didn't happen when running eval_ppo_single_model.py.

It should be fixed by adding

valid_batch_size = 1
valid_dataset = valid_dataset.remove_columns(['prompt', 'query'])
for key in ['key', 'text', 'response']:
    if key in valid_dataset.column_names:
        valid_dataset = valid_dataset.remove_columns(key)

Is this correct?

Questions on dataset processing

In ric/utils.py line 223, I guess the sample["input_ids"] = tokenizer.encode(sample["text"]) should be sample["input_ids"] = tokenizer.encode(sample["prompt"]) instead.

A question on plot.py

Why should the pref_list be reversed?

if 'helpful' in path and 'humor' in path:
        if reverse:
            obtained_scores = obtained_scores[:, ::-1]
        if len(pref_lis):
            pref_lis = [1 - x for x in pref_lis]