xiangli1999/contrastivedecoding

contrastive decoding

contrastivedecoding's Issues

How to add new model

Hi, I am wondering how I can add new model into this repo like mistral and llama.

Could you provide scripts to reproduce the results?

@XiangLi1999 Thanks for your great work!

I am trying to reproduce the results on wikitext but meet some problems.

I use your script:

python run_generation.py --model_name_or_path gpt2-xl --model_type gpt2 --length 256 --prompt_file wikitext --student_name_or_path gpt2 --st_coef 1.0   --student_temperature 0.5  --outfile outputs/temp_out.jsonl    --ignore_prefix no

And then evaluate the output file by:

python eval_script.py ./outputs/temp_out.jsonl

The output is

{'name': './outputs/temp_out.jsonl', 'rep-2': 9.5, 'rep-3': 1.87, 'rep-4': 0.4, 'diversity': 0.8845241939999999, 'mauve': 0.8812567264373257, 'coherence': 0.5913593170305366} (I disable other metrics)

which is different from reported results in the paper (coherence = 0.59 v.s. 0.69).

I find that ./outputs_ignorePrefix_ccnews_256/wikitext_results/wikitext_gpt2-1.0-t0.5_gpt2-xl_256.jsonl can produces correct metric values. May I ask two questions:

1. What is the generation script used to produce the correct outputs?
2. What does the values in wikitext_gpt2-1.0-t0.5_gpt2-xl_256.jsonl mean? For example, 256 seems output length, 0.5 is student temperature. What does 1.0 indicate?

Question about amateur context window

Dear @XiangLi1999 and @ari-holtzman,
if I understand correctly the paper, in section 3.4, mentions that the amateur (student) model is conditioned on a context window which starts from the last token of the prompt. I cannot find any trace of such a choice in the code, for instance here and here the whole input is passed to the amateur model, as seen by the expert too.
I cannot find the relative study in the ablation script either.

Am I missing some argument/logic that sets the amateur's context window somewhere else in the code?

Best,
Marco

What does coherence score refer to in the paper?

Seems no citation is given.

Looking forward to seeing the cleaner version.

Why change code in the whole Transformers library?

Thanks for open source the Contrastive Decoding, it is really an interest and smart method.
I found you have copied the whole Transformers library, and just changed around two files.
Why not use python dynamic binding to patch Transformers generation methods? I think it will be easier to maintain and debug.
e.g.

import torch
import transformers.generation_utils as gu

@torch.no_grad()
def generate(self, ...):
    ...

@torch.no_grad()
def sample(self, ...):
    ...

gu.GenerationMixin.generate = generate
gu.GenerationMixin.sample = sample

Question about ignore_prefix flag

Hi @XiangLi1999 and @ari-holtzman,
Thanks for releasing your codebase and interesting work! I had a quick question about the --ignore_prefix flag. Would you suggest setting this to yes or no? In the ablation experiments in Table 5, it seems like setting it to yes was slightly better? However, the default setting in the README file is no.

I was wondering how much this setting affects MAUVE score and quality, and what would you suggest?

Best Wishes,
Kalpesh

Question about typical decoding baseline

Hello @XiangLi1999 and @ari-holtzman,
I'm trying to follow the implementation of the typical decoding baselines in the code base.
Here a typical argument is passed, but the generate method does not accept any such argument but only an optional argument typical_p. At the same time I am unable to find any call to model_kwargs['"typical"] anywhere so it seems no warping of the logits to do typical sampling is ever performed.
Am I missing something trivial?

Getting the error: "ValueError: The following `model_kwargs` are not used by the model:....."

@XiangLi1999 Hello there, I am trying to run your code as it is with the commands provided with no changes whatsoever, however, I am still getting the following error.
"ValueError: The following model_kwargs are not used by the model: ['min_prob', 'student_lm', 'teacher_student', 'model_kwargs_student', 'st_coef', 'tokenizer', 'student_min_prob', 'student_temperature', 'use_cap_student', 'use_switch'] (note: typos in the generate arguments will also show up in this list)"

Can you help me with this? Is there any specific way to rectify this or am I supposed to simply drop these arguments from the parts where they are coded?

pytorch version for reproducing the results

Hi, Lisa! Thanks for the great work and many thanks for releasing the code and data!

I just found the results of contrastive decoding can be a little different with different pytorch versions.
Could you please advise which specific pytorch version you were using when producing those *jsonl files?

Many thanks.