yg211/summary-reward-no-reference

A reference-free metric for measuring summary quality, learned from human ratings.

summary-reward-no-reference's Issues

Training details

Hi,
Congratulations on your work and thanks for sharing your code!

I am doing a research on summarization and now I am trying to use the code.
I would like to train the model on the other dataset.
Would you please share some details on the training steps when training the model on CNN-DM dataset?
Such as hyperparameters, the time spent to train the code and when (=which epoch) did you stoped the training.

ps) 50 epochs of training, which is the example command-line option, took only a few minutes (like less than an hour) on my GPU. Is this normal?

Thanks,
Wonjin

Cannot download ROUGE package

Is there another way to download the ROUGE package? The ISI website isn't responding.

How to interpret the values

Thank you for this great contribution, I'm sure it will help developing RL summarization systems.

One thing I don't understand is how to interpret the values return from the rewarder. I'd assume that higher scores indicate higher-quality summaries. Running a few tests, the values are not what I expected:

rewarder = Rewarder(os.path.join('trained_models', 'sample.model'))
doc = '''Bilbo was very rich and very peculiar, and had been the wonder of the Shire for sixty years, ever since his remarkable disappearance and unexpected return. The riches he had brought back from his travels had now become a local legend, and it was popularly believed, whatever the old folk might say, that the Hill at Bag End was full of tunnels stuffed with treasure. And if that was not enough for fame, there was also his prolonged vigour to marvel at. Time wore on, but it seemed to have little effect on Mr. Baggins. At ninety he was much the same as at fifty.'''

    summ1 = '''Bilbo was very rich and at  age ninety as vigorous as at fifty.'''
    summ2 = '''Bilbo was very wealthy and peculiar, the riches he brought back from his journey made him a local legend. He was also very vigorous for his age.'''
    summ3 = '''The lord of Bag End is called Bilbo the Mighty and he is known for ruling the Shire with an iron fist.'''
    summ4 = '''Last weekend, a man died after a car crash.'''

    print(
        rewarder(doc, summ1),
        rewarder(doc, summ2),
        rewarder(doc, summ3),
        rewarder(doc, summ4)
    )
   outputs: -1.828371 -0.8733603 -0.02868136 -0.747489

Am I using it incorrectly or do I need to apply any kind of preprocessing beforehand? If this is the correct usage, is this just an unfortunate example / out of domain?

Also, when using a cpu for inference, the torch.load function in rewarder.py needs an additional parameter, as it defaults to cuda.

 self.reward_model.load_state_dict(torch.load(weight_path, map_location=torch.device(device)))

Kind Regards,
Michael

Issue with the environment

Hi!
I met this error when running python compare_reward.py after installing the required packages:

I have CUDA 11.7 and pytorch 1.2.0. May I ask whether I missed something in the installation? How should I resolve the issue?

Thanks a lot for your help!