asappresearch/scale-score

scale-score's Issues

Issue with the prompt - scale score and sentli

@blattimer-asapp @fvivas-asapp

Below is the prompt used in the code:

'{{premise}} Question: Does this imply that "{{hypothesis}}"? Yes or no?'

Is it expected to be "no" or it should be "No". I see you are considering "Yes" & "No" tokens.

Replication for true benchmark

Hi,

Thanks for sharing your work!
I wanted to replicate results on the True Benchmark. Can you please let me know how were the sentence level scores combined to get a single score for the summary. A single inconsistent sentence makes the whole summary inconsistent I guess. Also the thresholds used at the sentence level.

Thanks.

Factual consistency of a simple sentence pair is not giving expected result.

Hi ,
I try to test this on a simple sentence and it seems the score doesn't confirm the factual consistency of the premise and hypothesis. here is the code and output

`
from scale_score.scorer import SCALEScorer
scorer = SCALEScorer(size='small', device='mps')
premise=["My name is Ibrahim Lincoln"]
hypothesis=["I am Ibrahim Lincoln"]
scorer.score(premise,[hypothesis])

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.03it/s]
[0.053020697087049484]
`

Is there something wrong with the code ? Any possible explanation of the result above ?