poethan/lepor

LEPOR: Automatic Machine Translation Evaluation Metric Model Series, with linguistic features and augmented factors

lepor's Issues

hlepor of reference is not 100%

I tried to use this version of LEPOR

I see 3 problems:

• if the hyp file is identical to the reference, the score is not 100%; in my case I get 0.997073
• there are few wanring of this type

Use of uninitialized value in string eq at my_system_level_score_A.pl line 219.

• there are lots of warnings of this type

Use of uninitialized value in addition (+) at system_level_score_A.pl line 335.

Problems with some characters

I got two errors/errors.

The first seems always present whatevere hyp and res are used
Use of uninitialized value in addition (+) at LEPOR/Sentence_level_score.pl line 373.

The second seems correlated with
Wide character in printf at LEPOR/Sentence_level_score.pl line 83, line 1744.

Actually, I am not sure if they impact on the LEPOR score.

system_level_score_A.pl vs system_level_score_B.pl

which are the differences between the two versions?

parametrization of ref and hyps

I prepared an enhanced version, which reads ref and hyp files from command line.
If you are interested I can provide it to you.

warning/errors with system_level_score_A.pl

After modifying the hardcoded filenames, langpair and task name,
I tried with a toy example containing the following 7 sentences

HYP

ciao ciao ciao ciao Ciao!
Ciao!
Ciao !
Ciao!
Ciao !
Addio!
Addio !

REF

ciao ciao ciao ciao Ciao!
Ciao!
Ciao !
Addio!
Addio !
Addio !
Addio!

I got this output with lots of warnings or errors, the results for each single sentence,
but no global results.

$>  perl system_level_score_A.pl 
Use of uninitialized value within @common_num in numeric ne (!=) at system_level_score_A.pl line 151.
Use of uninitialized value within @common_num in numeric ne (!=) at system_level_score_A.pl line 151.
Use of uninitialized value within @common_num in numeric ne (!=) at system_level_score_A.pl line 151.
Use of uninitialized value in string eq at system_level_score_A.pl line 224.
Use of uninitialized value in string eq at system_level_score_A.pl line 224.
Use of uninitialized value in addition (+) at system_level_score_A.pl line 340.
Use of uninitialized value in addition (+) at system_level_score_A.pl line 340.
Use of uninitialized value in addition (+) at system_level_score_A.pl line 340.
Use of uninitialized value in addition (+) at system_level_score_A.pl line 340.
Use of uninitialized value in addition (+) at system_level_score_A.pl line 340.
Use of uninitialized value in addition (+) at system_level_score_A.pl line 340.
Use of uninitialized value in addition (+) at system_level_score_A.pl line 340.
LEPOR-1	en__it	xxx	prova.it	0.517630550898623	No	poethan.github.com

Am I doing anything wrong?

differences in score with respect to Inten.to API

The score computed by system_level_score_A.pl and those returned by Inten.to API slightly differ: 0.517630550898623 vs 0.508832387099159

Do you know why?
Is it a matter of hype-parameters?
Do you know which are the hyper-parameters values for Italian used by Inten.to? if not, could you ask them?
It is a bit annoying having different results. Which one can I trust on?