Hi,
I want to reproduce the same (or at least very similar) m2 scores on the official conll14 test set. Following the README file, I successfully set up the environment and could get some results by the following command:
python2 models/run_gecsmt.py \
-f models/moses.dense-cclm.mert.avg.ini \
-w reproduce/ \
-i conll14st-test/noalt/official-2014.combined.m2 \
--m2 \
-o reproduce/conll.out \
--moses $PWD/build/mosesdecoder \
--lazy $PWD/build/lazy \
--scripts $PWD/train/scripts
The output file was supposed to be almost (if not exactly) the same with your submission, and so should the m2 scores be. However, I only got the following m2 scores:
Precision : 0.5977
Recall : 0.2794
F_0.5 : 0.4868
while the reported F0.5 is 0.4893, which is what I was expecting.
I vimdiffed my output against yours, and found that my output contained a few casing mistakes while yours doesn't. For example, in the middle part of sentence 333, my output was:
... doctors to disclose information To Patients Relatives.It challenges The Confidentiality and privacy principles.Currently , under the Health Insurance Portability and ...
The bolded tokens look suspicious. Here their first letters are all capitalized, but the original input is not. Your output looks fine, too.
I digged a little into the script: models/run_gecsmt.py
, and realized maybe there is something wrong during the recasing phase? More specifically, at line 78:
|
run_cmd("cat {pfx}.out.tok" \ |
|
" | {scripts}/impose_case.perl {pfx}.in {pfx}.out.tok.aln" \ |
|
" | {moses}/scripts/tokenizer/deescape-special-chars.perl" \ |
|
" | {scripts}/impose_tok.perl {pfx}.in > {pfx}.out" \ |
|
.format(pfx=prefix, scripts=args.scripts, moses=args.moses)) |
It looks like we are recasing the output (tokenized) using the raw input (untokenized) and the alignment file. I suspect this is incorrect because the alignment file is based on the tokenized files, and we should do something like this:
{scripts}/impose_case.perl {pfx}.in.tok {pfx}.out.tok.aln
I did try doing so. While I successfully got the correct cases for the example above, now all sentence beginning letters are in lowercase too.
This got me totally confused. How can I get the expected results and scores? What seems to be the problem? Could you shed some light?
For your reference, I also attached my output and logs here.
run.log
conll.out.txt