python evaluate_single_dataset.py dev.json dev.json does not result in 100% F1

[jerbarnes]

python evaluate_single_dataset.py dev.json dev.json does not result in 100% F1

[janpf]

Hi Jeremy,
thank you for your quick reply and quick fix!

[jerbarnes]

Hi,

No you're correct. After looking into it, it's not actually a problem with the evaluation script, but again with the data. There were some examples where the polar expression was incorrectly annotated starting from the inside of one of the tokens. This led to errors when parsing them in the evaluation script that gave empty polar expressions. The main assumption the evaluation script makes is that there can be no empty polar expressions, so this was a problem.

I've updated the processing scripts for both MPQA and Darmstadt to remove these problems, so it will be necessary to reprocess them.