Word level QE reproduction

[zouharvi]

Word level QE reproduction

[captainvera]

Hey @zouharvi, thanks for your interest in OpenKiwi and the detailed issue!

I believe it is not an error that you're making but a misinterpretation of the results.
What we model is the probability of a word being BAD and not the probability of a word being OK. With that in mind, the results you're getting are completely expected 🙂

See below:

Golden Tags (removed gaps)

 OK  OK  OK  OK  OK  OK  BAD  OK  BAD  OK  OK  OK  OK 

Your results

 OK  OK  OK  OK  OK  OK BAD BAD BAD  OK  OK  OK  OK

The model is actually only getting one tag wrong. So it's getting around 92% accuracy, not too bad!

[zouharvi]

Thank you for your reply,

What we model is the probability of a word being BAD

This is somewhat counterintuitive to what we expected from QuEst++ and deepQuest, but we're glad it has been resolved. The data now makes sense. 🙂

We'll rerun our experiments on WMT17 en_de and our custom cs_de data and will let you know of the results (cs_de didn't work out the last time).

[captainvera]

Great, glad I could help you guys.

I'll close the issue for now, but feel free to reopen it if you have any doubts about your results with the cs_de data!