missing WMT 2017 word_level/test.tags required for predictor-estimator evaluation

[unbabel]

missing WMT 2017 word_level/test.tags required for predictor-estimator evaluation

[captainvera]

Hi @ninalopatina thanks for experimenting with OpenKiwi!

The WMT 2017 test data is available on their website: here (there is a purple link with gold-standard labels)

We should probably also include a link for these in the Quickstart document to make them more visible.

On the other hand the test tags for 2018 are not public (because they are the same as 2019 which is currently accepting submissions)

I'm closing this issue as it is solved and not really a bug with OpenKiwi. Feel free to re-open if you have any other questions!

Edit: I had wrongly stated that 2017 gold files were also not available

[ninalopatina]

Thanks for looking into this so quickly. @captainvera. I had attempted to run this with the test data for 2017 & 2018, which I had obtained from the same site you linked. For both years, the test data includes only a .mt, .src, and .align file. There is no .tags file for either year, nor for 2016. Should I replace the test set links with dev set, to have a .tags files to evaluate with?

[captainvera]

Hey @ninalopatina ! The .tags file is downloaded from a different location than the other test files. It's pointed out here:

You can download the .tags from there and evaluate your model! You could also replace it with the dev set but (if you trained using the dev set as validation) that would just give you your validation scores which you should be familiar with and not a "real" evaluation.

[ninalopatina]

Thanks so much, @captainvera, I missed those links! I was thinking to test out the pipeline with the dev data until the test data becomes available, but this will work much better