Questions about prediction results and evaluate module

[unbabel]

Questions about prediction results and evaluate module

[trenous]

Hello HITHXX,

As you noticed the models in OpenKiwi predict a probability distribution over word tags. To transform these to tags, the most obvious decision threshold would be 0.5

The class ThresholdCalibrationMetric chooses the threshold to optimize for the F1 metric. We found in practice that tuning the bad-weight option for F1 score has a similar effect as a posteriori calibration though - so if you tuned your bad-weight, I would not expect any gains from the calibration (Actually the implementation of threshold calibration also has some issues that make it instable, but that's another topic)

The sents-avg option seems to be redundant at this point, you do not need to pass it. Word level probabilities are transformed into sentence scores by simple averaging, word level tags are first turned into 'bad-probabilities' of 0 for OK and 1 for BAD, then averaged.
We usually (but not always) observed gains of 1-3 Pearson from training sentence level directly as opposed to averaging word level tags. To achieve good performance, it is crucial that you pass the flag sentence-ll.

Could you let us now the datasets you used (for pretraining and QE training) as well as the output of the evaluate pipeline?

Best,
Sony

[HXX97]

Hello Sony:

Greatly appreciated your reply and attention. Now I know how to deal with the probability distributions.

I'm training the predictor with zh-en parallel corpora provided by CWMT 2018. As to QE training, datasets are provided by CCMT 2019.

I also tried passing the sentence-ll flag as you advised when training estimator, and the result is much more better. This is interesting and I'll try to figure out how it happens.

Anyway, thank you very much. You really did me a favor :)

[trenous]

Happy to Help :) I will change the default value of sentence-ll to True, it is unintuitive that you have to set the flag to get decent results. I will close the issue for now, but feel free to re-open it, or open a new one, if you have further questions.