dev loss

[taolei87]

dev loss

[taolei87]

Hi @christinazavou
Sorry for such a late response. Not sure why I missed the notification of this post..

The observation is interesting. I didn't take a look at dev loss before.
I guess this is due to the difference of how training and evaluation datasets were constructed.

• The training set is a combination of positive pairs + randomly sampled negative pairs;
• The evaluation set is constructed in a re-ranking setup. That is, the 20 candidates of each query question were generated / retrieved by the Lucene search engine.

It is easy to separate random negative pairs from positive ones. So the training loss keeps decreasing. However, the dev set contains candidates that are much more similar to the query (because they are the outputs of a search engine). The devloss0 keeps at around the margin value (1.0), which suggests that there exists a challenging negative candidate not separated from the positive. devloss1 (pairwise loss) does decrease which suggests overall most negatives are gradually separated from the positive.

This is not an data issue i think. The training loss encourages the model to separate positives from average negative samples, not the most challenging negative samples.