Jaccard metric bug?

[google-deepmind]

Jaccard metric bug?

[cdoersch]

https://arxiv.org/abs/2009.07736 is a careful analysis of object tracking evaluation metrics, and it's the paper we used as the basis for making these decisions. In particular, in figure 2 makes it clear that they're advocating for this similar approach: a single frame is counted as both a false positive and a false negative.

[cdoersch]

Hi @serycjon ,

I'm not sure I understand the issue, as I believe the behavior you're describing is correct for the Jaccard metric. Jaccard is intersection over union: true positives/(true positives + false positives + false negatives). If the algorithm outputs a point that is outside the threshold, this is counted as both a false positive and a false negative. Therefore, a single ground-truth point will count twice in the denominator. gt_positives counts true positives and false negatives; false_positives counts false positives.

In the tracking literature, this corresponds to human judgements that predicting an incorrect track is a worse error than simply losing track of an object.

Please let me know if I'm misunderstanding.

[serycjon]

Hi @cdoersch, thanks for quick reply.

Yeah, I understand Jaccard as IoU, or TP / (TP + FP + FN).
I just find the idea of counting a prediction as both a false positive and a false negative very unusual.
As far as I know, the FP and the FN sets should be disjoint. (BTW I wouldn't really call it intersection over union when the denominator is bigger than the union (some elements counted twice)).

Do you know about some prior work that uses this (in my opinion) unusual way of computing Jaccard index?

[serycjon]

Thanks, I don't see that immediately in figure 2 (the yellow FPA, green TPA, and red FNA boxes are mutually exclusive), but I will read the paper carefully later.

[yangyi02]

Will close this issue for now.