Problems with MAP

[amenra]

Problems with MAP

[AmenRa]

Hi Stefano,

Almost all the metrics ignore qrels with zero scores, including NDCG.
So the difference you get is not because of that.

However, I think your results are entirely possible if you have many relevance judgments for each query.
Note that the Average Precision denominator is equal to the number of relevant documents, regardless of the cut-off.
Conversely, DCG and, therefore, NDCG depends on the cut-off and not on the number of relevant documents.
(Please, cross check this for correctness.)

Here is a toy example:

from ranx import Qrels, Run, evaluate

# 100 relevant docs
qrels = Qrels({
    "q1": {f"d{i}": 1 for i in range(100)}
})

# Only one relevant doc is returned
run = Run({
    "q1": {**{"d1":1000}, **{f"dd{i}": i for i in range(99)}}
})

>>>
{
    "map": 0.01,
    "map@10": 0.01,
    "ndcg@10": 0.22009176629808017,
    "ndcg@100": 0.047758523260819974,
}

From my experience, MAP is commonly used with larger cut-offs than 10 (usually 100) or with no cut-off at all.

Best,

Elias

[AmenRa]

Closing for inactivity.
Feel free to re-open if needed.