zakajd/huawei2020

Solution of Huawei Digix Global AI Challenge

huawei2020's Issues

Potential bug in the MAP@R metric

Hello!

I tried to adapt MAP@R metric for my task and found out that your implementation has a bug. I just took examples from the original paper and put into your function:

import torch
import numpy as np


conformity_matrix = torch.tensor([[True for _ in range(10)] + [False for _ in range(10)]])

permutation_matrix = torch.tensor([[0, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 1, 2, 3, 4, 5, 6, 7, 8, 9]])
np.abs(map_at_k(permutation_matrix, conformity_matrix, topk=None) - 0.1) <= 1e-6
# False

permutation_matrix = torch.tensor([[0, 10, 11, 12, 13, 14, 15, 16, 17, 1, 18, 19, 2, 3, 4, 5, 6, 7, 8, 9]])
np.abs(map_at_k(permutation_matrix, conformity_matrix, topk=None) - 0.12) <= 1e-6
# False

permutation_matrix = torch.tensor([[0, 1, 11, 12, 13, 14, 15, 16, 17, 18, 19, 10, 2, 3, 4, 5, 6, 7, 8, 9]])
np.abs(map_at_k(permutation_matrix, conformity_matrix, topk=None) - 0.2) <= 1e-6
# False

permutation_matrix = torch.arange(20).reshape(1, 20)
np.abs(map_at_k(permutation_matrix, conformity_matrix, topk=None) - 1.0) <= 1e-6
# False

All tests pass correctly if we change line https://github.com/zakajd/huawei2020/blob/master/src/callbacks.py#L91 by:

average_precision = precision.sum(dim=-1) / R

Paper review

1. 1st Place Solution to Google Landmark Retrieval 2020

• Used GAP after extractor model
• Reduced feature dimension to 512
• Applied cosine softmax to classify a number of classes, scale value was automatically determined by fixed adacos. Margin value was set to 0
• weighted CE to deal with imbalanced classes,
• Progressive increase in image sizes reliably boosted scores

Note: In Google Landmarks task was only to generate good features. No post-processing was allowed, so it's not covered here.

2. 2nd Place Solution to Google Landmark Retrieval Competition 2020

• Used concatenation of features from 2 models as an ensembling technique
• arcmargin loss
• Progressive resize

3. 3rd Place Solution to “Google Landmark Retrieval 2020”

• Post- processing methods: DBA [1] , QE [12] and re-rank.
• Used Corner-Cutmix. Reasoning:
  1. Important features are often in the centre, so don't want to cover it.
  2. Networks learns to look at image in different scales, which is useful for real-life scenarios

4. Triplet loss
   Often mentioned in earlier papers.
   Takes a triplet (baseline (anchor) input, positive (truthy) input and a negative (falsy) input). Tries to make embeddings for simmilar object close to each other and for not-simmilar further. Distance is Euclidian.

Results

Started training baseline models:

1. ResNet50 with ArcFace and CosFace losses
2. ResNet34 with ArcFace

Results aren't good, loss decreasing very slow. For now only loss is tracked, metrics not yet measured.

There may be a bug in the code, so I'll try to just learn a classification task using CrossEntropy Loss first and see if it's possible to learn anything.
[UPDATE WITH RESULTS]