Performance of Meta-DETR without meta-finetuning?

[stwerner97]

Performance of Meta-DETR without meta-finetuning?

[nanfangAlan]

I've evaluated the provided model after base training on novel sets. It seems are overfitting on base sets.

COCO:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.006
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.011
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.005
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.008
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.009
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.040
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.085
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.093
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.051
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.085
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.166

VOC1:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.003
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.005
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.003
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.003
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.004
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.088
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.188
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.216
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.085
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.161
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.259

[ZhangGongjie]

Yes. Meta-DETR requires fine-tuning to work on novel classes.

[nanfangAlan]

Which reason do you think may limit the Meta-DETR to work on novel sets w/o fine-tuning, and how to solve this? As a class agnostic predictor, I thought Meta-DETR would get better results w/o fine-tuning.

Yes. Meta-DETR requires fine-tuning to work on novel classes.

[stwerner97]

@nanfangAlan I think this is a phenomenon that can be observed for most meta learners. On the upside, there are some works that target this settings specifically. Maybe FS-DETR could be an interesting paper to look at for you 😊

[nanfangAlan]

@stwerner97 Thanks for your reply. I've notice the FS-DETR, but it's improvement of results seems mainly because the use of pretrained UP-DETR, which can be found in Tab 4 of their paper, it cannot get comparable results without pretraining. I think it's a little unfair, cause other FSODs do not load UP-DETR params. Do you have any idea or know about other works which solve this problem? Thanks again.

[stwerner97]

@nanfangAlan I agree that most of the improvements are due to the pre-training stage and would also apply to related works in some capacity.

I still think that its a very worthwhile read, since there are too few meta-learning works reporting their performance without fine-tuning and pre-training is a valid method to improve overall performance. I also don't see what other components to change to close the fine-tuning vs. non-fine-tuning gap, since improving the architectural design most likely will benefit fine-tuning performance as well.

Unfortunately, I am not and would also be highly interested in such works. Of course there are a lot of works on zero-shot detection using vision-language models, but that's not what I am looking for. 😊

[stwerner97]

Ah, I just remembered AirDet: Few-Shot Detection without Fine-Tuning for Autonomous Exploration, which also claims being a few-shot object detector without fine-tuning. 😊