Performance gap on OTB2015

[visionml]

Performance gap on OTB2015

[martin-danelljan]

Great, thanks. Didn't think that it would make such a difference. We will look in to it and possibly add it.

[goutamgmb]

Are the results that you mention (0.678 and 0.657) from a single run of the tracker, or an average over multiple runs? Due to the stochastic nature of the tracker, the results of the tracker can very slightly over different runs. Hence in the paper, we report an average over 5 runs.

[noUmbrella]

Are the results that you mention (0.678 and 0.657) from a single run of the tracker, or an average over multiple runs? Due to the stochastic nature of the tracker, the results of the tracker can very slightly over different runs. Hence in the paper, we report an average over 5 runs.

Sure, I run 5 times, and average the results.

[goutamgmb]

Ok. I will recheck the models we have trained and the train settings and get back to you.

[noUmbrella]

Ok. I will recheck the models we have trained and the train settings and get back to you.

Thank you for your attention! I find that the max frame interval in source code is 50, whereas in paper is 100. I have not validate where this difference will influence the performance. In addition, I ran in pytorch0.4.1, centos7, python3.7, and TITAN X(pascal).

[noUmbrella]

Ok. I will recheck the models we have trained and the train settings and get back to you.

I have not validate where -> I have not validated whether. Sorry for clerical error.

[goutamgmb]

The released models were trained using max gap 50 as well. We have updated this on arxiv (https://arxiv.org/pdf/1811.07628.pdf).

[noUmbrella]

The released models were trained using max gap 50 as well. We have updated this on arxiv (https://arxiv.org/pdf/1811.07628.pdf).

In addition, as you paper said " We use image flipping and color jittering for data augmentation." But it seems that only ToGrayscale and ToTensorAndJitter augmentation methods are employed during training.

[goutamgmb]

Yes, the image flipping and color jittering are indeed missing from the release code. We hope to integrate them soon. The released models were however also trained without them. So they should not be the reason for the performance gap.

[noUmbrella]

Yes, the image flipping and color jittering are indeed missing from the release code. We hope to integrate them soon. The released models were however also trained without them. So they should not be the reason for the performance gap.

The color jittering may not missing. It seems that ToTensorAndJitter includes the color jittering. Thus, only flipping is missing.

[martin-danelljan]

Its true, only flipping is missing. Regarding the OTB results, in the ATOM paper we calculate AUC for all methods with 100 threshold values, to get a more accurate estimate. In the default setting of using 20 threshold, ATOM has 0.669.

[noUmbrella]

Its true, only flipping is missing. Regarding the OTB results, in the ATOM paper we calculate AUC for all methods with 100 threshold values, to get a more accurate estimate. In the default setting of using 20 threshold, ATOM has 0.669.

It doesn't seem to be a question of evaluation method. I train the model using the released code, then testing the OTB2015 results (denoted by R1). Then, I test the OTB2015 results (denoted by R2) of the released model. Both R1 and R2 are calculated AUC with 100 threshold values, and their means are 0.657 and 0.678, respectively.

[noUmbrella]

Its true, only flipping is missing. Regarding the OTB results, in the ATOM paper we calculate AUC for all methods with 100 threshold values, to get a more accurate estimate. In the default setting of using 20 threshold, ATOM has 0.669.

When flipping is missed, the final train loss is about 0.04, and when flipping is employed, the final train loss is about 0.09. Are these consistent with your training?

[martin-danelljan]

If you do flipping, you must also remember to flip the bounding box annotations.

[noUmbrella]

If you do flipping, you must also remember to flip the bounding box annotations.

Sure, I do this.

[dongfangduoshou123]

I don't know if the closed issue's new comment anthor could receive the notice or not. I have some doubt at #8 (comment) after read the ATOM's track logic.
Expecting answers， best regards!

[martin-danelljan]

@dongfangduoshou123 We get notifications of your posts, even if the issue is closed. So you do not have to post here.

[noUmbrella]

If you do flipping, you must also remember to flip the bounding box annotations.

Now, after adding flip, I get the tracking performance which is the same as the released model, that is 0.676 AUC on OTB2015.

[xi-mao]

Thank you for your excellent work and sharing!
I trained the atom using the source code, and all configurations and employed training sets are same as those in the source code. Then, use the last epoch to test on OTB2015. However, compared the released model whose AUC on OTB2015 is 0.678, I only get 0.657 AUC with the model trained by myself.
Did you write training code yourself?

[wangaixue]

@ @noUmbrella can you tell me how to add filp？ thank you for you work