Test with validation data？

[taoyang1122]

Test with validation data？

[taoyang1122]

Yes, but it is not unreasonable in terms of comparing the results with other methods.

1. On ImageNet, most current works did the hyperparameter-tuning on validation dataset to choose the best hyperparameters, and some reported the highest accuracy in the whole training process rather than the last epoch. Our process of testing the model on validation dataset to choose the best configuration is also hyperparameter-tuning on validation dataset. Note that in real deployment you directly choose the best configurations from the query table.
2. We also did the experiments to hold out a sub-set of training dataset to choose the best configuration. The generated query table is basically the same, with only 2 or 3 points having different configs. Even for those different configs, the accuracy difference is very small. So the overall accuracy curve is basically the same.

[AlbertHuyb]

I see, thanks for your detailed reply.

Then I'll close the issue.

Thanks again.