DDAIG: different hyperparameters for different domains violates the DG settings

[BurningFr]

DDAIG: different hyperparameters for different domains violates the DG settings

[BurningFr]

Thank you for your reply.

I think we need keep the hyper-parameters the same among different domains, because the compared performance of DG methods is the average accuracy on 4 leave-one-domain-out experiments. If we tune hyper-parameters per domain, the universality of proposed method is limited.

I agree with that sometimes we have to tune hyper-parameters by the test data, but use different hyper-parameters on different domains is a 'less' general way to deal with DG problem.

[KaiyangZhou]

hi, thanks for your interest in our work

it's a very good question—indeed, it's still a difficult problem for param tuning in DG

in the existing work, people either 1) tune params on the test data to show the best they can achieve for their methods (otherwise how do you adjust the model design in prototyping?), or 2) use cross-val where the val images come from the held-out set of source domains

the 1st way is not wrong as you are not touching the test data at all during training. in practice you could prepare several models trained with different hyper-params, collect a small test dataset and deploy the model that has the best performance on it—the performance really matters for any applications so why not? for comparison, it's not uncommon to pick your best-performing model (I agree it sounds weird for DG)

the 2nd way doesn't make too much sense though I used it in another work. say you pick the best-performing model in the val data of source domains, it could well be the most overfitting one. therefore, the tradtional cross-val doesn't perfectly fit DG

this topic of param tuning could be a good direction to explore for future work if you'd like to :)

[KaiyangZhou]

Came across this paper which discusses model selection for DG https://openreview.net/forum?id=lQdXeXDoWtI