Comments (3)
Thank you for your reply.
I think we need keep the hyper-parameters the same among different domains, because the compared performance of DG methods is the average accuracy on 4 leave-one-domain-out experiments. If we tune hyper-parameters per domain, the universality of proposed method is limited.
I agree with that sometimes we have to tune hyper-parameters by the test data, but use different hyper-parameters on different domains is a 'less' general way to deal with DG problem.
from dassl.pytorch.
hi, thanks for your interest in our work
it's a very good question—indeed, it's still a difficult problem for param tuning in DG
in the existing work, people either 1) tune params on the test data to show the best they can achieve for their methods (otherwise how do you adjust the model design in prototyping?), or 2) use cross-val where the val images come from the held-out set of source domains
the 1st way is not wrong as you are not touching the test data at all during training. in practice you could prepare several models trained with different hyper-params, collect a small test dataset and deploy the model that has the best performance on it—the performance really matters for any applications so why not? for comparison, it's not uncommon to pick your best-performing model (I agree it sounds weird for DG)
the 2nd way doesn't make too much sense though I used it in another work. say you pick the best-performing model in the val data of source domains, it could well be the most overfitting one. therefore, the tradtional cross-val doesn't perfectly fit DG
this topic of param tuning could be a good direction to explore for future work if you'd like to :)
from dassl.pytorch.
Came across this paper which discusses model selection for DG https://openreview.net/forum?id=lQdXeXDoWtI
from dassl.pytorch.
Related Issues (20)
- How to download "Synthetic Numbers" of Digits-DG?
- Run SimCLR using Dassl HOT 1
- Feature Request: Implementation of SSDA like data loaders HOT 2
- Replication Parameters and How to use MixStyle and EFDM HOT 1
- Issue with FixMatch training on customs datasets HOT 4
- warnings.warn('No file found at "{}"'.format(fpath)) HOT 1
- use adabn.py HOT 1
- Gradient Accumulation Support HOT 2
- How to register the name into "TRAINER" registry'? I think I did everything correctly HOT 2
- TypeError: unsupported format string passed to NoneType.__format__ HOT 2
- dg dataset in vlcs
- RuntimeError Occur When Using DDAIG Model
- ImportError: dlopen: cannot load any more object with static TLS
- how to train from a checkpoint?
- Can I use the Dassl in Horovod frame?
- Set different learning rate
- How can I change the vision encoder and text encoder to custom pretrained clip vision encoder and text encoder?
- add CUB-200-2011 to CoOp, display error causes an error to run HOT 1
- 重写evaluator的Classification类
- Send nn.parameter to optimizer
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dassl.pytorch.