CONFIG = $1, where to find config?

[zhangyuanhan-ai]

CONFIG = $1, where to find config?

[ZhangYuanhan-AI]

Thanks for your interest.

configs are located here: https://github.com/ZhangYuanhan-AI/NOAH/tree/main/experiments/VPT

[zhaoedf]

https://github.com/ZhangYuanhan-AI/NOAH/tree/main/experiments/VPT

thx for the quick reply, i will look into it

[ZhangYuanhan-AI]

Enjoy😆

[zhaoedf]

Enjoy😆

i have a question, it seems that VPT has different optimal hyperparameters for different datasets, but it seems that the folder provided does not have anything specific to a dataset?

i was wondering if your project could reproduce the results in VPT or simply it just could reproduce the results in NOAH?

thx again.

[ZhangYuanhan-AI]

We can only achieve on-par performance with VPT and reproduce the results in NOAH.

[zhaoedf]

We can only achieve on-par performance with VPT and reproduce the results in NOAH.

i understood, so if i want to achieve on-par perf with VPT, i should follow the hyperparams in SUPERNET only? since --mode default value is super, the SEARCH_SPACE and RETRAIN hyperparams won't have any effect, right?

[ZhangYuanhan-AI]

exactly

[zhaoedf]

exactly

is it possible to run your code w/o slurm?

[zhaoedf]

Enjoy😆

i have a question, it seems that VPT has different optimal hyperparameters for different datasets, but it seems that the folder provided does not have anything specific to a dataset?

i was wondering if your project could reproduce the results in VPT or simply it just could reproduce the results in NOAH?

thx again.

by the way, your config is not dataset specific? is this not necessary?

[ZhangYuanhan-AI]

#5

[zhaoedf]

#5

thx, i will have a try.

[ZhangYuanhan-AI]

• Since currently, VPT does not report its optimal learning rate and weight decay and only reports its optimal prompt length in Table. 13, we do use the optimal prompt length to reproduce performance in VTAB and achieve the on-par performance though still report VPT's results in this table (a little bit better than our reproduction).
• To exactly reproduce VPT's results, you could do an exhaustive hyperparameter search(Table 7. in VPT). However, the exhaustive hyperparameter search requires a huge computing resource (240 experiments for a single dataset). Therefore, we cannot fully follow their way to do the all experiments, and we hope you can understand. If you want to do so, just modify the parameter in the config file.
• Noted that all the results in NOAH are based on the same experimental setting---same learning rate, and weight decay---for a fair comparison. Therefore, though we do not do the exhaustive hyperparameter search, we still think our result is reasonable and convinced, what do you think?
  @zhaoedf

[zhaoedf]

thx a lot!

i recently conduct a grid search as VPT proposed and you are right that it is really time-consuming.

Thus, when i learnt that you achieve on-par performance with VPT, i thought you might have conducted a grid search or sth to find the optimal hyperparams.

since you have not, maybe i will continue my grid search as well as using your code to get results.

would you mind if i post my grid search results here in this issue? cos it seems you are one of the few repos that have VPT reproduced and our discussion might help others.

[ZhangYuanhan-AI]

thx a lot!

i recently conduct a grid search as VPT proposed and you are right that it is really time-consuming.

Thus, when i learnt that you achieve on-par performance with VPT, i thought you might have conducted a grid search or sth to find the optimal hyperparams.

since you have not, maybe i will continue my grid search as well as using your code to get results.

would you mind if i post my grid search results here in this issue? cos it seems you are one of the few repos that have VPT reproduced and our discussion might help others.

Sure! maybe it is better if we can talk about the grid search result through e-mail at first, and post the final result after we double-check the results. Otherwise, this post might be too long for others to catch the most useful information. @zhaoedf This is my e-mail: [email protected] feel free to chat.

[zhaoedf]

ok!

[zhaoedf]

does get_vtab1k.py simply generate .txt files under tensorflow env and after this, it is easy to use your lib.dataset to get dataset with the txt files and original images under torch env?

would you mind if you send me these txt files, cos i am in mainland china, accessing google api is not that convenient. thx.