Questions about reproduction.

[MouxiaoHuang]

Questions about reproduction.

[ZhangYuanhan-AI]

Hi Mouxiao

Could you please kindly provide me with the training log of your experiments?

[MouxiaoHuang]

Thank you for getting back to me promptly.

Certainly. As an illustration, I have provided the training logs for Caltech101, which involve VPT, Adapter, LoRA, Supernet, and the corresponding searching log for Supernet, via the following link: link.

[ZhangYuanhan-AI]

Regarding VPT / Adapter / LoRA:

1. I conducted evaluations for each configuration using three distinct random seeds, subsequently reporting the mean performance. It is recommended to replicate this process. In the event of persisting issues, we can further analysis.
2. I get this from the configuration of VPT (visual prompt tuning) paper.

Regarding NOAH:

1. Indeed, one potential limitation of NOAH lies in its efficiency during the search phase, particularly when faced with small "LIMITS" values. Exploring ways to enhance efficiency under these conditions may prove to be a fruitful direction for improving NOAH. Notably, some solutions have been proposed within the context of Neural Architecture Search (NAS). It is imperative to verify the accurate loading of the supernet.
2. Sorry, I don't fully understand this question. I get this configuration from the searching results.

Regarding others:

1. Yes, it is the prefix-tuning method. I found this method perform similarly to VPT, so I do not report this performance.

[MouxiaoHuang]

Thank you for your response, which helped me clarify most of the questions I had.

In addition, I am still confused about the results of evolutionary search experiments. Would it be possible for you to provide a sample search log for any one of the vtab-1k sub-datasets?

[ZhangYuanhan-AI]

Sure.
Check this:https://drive.google.com/file/d/1lW980rK9QIjOidCzdzzs_3s6cPevh1Ey/view?usp=sharing
The log for this configure:https://github.com/ZhangYuanhan-AI/NOAH/blob/main/experiments/NOAH/subnet/few-shot/ViT-B_prompt_oxford_pets_shot16-seed0.yaml

[MouxiaoHuang]

Oh, I figured out why my accuracy results from the search algorithm were confusing and nearly random. It turns out that the issue was caused by the difference in the timm.models.load_checkpoint function between version 0.4 and versions 0.5 or higher.

Thanks again for your prompt and helpful response.