About #params of Adapter

[JieShibo]

About #params of Adapter

[ZhangYuanhan-AI]

Thanks for your reminder, and sorry for this confusion.
We revise this in our newest arXiv version.
Specifically, the Adapter is 0.16M, LoRA is 0.29M, VPT is 0.64M, and NOAH is 0.43M.
For a fair comparison, we will add the VTAB performance of the 4X8 dim Adapter in the newest version. Please stay tuned.

[JieShibo]

Thanks a lot for your reply.
BTW it seems that the lib folder has not been uploaded yet, which causes an error when running the scripts.

[ZhangYuanhan-AI]

Thanks a lot for your reply. BTW it seems that the lib folder has not been uploaded yet, which causes an error when running the scripts.

Yes, Shibo.

Actually, we are cleaning our code in the lib folder these days. We will upload this folder this week, please stay tuned.

[JieShibo]

Hi, thanks again for sharing the code.
I successfully reproduced most of the results about VTAB-1K in the paper except for Retinopathy.
I ran the following commands

DATASET=diabetic_retinopathy
#adapter
python supernet_train_prompt.py --data-path=../vtab-1k/${DATASET} --data-set=${DATASET} --cfg=./experiments/Adapter/ViT-B_prompt_adapter_8.yaml --resume=../ViT-B_16.npz --output_dir=./saves/${DATASET}_lr-0.001_wd-0.0001_adapter --batch-size=64 --lr=0.001 --epochs=100 --is_adapter --weight-decay=0.0001 --no_aug --mixup=0 --cutmix=0 --direct_resize --smoothing=0 --launcher="none"
#lora
python supernet_train_prompt.py --data-path=../vtab-1k/${DATASET} --data-set=${DATASET} --cfg=./experiments/LoRA/ViT-B_prompt_lora_8.yaml --resume=../ViT-B_16.npz --output_dir=./saves/${DATASET}_lr-0.001_wd-0.0001_lora --batch-size=64 --lr=0.001 --epochs=100 --is_LoRA --weight-decay=0.0001 --no_aug --mixup=0 --cutmix=0 --direct_resize --smoothing=0 --launcher="none"
#noah
python supernet_train_prompt.py --data-path=../vtab-1k/${DATASET} --data-set=${DATASET} --cfg=experiments/NOAH/subnet/VTAB/ViT-B_prompt_${DATASET}.yaml --resume=../ViT-B_16.npz --output_dir=saves/${DATASET}_supernet_lr-0.0005_wd-0.0001/retrain_0.001_wd-0.0001  --batch-size=64 --mode=retrain --epochs=100 --lr=0.001 --weight-decay=0.0001 --no_aug --direct_resize --mixup=0 --cutmix=0 --smoothing=0 --launcher="none"

and got
{"train_lr": 1.0976769428005575e-05, "train_loss": 0.053501952129105725, "test_loss": 1.5837794360286461, "test_acc1": 71.1038200165646, "test_acc5": 100.0, "epoch": 99, "n_parameters": 160613}
{"train_lr": 1.0976769428005575e-05, "train_loss": 0.02753125037997961, "test_loss": 2.0185347567061465, "test_acc1": 67.27443168466701, "test_acc5": 100.0, "epoch": 99, "n_parameters": 298757}
{"train_lr": 1.0976769428005575e-05, "train_loss": 0.042937366478145125, "test_loss": 1.6862158768191309, "test_acc1": 69.75392547600269, "test_acc5": 100.0, "epoch": 99, "n_parameters": 8462261}

Did I do something wrong?

[ZhangYuanhan-AI]

Hi Shibo,

I want to know if the shown test_acc1 is the best accuracy OR the accuracy of the final checkpoint?

[JieShibo]

The final checkpoint

[ZhangYuanhan-AI]

Ok, we report the best accuracy.

[JieShibo]

Thank you.