DeiT depth 24 (CaiT - TABLE 1)

[GoJunHyeong]

DeiT depth 24 (CaiT - TABLE 1)

[TouvronHugo]

Hi @GoJunHyeong ,
Thanks for your message.
Could you provide the hparams you are using for this run?
Best,
Hugo

[GoJunHyeong]

Hello, @TouvronHugo, Thank you for your reply.

I used the official training script of DeiT-S.
https://github.com/facebookresearch/deit/blob/main/README_deit.md

To summarize I use:

Batch size: 1024 (8 GPUs)
stochastic depth ratio : 0.2
dropout ratio : 0
lr: 5e-4
epochs: 300
weight-decay: 0.05
sched: cosine
input size: 224
smoothing: 0.1
repeated_aug: True
mixup: 0.8
cutmix: 1.0
warmup epochs:
rand aug: rand-m9-mstd0.5-inc1
erase prob: 0.25
Optimizer: AdamW

There is no layer scale or scalar alpha weighting scheme.

The accuracy mentioned in the paper is 81.0, but I achieved 79.7 in my case.

Thank you.