Train and evaluate vanilla SimCLR

[alinlab]

Train and evaluate vanilla SimCLR

[jihoontack]

Hi, thank you for your interest in our work!

I think you should run the SimCLR code with --shift_trans_type none and also test with the same option (since as you already mentioned, SimCLR does not require shifting transformation classification layer).

If you don't want to train the model again, maybe forcing the P.K_Shift=1 (after get_shift_module function) might resolve the problem (not sure about this...).

And by the way, I recommend lars optimizer since I had never tried with adam and I am not sure about the performance. When I used sgd rather than lars optimizer, the overall performance had dropped around 1~2%. Also, If you are training SimCLR, you should set the batch size to 128 (then it will make 4 * 128 in total, for CSI it was 32 since we multiply with the number of rotation transformation).

Thank you again for your interest.

[hyun78]

Hi, I run the following code.
for training:
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 train.py --dataset cifar10 --model resnet18 --mode simclr --shift_trans_type none --batch_size 128 --one_class_idx 0 --optimizer adam --suffix new
for evaluation:
python3 eval.py --mode ood_pre --dataset cifar10 --model resnet18 --ood_score simclr --shift_trans_type none --print_score --ood_samples 10 --resize_factor 0.54 --resize_fix --one_class_idx 0 --load_path <load_path>

And I got a different result.

I think it should be 87.9 (in the paper, Table 7a). It may be lower due to the optimizer, but it is too low.

1. Is there any problem with my command?
2. Could you provide sample codes to reproduce the Table 7a result?

Thanks

[hyun78]

Thanks a lot!
I will try --shift_trans_type none with batch size 128
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 train.py --dataset cifar10 --model resnet18 --mode simclr --shift_trans_type none --batch_size 128 --one_class_idx 0 --optimizer adam --suffix new

[jihoontack]

If you have any problems, feel free to reopen the issue.
Thank you.