The performance of l2_attack on Pytorch

[carlini]

The performance of l2_attack on Pytorch

[carlini]

There are basically four things that the code does:

(1) Use a hinge loss as defined in the paper. Make sure you do this PRE-SOFTMAX, otherwise things will not be correct.
(2) Use a better optimizer than standard SGD. Exactly which one shouldn't be important.
(3) Choose the lagrangian multiplier through binary search.
(4) Run for many iterations.

If you're testing on MNIST, then you should be able to set the lagrangian multiplier to 10.0 or so as a baseline. If you use a learning rate of 10^-1, then only 100 iterations should give approximately reasonable numbers. Those settings should give reasonable results if everything else is correct.

[zjysteven]

Thanks! I'm now testing on CIFAR-10, and I set learning rate: 0.1, iterations: 100, binary search step: 1. I guess I will try MNIST first and see what's going on.

[carlini]

Those settings should be reasonable, too. I would recommend verifying you are doing the pre-softmax correctly. If you do things post-softmax, results are very bad.

[zjysteven]

Thanks for your suggestions. I finally solved this issue by letting the learning rate decay to its half every 20 iterations (max iterations: 100) and then all adversarial samples could be found. So I guess the issue is mainly caused by the Adam optimizer of Pytorch, since with Tensorflow one could get desirable results without decaying the learning rate of Adam.

[carlini]

Thanks for the update. That's interesting. Adam is supposed to set the learning rate adaptively, so not sure why the pytorch one isn't doing it.