ziangyan/subspace-attack.pytorch

Implementation of our NeurIPS 2019 paper: Subspace Attack: Exploiting Promising Subspaces for Query-Efficient Black-box Attacks

subspace-attack.pytorch's Issues

Where does the statistics data of success_query_all come from?

The success_query_all is initialized as success_query_all = torch.zeros_like(query_all) in
https://github.com/ZiangYan/subspace-attack.pytorch/blob/master/attack.py#L264

But after initialization, it never changes, in the final statistics result report as shown in :
https://github.com/ZiangYan/subspace-attack.pytorch/blob/master/attack.py#L514

How does success_query_all change to the real statistics result?

Can you provide config for imagenet under L2 distance constrain?

Do you use data-prior like Bandit-TD does?

The bug report of calculating attack failure rate

I found a bug of computing the attack failure rate , because in
https://github.com/ZiangYan/subspace-attack.pytorch/blob/master/attack.py#L512
you directly use not_done_all.mean().item() as the failure rate.
However, due to bandits attack's code:
https://github.com/MadryLab/blackbox-bandits/blob/master/src/main.py#L178

You miss the case that some benign images are already misclassified by the classifier before attacking. The attack success rate or failure rate should use the correctly classified images to calculate. You must remove/filter these misclassified clean images in the computation. Thus, your paper's failure rate is not correct strictly speaking, unless all 1000 images are already correctly classified by the victim model.

Also, the code of https://github.com/thu-ml/Prior-Guided-RGF/blob/master/attack.py#L372
supports my suggestion.