Attacks are not run on defenses in an all-pairs manner

[ryderling]

Attacks are not run on defenses in an all-pairs manner

[ryderling]

As we stated at the beginning of Section II, "In this paper, we consider the non-adaptive and white-box attack scenarios, where the adversary has full knowledge of the target DL model but is not aware of defenses that might be deployed." Readers or potential users of this paper and codes should be aware of this basic prerequisite at first.

It seems you suggest that the only way evaluating the defense method is to apply a totally adaptive white-box attack, in which the attacker needs to try-his/her-best to find different "best" adaptive attack strategies instead of the default and sample strategy for different defense strategies. In most cases, this adaptive attack relies heavily on the knowledge and skill of attackers, when considering different defenses (totally different defense methods or one defense with different kinds of hyper-parameters), and none of us can guarantee or prove that the adaptive attack strategy is the "best". It may be another research direction in adversarial examples.

On the other hand, if users want to perform the adaptive attack on defenses, the DEEPSEC code can support to run the attacks on most of defense-enhanced models by replacing the raw model file with the defense-enhanced model file.

[carlini]

Okay, so let's put aside the question of what it means to do a security evaluation. I think we have fundamental disagreements there that aren't going to be resolved over a github issue. The security community has (since it's inception) decided that adaptive attacks are what are necessary to judge robustness, and if you want to do something different then that's fine I guess.

My main point is that the paper consistently presents itself as a thorough it actually runs white-box attacks on defenses and finds these defenses "more or less effective".

And it's great that you admit what you're doing is a transferability analysis in Section 2, but I would expect this to be stated clearly in the abstract, introduction, and conclusion, and then every time you make any sweeping generalization.

For example, you may want to change the following:

"Leveraging DEEPSEC, we systematically evaluate the existing adversarial attack and defense methods, and draw a set of key findings" -> "Leveraging DEEPSEC, we systematically evaluate the existing adversarial attack and defense methods in the black-box, zero-knowledge, transfer-only threat model, and draw a set of key findings"

"For complete defenses, most of them have capability of defending against some adversarial attacks" -> "For complete defenses, most of them have capability of defending against transfer-only attacks."

"All detection methods show comparable discriminative ability against existing attacks." -> "All detection methods show comparable discriminative ability against existing transferable adversarial examples."

The conference hasn't even happened yet, and already at NDSS one of the speakers used a quote from the DeepSec paper to argue that existing defenses are effective in the white-box setting.