zimmerrol/foolbox-native

Foolbox Native brings native performance to Foolbox

Foolbox Native

WARNING: The latest Foolbox Native release on PyPI (0.8.0) requires EagerPy 0.15.0 (newer versions will not work). We are currently making major changes to Foolbox Native. With the next version, this will be fixed.

Foolbox Native is an extension for Foolbox that tries to bring native performance to Foolbox. This extension is a prototype with the goal of ultimately becoming part of Foolbox itself. Please be aware of the the differences to Foolbox listed below.

Foolbox Native currently provides full support for:

• PyTorch
• TensorFlow
• JAX

Other frameworks can be used as well by falling back to Foolbox for the model API, see below.

Installation

pip install --upgrade foolbox-native

PyTorch Example

import foolbox.ext.native as fbn
import torchvision.models as models

# instantiate a model
model = models.resnet18(pretrained=True).eval()
preprocessing = dict(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], axis=-3)
fmodel = fbn.models.PyTorchModel(model, bounds=(0, 1), preprocessing=preprocessing)

# get data and test the model
images, labels = fbn.utils.samples(fmodel, dataset='imagenet', batchsize=16)
print(fbn.utils.accuracy(fmodel, images, labels))
# -> 0.9375

# apply the attack
attack = fbn.attacks.LinfinityBasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=0.03, step_size=0.005)  # L-inf norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))
# -> 0.0

# apply another attack
attack = fbn.attacks.L2BasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=2.0, step_size=0.2)  # L2 norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))
# -> 0.0

TensorFlow Example

import foolbox.ext.native as fbn
import tensorflow as tf

# instantiate a model
model = tf.keras.applications.ResNet50(weights='imagenet')
preprocessing = dict(flip_axis=-1, mean=[104., 116., 123.])  # RGB to BGR
fmodel = fbn.models.TensorFlowModel(model, bounds=(0, 255), preprocessing=preprocessing)

# get data and test the model
images, labels = fbn.utils.samples(fmodel, dataset='imagenet', batchsize=16)
print(fbn.utils.accuracy(fmodel, images, labels))

# apply the attack
attack = fbn.attacks.LinfinityBasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=0.03 * 255., step_size=0.005 * 255.)  # L-inf norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))

# apply another attack
attack = fbn.attacks.L2BasicIterativeAttack(fmodel)
adversarials = attack(images, labels, epsilon=2.0 * 255., step_size=0.2 * 255.)  # L2 norm
print(fbn.utils.accuracy(fmodel, adversarials, labels))

Robust Accuracy Evaluation

import foolbox.ext.native as fbn

# get fmodel, images, labels like above
fmodel = ...
images, labels = ...

attacks = [
    L2BasicIterativeAttack,
    L2CarliniWagnerAttack,
    L2ContrastReductionAttack,
    BinarySearchContrastReductionAttack,
    LinearSearchContrastReductionAttack,
]
epsilons = [0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0, 64.0, 128.0]

_, robust_accuracy = fbn.evaluate_l2(fmodel, x, y, attacks=attacks, epsilons=epsilons)
print(robust_accuracy)

# Plot an accuracy-distortion curve
plt.plot(epsilons, robust_accuracy)

Other Frameworks

Foolbox Native supports all frameworks supported by the standard Foolbox by simply wrapping the foolbox.models.* classes using fbn.model.FoolboxModel. This, however, comes with a performance penalty. Nevertheless, it still allows one to profit from the manually batched attack reimplementations that come with Foolbox Native.

Important differences to Foolbox

Unlike Foolbox:

• Foolbox Native does not yet guarantee API stability (expect breaking changes)
• Foolbox Native is currently limited to very few attacks
• Foolbox Native does not make any guarantees about the output of an attack
  • The user is responsible for checking if the returned samples are adversarial
  • Whether the size of the perturbations is guaranteed depends on the attack
  • Foolbox, on the other hand, searches for the smallest perturbation while guaranteeing that the returned samples are adversarial

Development

For development, it is recommended to do a an editable installation of Foolbox and Foolbox native using pip install -e . in the corresponding folders (after cloning the two repositories). Unfortunately, pip has a bug with editable installs and namespace packages like Foolbox Native. A simple workaround is to add a symlink to the foolbox/ext/native folder of Foolbox Native in the foolbox/ext/ folder of Foolbox itself.