mindspore-ai/mindarmour

A tool box for MindSpore users to enhance model security and trustworthiness.

mindarmour's Issues

DPOptimizer速度太慢

我使用DPOptimizer微调GPT，速度太慢，我使用了四条数据，batchsize设置为2，使用正常微调20epochs，花费<4min，但是使用DPOptimizer则无法微调，大概过了40分钟，被迫终止了运行。
我使用的包版本如下：

mindarmour                               1.8.0
mindformers                              0.3.0
mindinsight                              1.8.0
mindspore-ascend                         1.8.1
mindx-elastic                            0.0.1
modelarts-mindspore-model-server         1.0.4

因为每次使用DPOptimier都无法得到运行结果，所以没有具体的时间，我使用的代码如下：
因为微调的代码有数据集，不方便复现。我使用下面的代码也遇到了跑不出结果的问题，请问要如何解决？

from mindformers import GPT2LMHeadModel, GPT2Tokenizer
from mindarmour.privacy.diff_privacy import DPOptimizerClassFactory
import mindspore as ms
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.set_train(False)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

GaussianSGD = DPOptimizerClassFactory(micro_batches=2)
GaussianSGD.set_mechanisms('Gaussian', norm_bound=1.0, initial_noise_multiplier=1.5)
opt = GaussianSGD.create('Momentum')(params=model.trainable_params(),
                                         learning_rate=0.001,
                                         momentum=0.9)
# opt = ms.nn.Adam(model.trainable_params())
grad_fn = ms.ops.value_and_grad(model, None, opt.parameters, has_aux=False)

model.set_train(True)
inputs = tokenizer(["hello world"],
                   padding='max_length',
                   max_length=model.config.seq_length+1,
                   return_tensors='ms')
# output = model(input_ids=inputs["input_ids"])
# print(output)  # 计算loss
loss, grad = grad_fn(inputs['input_ids'])
res = opt(grad)
print(loss)
print(res)

Operator[SoftmaxCrossEntropyWithLogits] is not support.

Environment

Hardware Environment(Ascend/GPU/CPU):

Uncomment only one /device <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/device cpu

Software Environment:

• MindSpore version (source or binary):binary
• Python version (e.g., Python 3.7.5):python3.7.5
• OS platform and distribution (e.g., Linux Ubuntu 16.04):ubuntu 18.04
• GCC/Compiler version (if compiled from source):

Describe the current behavior

mnist_attack_fgsm crash

Describe the expected behavior

Steps to reproduce the issue

1. python3.7 mnist_attack_fgsm.py

Related log / screenshot

mnist_attack_fgsm.py:87: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead
start_time = time.clock()
[ERROR] ME(3438,python):2020-05-07-23:11:49.929.959 [mindspore/ccsrc/device/cpu/cpu_session.cc:115] BuildKernel] Operator[SoftmaxCrossEntropyWithLogits] is not support.
Traceback (most recent call last):
File "mnist_attack_fgsm.py", line 119, in
test_fast_gradient_sign_method()
File "mnist_attack_fgsm.py", line 89, in test_fast_gradient_sign_method
np.concatenate(test_labels), batch_size=32)
File "/usr/local/lib/python3.7/dist-packages/mindarmour-0.2.0-py3.7.egg/mindarmour/attacks/attack.py", line 65, in batch_generate
adv_x = self.generate(x_batch, y_batch)
File "/usr/local/lib/python3.7/dist-packages/mindarmour-0.2.0-py3.7.egg/mindarmour/attacks/gradient_method.py", line 96, in generate
gradient = self._gradient(inputs, labels)
File "/usr/local/lib/python3.7/dist-packages/mindarmour-0.2.0-py3.7.egg/mindarmour/attacks/gradient_method.py", line 290, in _gradient
out_grad = self._grad_all(Tensor(inputs), Tensor(labels), sens)
File "/usr/local/lib/python3.7/dist-packages/mindspore/nn/cell.py", line 147, in call
out = self.compile_and_run(*inputs)
File "/usr/local/lib/python3.7/dist-packages/mindspore/nn/cell.py", line 301, in compile_and_run
_, compile_flag = _executor.compile(self, *inputs, phase=self.phase)
File "/usr/local/lib/python3.7/dist-packages/mindspore/common/api.py", line 363, in compile
result = self._executor.compile(obj, args_list, phase, use_vm)
RuntimeError: mindspore/ccsrc/device/cpu/cpu_session.cc:115 BuildKernel] Operator[SoftmaxCrossEntropyWithLogits] is not support.

Special notes for this issue