zaynmi/seada-vqa

A pytorch implemetation of data augmentation method for visual question answering

seada-vqa's Issues

Cannot find requirements.txt

Hi,

Thank you for sharing the code on seada paper. I was trying to use this repo to reproduce your results, but got stuck when setting up the environment. There is no requirements.txt found in the repo, could you kindly include it?

Thanks.

TypeError: make_src() takes 2 positional arguments but 3 were given

paraphrases = self.ps.generate_paraphrases(instance_for_onmt, topk=topk+1, edit_distance_cutoff=4, threshold=threshold)
File "/home/scan/DVSI/sscc/paraphrase_scorer.py", line 799, in generate_paraphrases
translation, mapping = choose_forward_translation(sentence, to, back, n=5)
File "/home/scan/DVSI/sscc/paraphrase_scorer.py", line 30, in choose_forward_translation
translations = to_translator.translate([sentence], n_best=n, return_from_mapping=True)[0]
File "/home/scan/DVSI/sscc/onmt_model.py", line 179, in translate
for batch in testData:
File "/usr/local/lib/python3.6/site-packages/torchtext/data/iterator.py", line 151, in iter
self.train)
File "/usr/local/lib/python3.6/site-packages/torchtext/data/batch.py", line 27, in init
setattr(self, name, field.process(batch, device=device, train=train))
File "/usr/local/lib/python3.6/site-packages/torchtext/data/field.py", line 188, in process
tensor = self.numericalize(padded, device=device, train=train)
File "/usr/local/lib/python3.6/site-packages/torchtext/data/field.py", line 308, in numericalize
arr = self.postprocessing(arr, None, train)
TypeError: make_src() takes 2 positional arguments but 3 were given

Do you know how to solve this problem？？？

Cannot change topk value

Hi again,

I have found a new problem. The code breaks if I change the topk parameter. For example, the following works well:

python main.py --attack_only --attack_mode q --attack_al sea --fliprate 0 --topk 1 --paraphrase_data val --attacked_checkpoint data/bs256.pth

But if I use 3 instead of 1 in topk:

python main.py --attack_only --attack_mode q --attack_al sea --fliprate 0 --topk 3 --paraphrase_data val --attacked_checkpoint data/bs256.pth

It produces the following error:

--------------------------------------------------
Namespace(adv_delay=10, adv_end=15, advloss_w=1, advtrain=False, advtrain_data='train', alpha=0.5, attack_al='sea', attack_mode='q', attack_only=True, attacked_checkpoint='data/bs256.pth', checkpoint=None, describe='describe your setting', epsilon=0.3, eval_advtrain=False, fliprate=0.0, generate_adv_example=True, iteration=2, lr_decay=15, name=[], paraphrase_data='val', resume=None, samples_frac=1, test_advtrain=False, topk=3, vqacp=False)
--------------------------------------------------
Num obj:  100
Num epochs:  23
Batch size:  256
Model type:  baseline
Optimization Method:  Adamax
Schedule Method:  warm_up
Loss Method:  binary_cross_entropy_with_logits
Clip Value:  0.25
Init LR:  0.001
LR decay step:  2
LR decay rate:  0.25
LR half life:  50000
Normalize visual feature:  False
Print Gradient:  False
Normalize Box Size:  False
Max answer choice:  3129
Manually set max question lenght:  666
Random Seed:  5225
gradual_warmup_steps:  [0.0005, 0.001, 0.0015, 0.002]
Weight Decay:  0.0
--------------------------------------------------
GPU ID 0
Loading model parameters.
Loading model parameters.
Loading model parameters.
Loading model parameters.
sea :   0% 0/838 [00:00<?, ?it/s]Traceback (most recent call last):
  File "main.py", line 74, in <module>
    main()
  File "main.py", line 60, in main
    attackvqa.attack(attackvqa.val_loader)
  File "/home/noa/Projects/seada-vqa/seada/adversarial_vqa.py", line 178, in attack
    v, b, v_mask, q_adv, q_len_adv, q_mask_adv, answer, q_str_adv, image_id_adv, q_id_adv = self.adversarial.perturb((v, b, q, q_str, v_mask, q_mask, image_id, q_id, q_len), y=answer, oripred=clean_logits)
  File "/home/noa/Projects/seada-vqa/seada/attacks.py", line 208, in perturb
    q_len_clue = sorted(q_len_clue, key=lambda x: x[0], reverse=True)
RuntimeError: bool value of Tensor with more than one value is ambiguous
sea :   0% 0/838 [02:05<?, ?it/s]

Any idea why this is happening?

Thanks

Cannot generate paraphrases of questions

Hi!

Thank you for your work and for releasing the code of this interesting paper.

I am at trying to generate paraphrases of the questions (step 1) but the code fails at some point.

This is the command I run:

python main.py --attack_only --attack_mode q --attack_al sea --fliprate 0 --topk 1 --paraphrase_data train --attacked_checkpoint data/bs256.pth

And I get the following output:

--------------------------------------------------
Namespace(adv_delay=10, adv_end=15, advloss_w=1, advtrain=False, advtrain_data='train', alpha=0.5, attack_al='sea', attack_mode='q', attack_only=True, attacked_checkpoint='data/bs256.pth', checkpoint=None, describe='describe your setting', epsilon=0.3, eval_advtrain=False, fliprate=0.0, generate_adv_example=True, iteration=2, lr_decay=15, name=[], paraphrase_data='train', resume=None, samples_frac=1, test_advtrain=False, topk=1, vqacp=False)
--------------------------------------------------
Num obj:  100
Num epochs:  23
Batch size:  256
Model type:  baseline
Optimization Method:  Adamax
Schedule Method:  warm_up
Loss Method:  binary_cross_entropy_with_logits
Clip Value:  0.25
Init LR:  0.001
LR decay step:  2
LR decay rate:  0.25
LR half life:  50000
Normalize visual feature:  False
Print Gradient:  False
Normalize Box Size:  False
Max answer choice:  3129
Manually set max question lenght:  666
Random Seed:  5225
gradual_warmup_steps:  [0.0005, 0.001, 0.0015, 0.002]
Weight Decay:  0.0
--------------------------------------------------
GPU ID 0
Loading model parameters.
Loading model parameters.
Loading model parameters.
Loading model parameters.
sea :   0% 0/1699 [00:00<?, ?it/s]Traceback (most recent call last):
  File "main.py", line 74, in <module>
    main()
  File "main.py", line 60, in main
    attackvqa.attack(attackvqa.val_loader)
  File "/home/noa/Projects/seada-vqa/seada/adversarial_vqa.py", line 178, in attack
    v, b, v_mask, q_adv, q_len_adv, q_mask_adv, answer, q_str_adv, image_id_adv, q_id_adv = self.adversarial.perturb((v, b, q, q_str, v_mask, q_mask, image_id, q_id, q_len), y=answer, oripred=clean_logits)
  File "/home/noa/Projects/seada-vqa/seada/attacks.py", line 193, in perturb
    q_adv, q_len_adv, q_mask_adv, q_str_adv, flipsign = self.find_flips(q_str[i], visual=(v[i], b[i], v_mask[i]), topk=topk, fliprate=fliprate, threshold=-10, oripred=oripred[i])
  File "/home/noa/Projects/seada-vqa/seada/attacks.py", line 233, in find_flips
    paraphrases = self.ps.generate_paraphrases(instance_for_onmt, topk=topk+1, edit_distance_cutoff=4, threshold=threshold)
  File "/home/noa/Projects/seada-vqa/seada/sea/paraphrase_scorer.py", line 797, in generate_paraphrases
    translation, mapping = choose_forward_translation(sentence, to, back, n=5)
  File "/home/noa/Projects/seada-vqa/seada/sea/paraphrase_scorer.py", line 31, in choose_forward_translation
    return_from_mapping=True)[0]
  File "/home/noa/Projects/seada-vqa/seada/sea/onmt_model.py", line 182, in translate
    self.translator.translate(batch, data))
  File "/home/noa/anaconda2/envs/seada/lib/python3.6/site-packages/OpenNMT_py-0.1-py3.6.egg/onmt/Translator.py", line 235, in translate
  File "/home/noa/anaconda2/envs/seada/lib/python3.6/site-packages/OpenNMT_py-0.1-py3.6.egg/onmt/Translator.py", line 235, in <listcomp>
  File "/home/noa/anaconda2/envs/seada/lib/python3.6/site-packages/OpenNMT_py-0.1-py3.6.egg/onmt/Translator.py", line 63, in buildTargetTokens
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number
sea :   0% 0/1699 [01:08<?, ?it/s]

Do you have any clue how can I solve it and run the code to generate the questions with paraphrases?

Thanks

Noa

I can't download the translation model from the drivegoogle link

thanks for your sharing.But I can't download the translation model . Can you share it with me? Thanks a lot!