zaynmi / seada-vqa Goto Github PK
View Code? Open in Web Editor NEWA pytorch implemetation of data augmentation method for visual question answering
License: MIT License
A pytorch implemetation of data augmentation method for visual question answering
License: MIT License
Hi,
Thank you for sharing the code on seada paper. I was trying to use this repo to reproduce your results, but got stuck when setting up the environment. There is no requirements.txt found in the repo, could you kindly include it?
Thanks.
paraphrases = self.ps.generate_paraphrases(instance_for_onmt, topk=topk+1, edit_distance_cutoff=4, threshold=threshold)
File "/home/scan/DVSI/sscc/paraphrase_scorer.py", line 799, in generate_paraphrases
translation, mapping = choose_forward_translation(sentence, to, back, n=5)
File "/home/scan/DVSI/sscc/paraphrase_scorer.py", line 30, in choose_forward_translation
translations = to_translator.translate([sentence], n_best=n, return_from_mapping=True)[0]
File "/home/scan/DVSI/sscc/onmt_model.py", line 179, in translate
for batch in testData:
File "/usr/local/lib/python3.6/site-packages/torchtext/data/iterator.py", line 151, in iter
self.train)
File "/usr/local/lib/python3.6/site-packages/torchtext/data/batch.py", line 27, in init
setattr(self, name, field.process(batch, device=device, train=train))
File "/usr/local/lib/python3.6/site-packages/torchtext/data/field.py", line 188, in process
tensor = self.numericalize(padded, device=device, train=train)
File "/usr/local/lib/python3.6/site-packages/torchtext/data/field.py", line 308, in numericalize
arr = self.postprocessing(arr, None, train)
TypeError: make_src() takes 2 positional arguments but 3 were given
Do you know how to solve this problem???
Hi again,
I have found a new problem. The code breaks if I change the topk parameter. For example, the following works well:
python main.py --attack_only --attack_mode q --attack_al sea --fliprate 0 --topk 1 --paraphrase_data val --attacked_checkpoint data/bs256.pth
But if I use 3 instead of 1 in topk:
python main.py --attack_only --attack_mode q --attack_al sea --fliprate 0 --topk 3 --paraphrase_data val --attacked_checkpoint data/bs256.pth
It produces the following error:
--------------------------------------------------
Namespace(adv_delay=10, adv_end=15, advloss_w=1, advtrain=False, advtrain_data='train', alpha=0.5, attack_al='sea', attack_mode='q', attack_only=True, attacked_checkpoint='data/bs256.pth', checkpoint=None, describe='describe your setting', epsilon=0.3, eval_advtrain=False, fliprate=0.0, generate_adv_example=True, iteration=2, lr_decay=15, name=[], paraphrase_data='val', resume=None, samples_frac=1, test_advtrain=False, topk=3, vqacp=False)
--------------------------------------------------
Num obj: 100
Num epochs: 23
Batch size: 256
Model type: baseline
Optimization Method: Adamax
Schedule Method: warm_up
Loss Method: binary_cross_entropy_with_logits
Clip Value: 0.25
Init LR: 0.001
LR decay step: 2
LR decay rate: 0.25
LR half life: 50000
Normalize visual feature: False
Print Gradient: False
Normalize Box Size: False
Max answer choice: 3129
Manually set max question lenght: 666
Random Seed: 5225
gradual_warmup_steps: [0.0005, 0.001, 0.0015, 0.002]
Weight Decay: 0.0
--------------------------------------------------
GPU ID 0
Loading model parameters.
Loading model parameters.
Loading model parameters.
Loading model parameters.
sea : 0% 0/838 [00:00<?, ?it/s]Traceback (most recent call last):
File "main.py", line 74, in <module>
main()
File "main.py", line 60, in main
attackvqa.attack(attackvqa.val_loader)
File "/home/noa/Projects/seada-vqa/seada/adversarial_vqa.py", line 178, in attack
v, b, v_mask, q_adv, q_len_adv, q_mask_adv, answer, q_str_adv, image_id_adv, q_id_adv = self.adversarial.perturb((v, b, q, q_str, v_mask, q_mask, image_id, q_id, q_len), y=answer, oripred=clean_logits)
File "/home/noa/Projects/seada-vqa/seada/attacks.py", line 208, in perturb
q_len_clue = sorted(q_len_clue, key=lambda x: x[0], reverse=True)
RuntimeError: bool value of Tensor with more than one value is ambiguous
sea : 0% 0/838 [02:05<?, ?it/s]
Any idea why this is happening?
Thanks
Hi!
Thank you for your work and for releasing the code of this interesting paper.
I am at trying to generate paraphrases of the questions (step 1) but the code fails at some point.
This is the command I run:
python main.py --attack_only --attack_mode q --attack_al sea --fliprate 0 --topk 1 --paraphrase_data train --attacked_checkpoint data/bs256.pth
And I get the following output:
--------------------------------------------------
Namespace(adv_delay=10, adv_end=15, advloss_w=1, advtrain=False, advtrain_data='train', alpha=0.5, attack_al='sea', attack_mode='q', attack_only=True, attacked_checkpoint='data/bs256.pth', checkpoint=None, describe='describe your setting', epsilon=0.3, eval_advtrain=False, fliprate=0.0, generate_adv_example=True, iteration=2, lr_decay=15, name=[], paraphrase_data='train', resume=None, samples_frac=1, test_advtrain=False, topk=1, vqacp=False)
--------------------------------------------------
Num obj: 100
Num epochs: 23
Batch size: 256
Model type: baseline
Optimization Method: Adamax
Schedule Method: warm_up
Loss Method: binary_cross_entropy_with_logits
Clip Value: 0.25
Init LR: 0.001
LR decay step: 2
LR decay rate: 0.25
LR half life: 50000
Normalize visual feature: False
Print Gradient: False
Normalize Box Size: False
Max answer choice: 3129
Manually set max question lenght: 666
Random Seed: 5225
gradual_warmup_steps: [0.0005, 0.001, 0.0015, 0.002]
Weight Decay: 0.0
--------------------------------------------------
GPU ID 0
Loading model parameters.
Loading model parameters.
Loading model parameters.
Loading model parameters.
sea : 0% 0/1699 [00:00<?, ?it/s]Traceback (most recent call last):
File "main.py", line 74, in <module>
main()
File "main.py", line 60, in main
attackvqa.attack(attackvqa.val_loader)
File "/home/noa/Projects/seada-vqa/seada/adversarial_vqa.py", line 178, in attack
v, b, v_mask, q_adv, q_len_adv, q_mask_adv, answer, q_str_adv, image_id_adv, q_id_adv = self.adversarial.perturb((v, b, q, q_str, v_mask, q_mask, image_id, q_id, q_len), y=answer, oripred=clean_logits)
File "/home/noa/Projects/seada-vqa/seada/attacks.py", line 193, in perturb
q_adv, q_len_adv, q_mask_adv, q_str_adv, flipsign = self.find_flips(q_str[i], visual=(v[i], b[i], v_mask[i]), topk=topk, fliprate=fliprate, threshold=-10, oripred=oripred[i])
File "/home/noa/Projects/seada-vqa/seada/attacks.py", line 233, in find_flips
paraphrases = self.ps.generate_paraphrases(instance_for_onmt, topk=topk+1, edit_distance_cutoff=4, threshold=threshold)
File "/home/noa/Projects/seada-vqa/seada/sea/paraphrase_scorer.py", line 797, in generate_paraphrases
translation, mapping = choose_forward_translation(sentence, to, back, n=5)
File "/home/noa/Projects/seada-vqa/seada/sea/paraphrase_scorer.py", line 31, in choose_forward_translation
return_from_mapping=True)[0]
File "/home/noa/Projects/seada-vqa/seada/sea/onmt_model.py", line 182, in translate
self.translator.translate(batch, data))
File "/home/noa/anaconda2/envs/seada/lib/python3.6/site-packages/OpenNMT_py-0.1-py3.6.egg/onmt/Translator.py", line 235, in translate
File "/home/noa/anaconda2/envs/seada/lib/python3.6/site-packages/OpenNMT_py-0.1-py3.6.egg/onmt/Translator.py", line 235, in <listcomp>
File "/home/noa/anaconda2/envs/seada/lib/python3.6/site-packages/OpenNMT_py-0.1-py3.6.egg/onmt/Translator.py", line 63, in buildTargetTokens
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number
sea : 0% 0/1699 [01:08<?, ?it/s]
Do you have any clue how can I solve it and run the code to generate the questions with paraphrases?
Thanks
Noa
thanks for your sharing.But I can't download the translation model . Can you share it with me? Thanks a lot!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.