hsinyuan-huang / flowqa Goto Github PK
View Code? Open in Web Editor NEWImplementation of conversational QA model: FlowQA (with slight improvement)
Implementation of conversational QA model: FlowQA (with slight improvement)
I see that the FlowQA has already got a score on the leaderboard of the CoQA.
CoQA requires run experiments on Codalab.
So , Do you have a docker image that can run your code on.
Please give the link.
What are the hyper parameter setting of your best model ?
I have a hard time to reproduce your experiments result.
I think there may be something I done wrong.
hi, i don't know what the means of "*3", what 3 means?
context_feature = torch.Tensor(batch_size, question_num, context_len, \ feature_len + (self.dialog_ctx * 3)).fill_(0).to(self.device)
and
if prv_ans_choice == 3: # There is an answer for k in range(prv_ans_st, prv_ans_end + 1): if k >= context_len: break context_feature[i, j, k, feature_len + prv_ctx * 3 + 1] = 1 else: context_feature[i, j, :select_len, feature_len + prv_ctx * 3 + 2] = 1
How to use this on our input
Can you know which part of the code the hidden representation described in the paper is?
Well, I calculated the final F1 score of all the extracted answers with max F1 of gold answers and it's 89.
So do you think it's may cause bad performance by using them as supervised? I don't know how to use the span_text and input_text now. Thanks!
Hello, I want to ask a question. In the training, each question is provided with previous N gold answers, and it is also operated on the development set, but at the time of testing, I don't know what the gold answer is. I don't know how to deal with this? Use the answer predicted by model on the previous questions?
When I execute python train_QuAC.py,there are errors as following:
Traceback (most recent call last):
File "train_QuAC.py", line 324, in
main()
File "train_QuAC.py", line 175, in main
model = QAModel(opt, train_embedding)
File "/home/zoulongkun/project/FlowQA-master/QA_model/model_QuAC.py", line 30, in init
self.network = FlowQA(opt, embedding)
File "/home/zoulongkun/project/FlowQA-master/QA_model/detail_model.py", line 47, in init
self.CoVe = layers.MTLSTM(opt, embedding)
File "/home/zoulongkun/project/FlowQA-master/QA_model/layers.py", line 161, in init
state_dict = torch.load(opt['MTLSTM_path'])
File "/home/zoulongkun/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 358, in load
return _load(f, map_location, pickle_module)
File "/home/zoulongkun/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 532, in _load
magic_number = pickle_module.load(f)
EOFError: Ran out of input
Your help will be highly appreciated!
i use --resume, after i evaluate the model and then train, i get this error:
Traceback (most recent call last):
File "train_CoQA.py", line 229, in
main()
File "train_CoQA.py", line 122, in main
model.update(batch)
File "/home/susht3/workspace/flow/QA_model/model_CoQA.py", line 128, in update
self.optimizer.step()
File "/home/susht3/local/anaconda3/envs/susht/lib/python3.6/site-packages/torch/optim/adamax.py", line 75, in step
exp_avg.mul_(beta1).add_(1 - beta1, grad)
RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #4 'other' [INFO/MainProcess] process shutting down
answer_choice = 0 if answer == 'CANNOTANSWER' else
1 if qa['yesno'] == 'y' else
2 if qa['yesno'] == 'n' else
3 # Not a yes/no question
if answer_choice != 0:
"""
0: Do not ask a follow up question!
1: Definitely ask a follow up question!
2: Not too important, but you can ask a follow up.
"""
answer_choice += 10 * (0 if qa['followup'] == "n" else
1 if qa['followup'] == "y" else
2)
Hi, this code comes from a function named proc_tran(ith,article) in preprocess_Quac.py . But I found that the qa['yesno'] doesn`t have a value 'n' while the qa['followup'] have a value 'm' in train.json .
Hello, I want to know how you solve the problem of co-reference and ellipsis in the conversation.
i change the code to evaluate for n batches, and get this error:
Traceback (most recent call last):
File "train_CoQA.py", line 234, in
main()
File "train_CoQA.py", line 122, in main
model.update(batch)
File "/home/susht3/workspace/flow/QA_model/model_CoQA.py", line 87, in update
score_s, score_e, score_c = self.network(*inputs)
File "/home/susht3/local/anaconda3/envs/susht/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/susht3/workspace/flow/QA_model/detail_model.py", line 163, in forward
representation_with_bos_eos, mask_with_bos_eos
File "/home/susht3/local/allennlp/allennlp/nn/util.py", line 1222, in remove_sentence_boundaries
sequence_lengths = mask.sum(dim=1).detach().cpu().numpy()
RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
when i eval the model, i get this error:
Traceback (most recent call last):
File "train_coqa.py", line 126, in
if name == 'main':
File "train_coqa.py", line 103, in main
if epoch % args.eval_per_epoch == 0:
File "train_coqa.py", line 53, in eval_epoch
for batch in (batches):
File "/home/workspace/flowQA/QA_model/model_CoQA.py", line 141, in predict
# Run forward
File "/home/workspace/flowQA/QA_model/model_CoQA.py", line 209, in update_eval_embed
File "/home/local/anaconda3/envs/susht/lib/python3.6/site-packages/torch/nn/modules/module.py", line 518, in getattr
type(self).name, name))
AttributeError: 'FlowQA' object has no attribute 'eval_embed'
Hello, I want to ask a question. What is the version number of the package in the requirement document?
Hi,
Thank you for sharing the code! It really helps us better understand the FlowQA model.
It seems that in your implementation, both model_CoQA and model_QuAC use the same FlowQA network defined in detail_model.py. I was wondering if you actually used different answer predictors for CoQA and QuAC to respect the difference between the two datasets. Also I noticed that when computing loss and evaluating results, you did treat the output of the network differently for CoQA and QuAC in your code. Your help will be highly appreciated!
Hi,
During training, validation is performed (on the dev dataset) per epoch and corresponding F1 is calculated. The best F1 of 64% was found during the 15th epoch(total 30 epochs) and I saved the model at the 15th epoch. Thereafter, I used this model in predict_QuAC to predict on dev dataset. The F1 score after prediction is down to 61%. I am clueless as to why there is a huge reduction in the accuracy, even though the dev dataset is used for both validation and prediction.
Please let me know why there is this slump in accuracy.
Thanks
I noticed that you used many transposes rather than one permutation in the flow operation, is that for better performance?
I think the following code is equivalent to the original code,
def flow_operation(cur_h, flow):
n, t, c = x1_full.size()
# flow_in = cur_h.transpose(0, 1).view(c, n, t, -1)
# flow_in = flow_in.transpose(0, 2).contiguous().view(t, n * c, -1).transpose(0, 1)
flow_in = cur_h.view(n, t, c, -1).permute(0, 2, 1, 3).contiguous().view(n * c, t, -1)
# [bsz * context_length, max_qa_pair, hidden_state]
flow_out = flow(flow_in)
# [bsz * context_length, max_qa_pair, flow_hidden_state_dim (hidden_state/2)]
if self.opt['no_dialog_flow']:
flow_out = flow_out * 0
# flow_out = flow_out.transpose(0, 1).view(t, n, c, -1).transpose(0, 2).contiguous()
# flow_out = flow_out.view(c, n * t, -1).transpose(0, 1)
# [bsz * max_qa_pair, context_length, flow_hidden_state_dim]
flow_out = flow_out.view(n, c, t, -1).permute(0, 2, 1, 3).contiguous().view(n * t, c, -1)
return flow_out
Hi,
Thank you for releasing the code! Could you please also share the code corresponding to the SCONE experiments?
@momohuang Thank you!!
hi,i find that you use previous rationale to get answer id, and use previous answer id for prediction.
i also use previous rationale to extract answer, but when i submit to the leaderboard, author tell me that i cannot use previous rationale
When I execute python train_QuAC.py,there are errors as following:
After Input LSTM, the vector_sizes [doc, query] are [ 250 250 ] * 2
Self deep-attention 250 rays in 750-dim space
Before answer span finding, hidden size are 250 250
12/20/2018 04:43:43 [dev] Total number of params: 11852394
12/20/2018 16:43:43 - INFO - main - [dev] Total number of params: 11852394
12/20/2018 04:43:45 Epoch 1
12/20/2018 16:43:45 - WARNING - main - Epoch 1
12/20/2018 04:43:46 updates[ 1] train loss[15.87973] remaining[1:26:47]
12/20/2018 16:43:46 - INFO - main - updates[ 1] train loss[15.87973] remaining[1:26:47]
12/20/2018 04:44:00 updates[ 21] train loss[10.87507] remaining[0:45:12]
12/20/2018 16:44:00 - INFO - main - updates[ 21] train loss[10.87507] remaining[0:45:12]
12/20/2018 04:44:14 updates[ 41] train loss[10.16175] remaining[0:44:17]
12/20/2018 16:44:14 - INFO - main - updates[ 41] train loss[10.16175] remaining[0:44:17]
12/20/2018 04:44:29 updates[ 61] train loss[9.96472] remaining[0:45:30]
12/20/2018 16:44:29 - INFO - main - updates[ 61] train loss[9.96472] remaining[0:45:30]
12/20/2018 04:44:48 updates[ 81] train loss[9.56536] remaining[0:49:21]
12/20/2018 16:44:48 - INFO - main - updates[ 81] train loss[9.56536] remaining[0:49:21]
12/20/2018 04:45:03 updates[ 101] train loss[9.38102] remaining[0:48:40]
12/20/2018 16:45:03 - INFO - main - updates[ 101] train loss[9.38102] remaining[0:48:40]
12/20/2018 04:45:17 updates[ 121] train loss[9.11970] remaining[0:47:07]
12/20/2018 16:45:17 - INFO - main - updates[ 121] train loss[9.11970] remaining[0:47:07]
12/20/2018 04:45:35 updates[ 141] train loss[8.99858] remaining[0:48:30]
12/20/2018 16:45:35 - INFO - main - updates[ 141] train loss[8.99858] remaining[0:48:30]
12/20/2018 04:45:49 updates[ 161] train loss[8.76992] remaining[0:47:21]
12/20/2018 16:45:49 - INFO - main - updates[ 161] train loss[8.76992] remaining[0:47:21]
12/20/2018 04:46:03 updates[ 181] train loss[8.64918] remaining[0:46:42]
12/20/2018 16:46:03 - INFO - main - updates[ 181] train loss[8.64918] remaining[0:46:42]
12/20/2018 04:46:17 updates[ 201] train loss[8.58265] remaining[0:46:12]
12/20/2018 16:46:17 - INFO - main - updates[ 201] train loss[8.58265] remaining[0:46:12]
12/20/2018 04:46:30 updates[ 221] train loss[8.47283] remaining[0:45:20]
12/20/2018 16:46:30 - INFO - main - updates[ 221] train loss[8.47283] remaining[0:45:20]
12/20/2018 04:46:43 updates[ 241] train loss[8.38734] remaining[0:44:24]
12/20/2018 16:46:43 - INFO - main - updates[ 241] train loss[8.38734] remaining[0:44:24]
12/20/2018 04:46:57 updates[ 261] train loss[8.36940] remaining[0:44:00]
12/20/2018 16:46:57 - INFO - main - updates[ 261] train loss[8.36940] remaining[0:44:00]
Traceback (most recent call last):
File "train_QuAC.py", line 324, in
main()
File "train_QuAC.py", line 209, in main
model.update(batch)
File "/home/zys/文档/FlowQA-master/QA_model/model_QuAC.py", line 83, in update
score_s, score_e, score_no_answ = self.network(*inputs)
File "/home/zys/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/zys/文档/FlowQA-master/QA_model/detail_model.py", line 306, in forward
highlvl_self_attn_hiddens = self.highlvl_self_att(x1_att, x1_att, x1_mask, x3=doc_hiddens, drop_diagonal=True)
File "/home/zys/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/zys/文档/FlowQA-master/QA_model/layers.py", line 285, in forward
alpha = F.softmax(scores, dim=2)
File "/home/zys/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 889, in softmax
return input.softmax(dim)
RuntimeError: CUDA error: out of memory
Hi, during my run time I have found that batch size parameter could impact the performance of the model during testing time?, Could it be due to fact that the same input sentence with different size of padding ([0,0,1,1] in batch size of 1 and [0,0,1,1,1,1,1] when mixing with other examples) can lead to some difference in output from final layer of the model?
As a recent SOTA model in dialog QA, do you plan to add the FlowQA model to AllenNLP library? Or other NLP library such as fairseq and openNMT.
Thank you very much!
@momohuang
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.