Comments (10)
Hi @zycdev ,
Here is an example for running the new version:
For task 1, run:
CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py --batch-size 16 --expname test --weight-decay 0.01
For task 2, run:
CUDA_VISIBLE_DEVICES=0 python train.py --load --load-path saved/bert-base-uncased-test.bin --mode '#2' --lr1 2e-5 --gradient-accumulation-steps 8 --expname test --tune
(that's right, now we only need one GPU for task 2)
For inference, run:
CUDA_VISIBLE_DEVICES=0 python infer.py --data-file data/hotpot_dev_fullwiki_v1_merge.json --model-file saved/bert-base-uncased-test.bin
Evaluation:
python scripts/hotpot_evaluate_v1.py data/hotpot_dev_fullwiki_v1_merge_pred.json data/hotpot_dev_fullwiki_v1_merge.json
from cogqa.
Hi, thank you for pointing out it. I have conducted some minor modification in the pre-processing scripts after generating the examples, but I do not think it is a main reason (maybe I am wrong). In my experiments, learning_rate, batch_size and early stop strategies(if you add) and some other parameters can affect the results up to 10%. Maybe you can try to delete the linear_warm_up in task #2(I realize that after finishing the paper)?
from cogqa.
Hi, @Sleepychord , very thank you for your reply! I retrained BERT for 1 epoch and then BERT & GNN for 1 epoch with hyperparameters as you showed in the paper, but I still can't reproduce the result of the paper on dev set.
My training commands:
export CUDA_VISIBLE_DEVICES=0,1,2,3 # 4 K80(12GB memory) GPUs
python train.py --batch-size=10 --lr1=1e-4
python train.py --load=True --mode='bundle' --batch-size=10 --lr1=4e-5 --lr2=1e-4 # haven't delete the linear_warm_up yet
and my evaluation result on dev set:
{'em': 0.2598244429439568, 'f1': 0.35564370767865855, 'prec': 0.37582762612134724, 'recall': 0.35888658012669966, 'sp_em': 0.07562457798784605, 'sp_f1': 0.3665706092242228, 'sp_prec': 0.4997955049676863, 'sp_recall': 0.3207705540014783, 'joint_em': 0.03349088453747468, 'joint_f1': 0.19135653981707093, 'joint_prec': 0.2720478977096129, 'joint_recall': 0.17037639264369026}
Could you provide more details about hyperparameters, training strategies of your best experimental result? I am looking forward to your advice.
Thanks!
from cogqa.
Hi @zycdev ,
I'm not sure what's the problem you encountered but I've successfully got reasonable results with the scripts you provided.
I also made an improved version of CogQA here, which is be much faster and far less resource-demanding for task 2, with slightly better results. You can try that out.
Hope this helps!
from cogqa.
@zycdev , I think that tuning the learning_rate in task #2 is effective. Thank @qibinc for improvement and maybe you can follow it.
from cogqa.
@qibinc @Sleepychord Thank you very much for your work, I am glad to try the new version!
from cogqa.
Hi @qibinc , I am grateful for the guideline, just wanting to ask for this :-D
from cogqa.
CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py --batch-size 16 --expname test --weight-decay 0.01
老哥你好,我看你的代码中数据用的是"hotpot_train_v1.1_refined.json",这个refined是因为对数据做了什么改变吗?
from cogqa.
@ditingdapeng refined是预处理过的,每个QA pair增加了两个域表示模糊匹配等算法抽取出来的真实cognitive graph的节点,对数据本身没有别的改动。
from cogqa.
from cogqa.
Related Issues (20)
- dump.rdb HOT 4
- RuntimeError: CUDA error: an illegal memory access was encountered HOT 12
- fullwiki data HOT 1
- 关于bert使用不加载链接,而是通过文件加载的问题 HOT 1
- 请问下"hotpot_dev_fullwiki_v1_merge.json"文件从哪来? HOT 2
- 论文中"提取一跳节点,但是不计算语义向量"的含义 HOT 2
- sem[x,Q,clues]在系统一和系统二中的作用 HOT 5
- 数据集中的答案并不是来源于该条数据的材料--问题 HOT 1
- 代码中utils中的GENERAL_WD解释 HOT 1
- 不好意思一次性问了很多细节问题,希望您有时间可以解答下 HOT 1
- where can i find hotpot_train_v1.1_refined.json HOT 1
- NEG TOO LONG! HOT 1
- 将bert换成albert时,加载输入数据时出了个错误 HOT 2
- model2 = CognitiveGNN(model1.config.hidden_size) 'NoneType' object has no attribute 'config'
- model1 = BertForMultiHopQuestionAnswering.from_pretrained(BERT_MODEL, cache_dir=PYTORCH_PRETRAINED_BERT_CACHE / 'distributed_{}'.format(-1)) HOT 2
- 请问训练速度和什么有关呀?显卡?内存?
- Where can i get `hotpot_train_v1.1.json`? HOT 1
- 文件夹里enwiki-20171001-pages-meta-current-withlinks-abstracts应该有什么文件?
- System 1 with roberta?
- 感觉这代码没法复现
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cogqa.