Comments (17)
I have faced the same repeating issues when training korean model.
After some amount of research, I have found that this is a general problem for natural language generation and known as degeneration.
I have added an extra module for the decoder, replacing beam search.
Please let me know if anyone is interested in it
Thanks
from presumm.
@colanim
Sorry for late reply
I have replace the decoder with the following method which proposes a new way of sampling tokens at decoding steps, rather than just depending on beam search.
The paper has suggested two methods and they are implemented on here
From my experience, I could avoid the repetition issue with the proposed method and hence improve the ROUGE score !
Hope my opinion helps
from presumm.
mine also getting the same results. All output are same
from presumm.
me too.
from presumm.
Please paste your training commands here
from presumm.
python3 train.py -task abs -mode train -bert_data_path bert_data/ -dec_dropout 0.2 -model_path model_abs/ -sep_optim true -lr_bert 0.002 -lr_dec 0.2 -save_checkpoint_steps 2000 -batch_size 140 -train_steps 200000 -report_every 50 -accum_count 5 -use_bert_emb true -use_interval true -warmup_steps_bert 20000 -warmup_steps_dec 10000 -max_pos 512 -visible_gpus 0 -log_file abs_bert_cnndm
from presumm.
With only 1 gpu for training, you need to accumulate the gradient for a much larger step, or the model cannot be trained effectively.
from presumm.
SIr, But I have only one gpu. Can't the training be effective on that.
from presumm.
You can use our Trained Models.
from presumm.
With only 1 gpu for training, you need to accumulate the gradient for a much larger step, or the model cannot be trained effectively.
so you need to have how many gpu for training??
from presumm.
For extractive summarization, the author trained the model on 3 GPU.
For abstractive summarization, the author trained the model on 4 GPU for 2 days.
from presumm.
@robinsongh381 We are interested !
So you replaced beam search and got better results ?
from presumm.
Thanks for the message !
Do you remember (approximately) how big is the difference in ROUGE score ?
from presumm.
@colanim
Sorry for late reply
I have replace the decoder with the following method which proposes a new way of sampling tokens at decoding steps, rather than just depending on beam search.The paper has suggested two methods and they are implemented on here
From my experience, I could avoid the repetition issue with the proposed method and hence improve the ROUGE score !
Hope my opinion helps
can share your results ??
from presumm.
Hi. I am using the pre-trained models for testing on CNN dataset.
This is the command I am giving:
python train.py -task abs -mode test -test_from ~/Downloads/cnndm_baseline_best.pt -batch_size 3000 -test_batch_size 500 -bert_data_path ../bert_data/test -log_file ../logs/val_abs_bert_cnndm -sep_optim true -use_interval true -visible_gpus 0 -max_pos 512 -max_length 200 -alpha 0.95 -min_length 50 -result_path ../logs/abs_bert_cnndm
But my candidate result is same as the very first single file i used. After that it isn't changing. What am i doing wrong?
from presumm.
while training on 1 gpu we need to set the grad accum count greater than 5, u said. how much should that be? Please help.
With only 1 gpu for training, you need to accumulate the gradient for a much larger step, or the model cannot be trained effectively.
from presumm.
python3 train.py -task abs -mode train -bert_data_path bert_data/ -dec_dropout 0.2 -model_path model_abs/ -sep_optim true -lr_bert 0.002 -lr_dec 0.2 -save_checkpoint_steps 2000 -batch_size 140 -train_steps 200000 -report_every 50 -accum_count 5 -use_bert_emb true -use_interval true -warmup_steps_bert 20000 -warmup_steps_dec 10000 -max_pos 512 -visible_gpus 0 -log_file abs_bert_cnndm
Using a single gpu generation of sentence results, do you solve the problem?
from presumm.
Related Issues (20)
- having [Errno 21] Is a directory: while running train for BertExtAbs
- step 4 converting to simpler json returning asci error
- TypeError: __init__() got an unexpected keyword argument 'temp_dir'
- example_add_guidance.py
- Error when testing BertAbs model HOT 2
- Acc is very low and does not converge during training
- Cannot load model via torch.load HOT 1
- xsum数据集 HOT 1
- data preprocessing: empty 'tgt' text HOT 1
- issue for converting to bert_data HOT 2
- Use pretrained model : train_from HOT 9
- Getting the same sequence for all input candidate in generation
- How to do inference using pretrained bertsum models?
- Training the BERT large extractive model
- How can i know if i download BERT successfully
- bert-base-uncased HOT 2
- error in step 3 HOT 1
- 在运行test模式,BertAbs模型时,遇到了RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float' HOT 1
- How to save the best model? HOT 1
- RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from presumm.