Comments (13)
Hi, you need to first train the keyword tagger and generate unconditional summaries. You can follow the README on "train the keyword tagger" and "evaluate CTRLsum" to replicate the results. Please let us know if you encounter any further issues.
from ctrl-sum.
Thanks for the response, I'll try to follow those steps, please allow me to keep the issue alive for a little longer
from ctrl-sum.
when you say run this command for evaluation,
bash scripts/test_bart.sh -g [GPUs] -s [source file name, NOT full path] -d [dataset] -p [ctrlsum checkpoint directory]
could you mention what exactly you mean by the source file, would it be test.source
in the following case?
bash scripts/test_bart.sh -g 1 -s test.source -d cnndm -p ../cnndm_ctrlsum
from ctrl-sum.
The source file is the actual input to CTRLsum, for example, it would be test.predwordsource
for unconditional summarization, and test.oraclenssource
would produce the oracle performance.
from ctrl-sum.
I'm off by a couple points when I ran the numbers,
1 ROUGE-1 Average_P: 0.36141 (95%-conf.int. 0.35914 - 0.36381)
1 ROUGE-1 Average_F: 0.43591 (95%-conf.int. 0.43398 - 0.43800)
---------------------------------------------
1 ROUGE-2 Average_R: 0.29378 (95%-conf.int. 0.29083 - 0.29666)
1 ROUGE-2 Average_P: 0.17329 (95%-conf.int. 0.17135 - 0.17529)
1 ROUGE-2 Average_F: 0.21002 (95%-conf.int. 0.20798 - 0.21212)
---------------------------------------------
1 ROUGE-L Average_R: 0.56304 (95%-conf.int. 0.56008 - 0.56615)
1 ROUGE-L Average_P: 0.33580 (95%-conf.int. 0.33365 - 0.33806)
1 ROUGE-L Average_F: 0.40557 (95%-conf.int. 0.40362 - 0.40768)
I'll recheck my process, but I did most of the stuff correctly.
from ctrl-sum.
Hi, can you check this thread to see if there is any helpful information there? One important point is that we used the tagger checkpoint with the best validation loss instead of the last checkpoint (because of overfitting).
I can try to help debug if you post your tagger training log here.
I am also glad to share our pretrained tagger if you contact me through email: [email protected]
from ctrl-sum.
Hello @jxhe, Could you tell me what was the compute you used to train the model? Im trying to replicate the actual training too (but Im afraid I might have a smaller GPU) so i just want to know what i would need
from ctrl-sum.
Hi, we used 8 16G v100 GPUs to train, which takes 1-2 days on the CNNDM dataset
from ctrl-sum.
@jxhe Do you have any suggestions if I'm trying to make this work on a GPU with 12GB vRAM
from ctrl-sum.
You can play with the max_tokens
and update_freq
variables in the training script to match our effective batch size:
ctrl-sum/scripts/train_bart.sh
Line 17 in b9afc42
If you want to train this on one GPU, then you may need to set update_freq
8x larger like 64 to match 8-gpu batch size; if max_tokens=1024
results in an out-of-memory error in your 12GB VRAM, you may need to set that smaller like 512 and further increase the update_freq
, for example, max_tokens=512, update_freq=128
, but this would take a long time to train
from ctrl-sum.
thanks a lot @jxhe for the tips :)
from ctrl-sum.
hello, @jxhe
2021-07-24 12:36:18 | WARNING | fairseq.data.data_utils | 232243 samples have invalid sizes and will be skipped, max_positions=(512, 512), first few sample ids=[99630, 150074, 103313, 240036, 226747, 108275, 29995, 138361, 64376, 130301]
I keep getting this error, I understand that if i set the max_tokens=512
, then i'll probably have to decrease the max_position=512
in the preprocessing script as well, but that didnt exactly solve the problem, do you have any idea?
and since a tonne of these examples are being skipped, the data loader is emptier than what is expected which results in
2021-07-24 13:17:12 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.
Sorry to fall back on you for every issue.
from ctrl-sum.
Hi, I am not sure why this happens, have you turned on --truncate-source
when training the model? Can you share your training log which would be better to debug?
from ctrl-sum.
Related Issues (16)
- Oracle entity in Table 2 VS. Oracle keywords in Table 7 HOT 5
- Question about the baselines used in the paper HOT 2
- Error: Could not find or load main class edu.stanford.nlp.process.PTBTokenizer HOT 1
- how to do test-time keyword extraction? HOT 9
- 'Confidence score' for a generated summary? HOT 2
- Error when using long text summaries HOT 2
- What is the input sequence size ? HOT 1
- hello! could you provide the contributions test data in your paper? thank you very much! HOT 1
- Tagger: bert or roberta? HOT 11
- fairseq-train: error: argument --restore-file: expected one argument HOT 4
- Qs on finetuning by myself on CNN dataset HOT 5
- package for CTRLsum (easy to use) HOT 5
- Summary length truncated even set min_length explicitly HOT 3
- Preprocess stuck HOT 5
- Use only selected sentences in source document HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ctrl-sum.