Comments (13)
Accuracy is computed here :
PreSumm/src/models/reporter.py
Lines 237 to 239 in 29a6b1a
So yes, it's a percentage. (Percentage of correctly guessed words)
When using the cmd in the README, my accuracy was around 60% ~ 65%.
So yes, I think you're going in the wrong direction. Try to reproduce the paper's result first, then change the code as you need and compare the results changes.
from presumm.
Hi @colanim,
May I ask what results you got by running the code? I used exactly the same settings as the author did but all the ROUGE scores are around 0.8 lower than the reported ones.
from presumm.
I got similar results (only ~0.1 of rouge score difference).
Did you try to reproduce their results with the checkpoint ?
Also on how many GPU did you run the code ?
from presumm.
Hi @colanim ,
Thanks for your answers!
-
I 've tried their bertextabs checkpoint and the result is around 0.18-0.25 lower than their reported ones in terms of ROUGE-1,2 and L.
-
I used 2 GTX 1080 GPUs and used accum_count=10 (200k steps)
-
I used the same command and the same pre-processed dataset in README.
May I ask if your reproduced results with the checkpoints are also ~0.1 rouge score difference from the reported ones?
Thanks a lot.
from presumm.
It's weird if you get lower results using their checkpoints.
When using their checkpoint, results are similar.
Training from scratch gave me ~0.1 difference.
I can see you updated the number of accumulation steps according to the number of GPU, so this is not the problem.
Anyway if you cannot reproduce the results with their checkpoint, the training procedure is not the problem here.
Maybe the problem come from your pyrouge installation ? Can you run the unit tests of the pyrouge package ?
python -m pyrouge.test
If some tests are failing, look at this tutorial to install pyrouge.
from presumm.
Hi @colanim ,
Thanks a lot for your answers!
I can pass the pyrouge unit test and I actually followed that tutorial to install pyrouge.
May I confirm with you that for the abstractive baseline they provided, are you comparing with BertAbs or BertExtAbs?
from presumm.
I'm comparing with BertExtAbs
from presumm.
Are you considering the average score of the 5 best checkpoints, or only the best one ?
For my score I considered only the best checkpoint.
from presumm.
In my case the results of the five best checkpoints are similar.
from presumm.
Hi @colanim ,
The CNN/DM model evaluation command given in the README.md did not specify "use_bert_emb" as "true" .
May I ask if you set it to true?
Thanks!
from presumm.
In the README cmd, they did set -use_bert_emb true
.
Indeed in my training I set it to True
. That might be the reason of your lower results.
from presumm.
Actually I am talking about "testing". In testing, they did not set -use_bert_emb to true in the command?
from presumm.
In testing it does not change anything : if you trained your model sharing embeddings, the model loaded will have shared embeddings too.
from presumm.
Related Issues (20)
- having [Errno 21] Is a directory: while running train for BertExtAbs
- step 4 converting to simpler json returning asci error
- TypeError: __init__() got an unexpected keyword argument 'temp_dir'
- example_add_guidance.py
- Error when testing BertAbs model HOT 2
- Acc is very low and does not converge during training
- Cannot load model via torch.load HOT 1
- xsum数据集 HOT 1
- data preprocessing: empty 'tgt' text HOT 1
- issue for converting to bert_data HOT 2
- Use pretrained model : train_from HOT 9
- Getting the same sequence for all input candidate in generation
- How to do inference using pretrained bertsum models?
- Training the BERT large extractive model
- How can i know if i download BERT successfully
- bert-base-uncased HOT 2
- error in step 3 HOT 1
- 在运行test模式,BertAbs模型时,遇到了RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float' HOT 1
- How to save the best model? HOT 1
- RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from presumm.