how is accuracy measured and is it a percent?

[nlpyang]

how is accuracy measured and is it a percent?

[astariul]

Accuracy is computed here :

PreSumm/src/models/reporter.py

Lines 237 to 239 in 29a6b1a

	def accuracy(self):
	""" compute accuracy """
	return 100 * (self.n_correct / self.n_words)

So yes, it's a percentage. (Percentage of correctly guessed words)

When using the cmd in the README, my accuracy was around 60% ~ 65%.

So yes, I think you're going in the wrong direction. Try to reproduce the paper's result first, then change the code as you need and compare the results changes.

[ShirleyHan6]

Hi @colanim,

May I ask what results you got by running the code? I used exactly the same settings as the author did but all the ROUGE scores are around 0.8 lower than the reported ones.

[astariul]

I got similar results (only ~0.1 of rouge score difference).

Did you try to reproduce their results with the checkpoint ?

---

Also on how many GPU did you run the code ?

[ShirleyHan6]

Hi @colanim ,

Thanks for your answers!

• I 've tried their bertextabs checkpoint and the result is around 0.18-0.25 lower than their reported ones in terms of ROUGE-1,2 and L.

• I used 2 GTX 1080 GPUs and used accum_count=10 (200k steps)

• I used the same command and the same pre-processed dataset in README.

May I ask if your reproduced results with the checkpoints are also ~0.1 rouge score difference from the reported ones?

Thanks a lot.

[astariul]

It's weird if you get lower results using their checkpoints.

When using their checkpoint, results are similar.
Training from scratch gave me ~0.1 difference.

I can see you updated the number of accumulation steps according to the number of GPU, so this is not the problem.

Anyway if you cannot reproduce the results with their checkpoint, the training procedure is not the problem here.

Maybe the problem come from your pyrouge installation ? Can you run the unit tests of the pyrouge package ?

python -m pyrouge.test

If some tests are failing, look at this tutorial to install pyrouge.

[ShirleyHan6]

Hi @colanim ,

Thanks a lot for your answers!

I can pass the pyrouge unit test and I actually followed that tutorial to install pyrouge.

May I confirm with you that for the abstractive baseline they provided, are you comparing with BertAbs or BertExtAbs?

[astariul]

I'm comparing with BertExtAbs

[astariul]

Are you considering the average score of the 5 best checkpoints, or only the best one ?
For my score I considered only the best checkpoint.

[ShirleyHan6]

In my case the results of the five best checkpoints are similar.

[ShirleyHan6]

Hi @colanim ,

The CNN/DM model evaluation command given in the README.md did not specify "use_bert_emb" as "true" .

May I ask if you set it to true?

Thanks!

[astariul]

In the README cmd, they did set -use_bert_emb true.

Indeed in my training I set it to True. That might be the reason of your lower results.

[ShirleyHan6]

Actually I am talking about "testing". In testing, they did not set -use_bert_emb to true in the command?

[astariul]

In testing it does not change anything : if you trained your model sharing embeddings, the model loaded will have shared embeddings too.