nayeon7lee/bert-summarization

Implementation of 'Pretraining-Based Natural Language Generation for Text Summarization'

Paper: https://arxiv.org/pdf/1902.09243.pdf

Versions

• python 2.7
• PyTorch: 1.0.1.post2

Preparing package/dataset

0. Run: pip install -r requirements.txt to install required packages
1. Download chunk CNN/DailyMail data from: https://github.com/JafferWilson/Process-Data-of-CNN-DailyMail
2. Run: python news_data_reader.py to create pickle file that will be used in my data-loader

Running the model

For me, the model was too big for my GPU, so I used smaller parameters as following for debugging purpose. CUDA_VISIBLE_DEVICES=3 python main.py --cuda --batch_size=2 --hop 4 --hidden_dim 100

Note to reviewer:

• Although I implemented the core-part (2-step summary generation using BERT), I didn't have enough time to implement RL section.
• The 2nd decoder process is very time-consuming (since it needs to create BERT context vector for each timestamp).

bert-summarization's Issues

Preparing package/dataset question

Sorry, but i do not understand how to get news_data_reader.py file. Can you help me?

Could you please update Python 3 code?

Maybe this code cannot be run on test set

When config.test is True, this part will be invoked.
However, the Transformer is not defined or imported, this is very confusing.
Besides, the data_loader_test is not defined, I this it should have been test_dl given the context.

Could you share the results ?

I was wondering if you could reproduce the paper's results with your implementation.

Is it possible for you to share your results ?

Why did u get OOV while using BERT

Bert uses word piece embedding which covers almost any text data. Why did the concept of OOV come here?

start_p = abstract.index(config.SENTENCE_START, cur) TypeError: a bytes-like object is required, not 'str'

when i use python3,how could i deal with this problem?

no gradient during training?

In the train_one_batch function, you call the self.generate_refinement_output.
This function has torch.no_grad() in side. Does that mean you do not track gradients during training?

Too slow training

I'm running your code on Colab's GPU, but the training is very slow, even using your debug configuration :

---

Any tip to run the training under 12h ?

---

Update

I have the same problem with my local GPU (using the same debug config as you). I'm using CNN Dailymail dataset.

Any thoughts on that @nayeon7lee ?

news_data_reader.py

Where is your news_data_reader.py?
Thank you~

can this code run on multi-gpus?