younei/nlp-project

Abstractive Summarisation

Natural Language Processing Assignment 2&3

Group member (UCL): Youning, Wan Jing, Zoey, YanSong

Due date: Saturday, 21 March 2020, 12:05 AM

Report and code link : LINK

The report is written in a ACL format. The code is mostly implemented in Pytorch. Meanwhile, it also requires the evaluation metric packages:

• ROUGE
• BERTScore

Coursework instruction: Link

Abstract:

This paper sets out to assess the performanceof Deep Reinforcement Learning (DRL) based abstractive summarization models. 4 different model variants are applied on 3 datasets: CNN/Daily Mail, Gigaword and WikiHow, with ROUGE and BERTScores evaluated. Working on the novel WikiHow dataset which is slightly more complex to train on has magnified the characteristics of the models. It exposes the instability of training on ROUGE-L scores in some cases and suggests BERTScore as an alternative.

Model implemented:

The model is an encoder-decoder LSTM network with attention mechanism applied on both encoder and decoder, also the pointer network (https://arxiv.org/abs/1602.06023) is included. The schemetic model plot is shown as:

Model training objective:

What distinguishes between different model variants is the training objective. We have tried four types:

• Maximum likelihood (ML): this is to maximise the log-probability of obtaining the correct outputs

• Reinforcement learning (RL) with ROUGE reward: this is analogous to the REINFORCE algorithm in policy gradient method where the objective is to maximise the probability of obtaining the highest ROUGE score. Different from ML, now we directly optimise the model w.r.t the evaluation metric and it is expected that it will have higher testing score

• RL with BERScore reward: same as previous one but with different reward

• Hybrid (ML+RL) objective : this is combining the objective function of ML and RL.

Saved models and data:

Below are the saved (pre-trained) NLP models under different training objectives and the datasets as well.

-----------------------------------WikiHow-----------------------------------------------------

WikiHow data pre_trained model (ML for 10,000 iterations): LINK

WikiHow data files: LINK

WikiHow RL training models: LINK

BERTScore: LINK

-----------------------------------CNN/DM-----------------------------------------------------

CNN train/valid/test/vocab files (Mike): LINK

CNN/DailyMail train/valid/test/vocab files (Youning)(including pre-trained models) : LINK

Pre-trained Model Colab(Youning): LINK

CNN ML trained models(Zoey): folder link

CNN RL(r) trained models(Zoey): folder link

CNN RL+ML trained models(YS): folder link

CNN RL(b) trained models(Zoey): folder link

-----------------------------------Gigaword-----------------------------------------------------

Gigaword pre-trained model (ML): LINK

Gigaword data: LINK

Gigaword ML+RL trained models: LINK

Gigaword ML trained models: LINK

Gigaword RL trained models: on DeepNotes.

Relavant publications

• Deep Transfer Reinforcement Learning for Text Summarization ---Yaser et al.(2019), code
• Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization ---Siyao et al.(2019)
• A DEEP REINFORCED MODEL FOR ABSTRACTIVE SUMMARIZATION ---Paulus et al.(2017), code, other implementations
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting, code

Not that relavant but very fundamental and useful publications

• Sequence to Sequence Learning with Neural Networks ---Sutskever et al., 2014. Encoder-Decoder Network.

Relavant datasets

• CNN-Daily dataset ---helper codes
• Newsroom dataset ---helper codes
• WikiHow dataset ---paper, data, Processed txt data, bin file
• BigPatent dataset ---paper, data

Some useful webset

• Figure-eight---dataset, not that good
• niderhoff ---NLP dataset
• Browse State-of-the-Art ---SOTA model and methods
• text_summurization_abstractive_methods ---This repo is built to collect multiple implementations for abstractive approaches to address text summarization
• Comprehensive Research Summary of summarisation in NLP ---This repo contains summarisation relevant dataset, word embedding method, sequence embedding method, etc, will be a good guide when we do background research and get hint about how to improve our methods.
• Embed, encode, attend, predict: The new deep learning formula for state-of-the-art NLP models ---Four steps strategy for deeplearning with text, examples attached.
• Encoder-Decoder Sequence to Sequence Model ---An explaination of Encoder-Decoder model in machine translation.

Advices from Jiang, Minqi

• Basic RL algorithm
• Policy Gradient Algorithms

Useful Recource for Experiment

• ROUGE evaluation ---sets of metric for evaluating Abstrative Summarisation result.
• Code for attention-based summarisation ----Neural Attention Model for Abstractive Summarization, Github.
• Text Summarization models ---With tutorials!
• Text-Summarizer-Pytorch ---I tried this one but failed to write the data into binary file.
• Ocean! ---NMT means Neural Machine Translation...
• einops ---useful package for tensor operation.

Useful Resource for Writing Paper

• Styling plots for publication with matplotlib - how to use matplotlib much cooler