smacawi/tweet-classifier

Training deep learning models to classify disaster-related tweets in the context of crisis management. Published as "Using deep learning and social network analysis to understand and manage extreme flooding" in Journal of Contingencies and Crisis Management

Using deep learning and social network analysis to understand and manage extreme flooding

Code for Using deep learning and social network analysis to understand and manage extreme flooding published in the Journal of Contingencies and Crisis Management.

SNA

For more information on the social network analysis, see https://smacawi.github.io/social-network-analysis-report.pdf

Citation

@article{romascanu2020using,
  title={Using deep learning and social network analysis to understand and manage extreme flooding},
  author={Romascanu, Andrei and Ker, Hannah and Sieber, Renee and Greenidge, Sarah and Lumley, Sam and Bush, Drew and Morgan, Stefan and Zhao, Rosie and Brunila, Mikael},
  journal={Journal of Contingencies and Crisis Management},
  volume={28},
  number={3},
  pages={251--261},
  year={2020},
  doi={https://doi.org/10.1111/1468-5973.12311},
  publisher={Wiley Online Library}
}

tweet-classifier's Issues

Tables for metrics:

• Number of oov words (from GlovE and Word2Vec)
• PRF metrics (All events, similar event, one event) (comparison of models - BERT, Glove RNN, Word2vec RNN)

Upload and link model tar.gz

BERT hyperparameter search:

• Optimizer (Adam LR 0.5e-3, 1e-3, 0.5e-2)
• Final layer (GRU, Bi-LSTM, Sent-pool)
• Batch-size (32, 64, 128)
  AlleNLP BERT implementation:
  https://medium.com/analytics-vidhya/fine-tuning-bert-with-allennlp-7459119b736c

Subword tokenization (byte pair encoding)

Is your feature request related to a problem? Please describe.
Social media text has a lot of non-standard English, leading to a lot of out-of-vocabulary words.
Subword tokenization algorithms such as byte-pair encoding can help with this issue, as well as bypass the need for text normalization which can lose a lot of information.

Describe the solution you'd like
Implement subword tokenization through AllenNLP library and evaluate it's impact on tweet classification.

Describe alternatives you've considered

• Word-level tokenization + normalization: not ideal due to reasons outlined above.
• Character-level tokenization: limited representational capabilities.

Additional context
NA

Update details about SNA

Upload experiment configs and outputs

Upload and link datasets

Update requirements to ensure replicability

Include library versions where applicable (e.g. allennlp) and add any missing requirements that prevent replicating the experiments.

Update code to most recent version on server

Update readme to ensure replicability

• specify steps for environment setup;
• specify steps for training;
• include link to allennlp documentation/examples for adequate version;
• specify twitter-scraper dependency
• specify steps for inference/deployment;