adellegia/watermelon

Document classification of Digital Markets Act public consultations using transfer learning with BERT

watermelon's Issues

RNN/CNN

Upload presentation slides pdf to repo

Transfer learning (TL): BERT text classification

• Training
• Hyperparameter tuning
• Validation
• Evaluation metrics
• Visualize results
• Interpretation of results

Video recording of tutorial notebook

Other BERT variants

Combine models into one tutorial notebook (with explanations)

Main

• Memo (300 words)
• Data description and preprocessing
• Baseline models (logit, XGBoost)
• CNN/RNN
• BERT and other variants
• Results and discussion
• Limitations
• Next steps
• References

Others

• Clean libraries & system requirements - ensure no duplicates
• Standardize train & test split
• Re-run notebook in new environment to test

Baseline model/s: logistic regression and/or XGboost with embeddings and tf-idf weights

Fine tuning base BERT model

Presentation slides due 6 Dec

Outline

1. Main problem – JANINE
   •motivation: classifying texts with relevance on EU policymaking (e.g., Digital Markets Act)
   •classifying "unstructured"/"varying structured" pdfs with NLP

2. Method: Using machine learning/deep learning methods for unstructured text classification
   (vs simple rule-based systems) – LORENZO & JANINE
     A. Unstructured text: parsing, data cleaning, language detection, tokenizing
     B. Embeddings, gradient boosting (briefly as the “simpler” method)
     C. Transfer learning with BERT
     BERT: freezing initial layers, fine-tuning last layers

3. Initial/Expected Results – LUKAS & ADELLE
   •model metrics
   •comparing performance (BERT vs. others)
   •Visualization

4. Recommendations – LUKAS
   •how to integrate results with policy process
   •feedback from different stakeholders
   •recommend technical and policy stuff

Parsing unstructured pdfs to tidy text data

Website: https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12416-Single-Market-new-complementary-tool-to-strengthen-competition-enforcement/public-consultation_en

• Parse unstructured pdf to txt files/dataframe
• Match parsed texts to labels from Excel data
• Combine answers from survey open-ended questions for responses w/o pdf submission
• Tag language to exclude non-English pdfs and answers
• Extract paragraphs as entries
• Clean how to split documents to paragraphs with n number of words
• Keep relevant information (body) based on number of words per line

Update readme