yuanbit/finbert-qa-notebooks

Notebooks for fine-tuning a BERT model and training a LSTM model for financial QA

Hi there, I am Bithiah (BITH-ee-ah) 👋

Here are some of my Machine Learning and Data Science projects:

• 💰 BERT-powered Financial Question Answering System
  • (NLP, Deep Learning, Search)
• ☁️ Production-ready cloud-native Question Answering System
  • (NLP, Deep Learning, Search, MLOps)
• 😄 Sentiment Analysis of Tweets
  • (NLP, Deep Learning)
• 📖 Scientific Literature Recommender System
  • (NLP, Machine Learning, Personalization, Distributed Computing)
• 🖥️ Product Recommender System for Amazon Electronic Products
  • (Machine Learning, Personalization)
• 👩‍🦰 Facial Recoginiton and Clustering
  • (Computer Vision, Deep Learning, Machine Learning)
• 🤖 Realtime Chat App with Sentiment Analysis
  • (Full-Stack Development, NLP, Machine Learning)

Feel free to reach out to me about NLP, Search, Recommender Systems, Design, and Real-Time Art

• 💬 Message me on Linkedin
• 📝 Follow my Data Science and Design blogs on Medium
• 🎨 Check out my Designs & Art here

finbert-qa-notebooks's Issues

Cannot reproduce the numbers on the paper with this Notebook

Hey! Thanks for open-sourcing this amazing project!
Just a quick question, I folllowed strictly with this notebook FinBERT_QA.ipynb and cannot reproduce the numbers reported on the paper.
More specificly, with using bert-qa as the starting point, after

config = {'bert_model_name': bert-qa,
      'max_seq_len': 512,
      'batch_size': 16,
      'learning_rate': 3e-6,
      'weight_decay': 0.01,
      'n_epochs': 3,
      'num_warmup_steps': 10000}

The results I had was
Epoch 2:

Train Loss: 0.069 | Train Accuracy: 98.39%
Validation Loss: 0.089 | Validation Accuracy: 98.09%
Average nDCG@10 for 333 queries: 0.476
MRR@10 for 333 queries: 0.442
Average Precision@1 for 333 queries: 0.381

Epoch 3:

Train Loss: 0.055 | Train Accuracy: 98.75%
Validation Loss: 0.097 | Validation Accuracy: 98.1%

Average nDCG@10 for 333 queries: 0.471
MRR@10 for 333 queries: 0.427
Average Precision@1 for 333 queries: 0.357

but the reported nDCG@10 should be 0.481.

Any ideas/suggestions? Thank you!