ftarlaci/gpt2sqa

Fine-tuning GPT-2 Small for Question Answering

gpt2sqa's Issues

Instructions for installing packages

Hi! I'm researching this for a college project and I need a bit of help setting this up.
I cloned the repo and created a virtual env in python 3.6, install the requirements.txt and I ran python setup.py build and then python setup.py install
after that I try to run the the training command:
python gpt2_squad.py --output_dir=output/ --train_file=dataset/dataset/train-v2.0.json --do_train --train_batch_size=32 --predict_file=dataset/dataset/dev-v2.0.json --do_predict

but it returns this Error:
Traceback (most recent call last): File "gpt2_squad.py", line 39, in <module> from gpt2sqa.file_utils import PYTORCH_PRETRAINED_GPT2_CACHE, WEIGHTS_NAME, CONFIG_NAME File "/home/domainproxy/NateCusack/QA_Proj/GPT2sQA/gpt2sqa/__init__.py", line 5, in <module> from gpt2sqa.gpt2.gpt2model import GPT2Model File "/home/domainproxy/NateCusack/QA_Proj/GPT2sQA/gpt2sqa/gpt2/gpt2model.py", line 7, in <module> from gpt2_question_answering.gpt2.layer_norm import LayerNorm ModuleNotFoundError: No module named 'gpt2_question_answering'
Sorry if this is a really basic question I'm still pretty new to ML.
Thanks in Advance!
Nathan

Training stops as soon as it logs the start of an Epoch

Created a fork, did some minor changes. Used the following pip installs

#!pip install torch>=0.4.1 tqdm boto3 regex
!pip install tqdm boto3 regex
!pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117

https://github.com/thistleknot/GPT2sQA

Get this as soon as epoch starts
total_missed:0
12/27/2022 20:56:23 - INFO - main - ***** Running training *****
12/27/2022 20:56:23 - INFO - main - Num orig examples = 130319
12/27/2022 20:56:23 - INFO - main - Num split examples = 130334
12/27/2022 20:56:23 - INFO - main - Batch size = 32
12/27/2022 20:56:23 - INFO - main - Num steps = 12216
0% 0/4073 [00:00<?, ?it/s]
Epoch: 0% 0/3 [00:00<?, ?it/s]^C

Can we get ballpark performance for the smallest model?

It would be great if you could share with us some rough results on SQuAD v2.0/v1.1 using the smallest model (117M). Thanks in advance!

System Requirements

Could you please specify the System requirements to run this program like GPU.
Because I have 8GB RAM and 4GB Graphics card and I am facing runtime errors while trying to run this code