This program implements data pre-process and preparation, fine-tuning process of T5 model, inference process of T5 model. We used the pre-trained T5 model published by hugging-face because of the shortage of GPU resources.
You can run scripts in following order to fine-tune a T5:
- python inference_sample.py
- python fine_tune.py
numpy, pip install numpy
pytorch, see https://pytorch.org/get-started/locally/
transformers, pip install transformers / conda install -c huggingface transformers
tensorboardX, pip install tensorboardX
Test the inference function of pre-trained T5 model and save it into pre-train model directory.
You need to run this script to get the pre-trained T5 model.
Fine-tune T5 model using ambignq dataset. Here I set the epochs as 20, learning rate variances from 3e-4 to 1e-5 using a Cosine function. Accumulation gradient is used here to overcome the short of GPU resources.
I built my dataset here.
This script is used to test the data processing approach, you do not need to run it when training T5.