This sample show you how to
- Train BERT, using huggingface on Amazon Sagemaker using Spot instances. Spot instances allow you to lower training costs.
- Use multi-gpu training where the instance has multiple gpus
- Implement Sagemaker checkpointing , so when a spot instance terminates, you can resume training from the checkpoint
- Use gradient accumulation to train with normal batch sizes.
- Deploy the BERT model for inference.
To get started, use the notebook BertTextClassification.ipynb
We use the Dbpedia ontology dataset, for more details, see https://wiki.dbpedia.org/services-resources/dbpedia-data-set-2014#2
In order to customise this sample, for your own dataset, perform the following steps
- Create a dataset class, that implements the PyTorch Dataset abstract class, see dbpedia_dataset.py as an example implementation.
- Create a label mapper class, that implements abstract class LabelMapperBase, to maps string labels to zero indexed integer labels. See an example implementation dbpedia_dataset_label_mapper.py.
- Replace the use of classes
DbpediaDataset
andDbpediaLabelMapper
in builder.py with your own custom dataset and label mapper class
-
Install python 3.7.4
-
Set up requirements.
pip install -r tests/requirements.txt
-
Verify set up
export PYTHONPATH=./src pytest
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.