First turn on Docker Daemon (open the Docker App)
# This image is updated with the correct packages
docker pull zzweeee/dockerhub:llmft_updated
Optionally, tag the image with:
- First find the image id
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
zzweeee/dockerhub llmft_updated efa258f98de3 ... 15.3GB
- Then tag the image:
$ docker tag zzweeee/dockerhub:llmft_updated llmft:updated
CD into the project directory and run the docker
$ docker run -it --rm --gpus=all --pid=host --ipc=host --user Sheng -v /$PWD:/llmft llmft:updated
source ./scripts/misc/setup.sh
source
is necessary because it saves the env variables in the current shell session
bash scripts/in_context/cola/run_minimal.sh cola 2 facebook/opt-125m 0 12345
Logs will be saved under the logfiles
directory.
- Download a model from MetaICL's checkpoint and save under the project root directory
./model.pt
. - In
run_minimal.sh
, add two more arguments after--model_name_or_path $model_name_or_path \
--model_name_or_path $model_name_or_path \
--config_name gpt2-large \
--tokenizer_name gpt2-large \
Since the MetaICL model are gpt-large, we need to specify its corresponding config_name and tokenizer_name.
3. Run the script while replacing the model argument to model.pt
bash scripts/in_context/cola/run_minimal.sh cola 2 model.pt 0 12345
When using the local gpt2 model, a wrapper is applied to it in models/local_gpt2_wrapper.py
where we only take the
transformer output and add a linear "score" layer that manually projects to vocab_size
dimension. This layer is untrained (while the rest of the model is trained by MetaICL),
which can make the final projection result really bad. We should revise the code to use the decoder output. The current wrapper
is written this way because we referenced the gptj wrapper class. But looking at the opt wrapper class, they use decoder output
which makes more sense.
Saarland University, University of Cambridge, Bar-Ilan University, Allen Institute for Artificial Intelligence, University of Washington
This repository contains code for the paper Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation.
Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out- of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.
This repository allows to finetune (large) decoder-only language models. Currently, the following models and fine-tuning approaches are supported.
Models:
- facebook/opt-125m
- facebook/opt-350m
- facebook/opt-1.3b
- facebook/opt-2.7b
- facebook/opt-6.7b
- facebook/opt-13b
- facebook/opt-30b
- EleutherAI/gpt-neox-20b
- EleutherAI/pythia-410m
- EleutherAI/pythia-1.4b
- EleutherAI/pythia-2.8b
- EleutherAI/pythia-6.9b
- EleutherAI/pythia-12b
Fine-tuning approaches:
- Vanilla fine-tuning with a randomly initialized classification head on top of the pre-trained decoder.
- Pattern-based fine-tuning (PBFT) leveraging the pre-trained language modeling head for classification.
Both of these approaches can be combined with the following paramter-efficient methods:
- BitFit (https://arxiv.org/abs/2106.10199)
- LoRA adapters (https://arxiv.org/abs/2106.09685)
In order to fine-tune (very) large models (>1.3b parameters) we heavily realy on deepspeed. See memory requirements for an estimate of the computantional resources required to fine-tune some of the models listed above.
-
Create docker image:
docker build -f ./docker/Dockerfile \ --build-arg USER_UID=$UID \ --build-arg USER_NAME=$(id -un) \ -t llmft:22.08-py3 .
Depending on your NVIDIA CUDA and NVIDIA driver version you will have to change the FROM nvcr.io/nvidia/pytorch:22.08-py3
line of the Docker file. You can find the correct version here.
-
Create docker container:
docker run -it --rm --gpus=all --pid=host --ipc=host --user <username> \ -v <path/to/llmft>:/llmft \ -v <path/to/datasets>:/datasets \ -v <path/to/logfiles>:/logfiles \ -v /<path/to/.cache>:/cache \ llmft:22.08-py3
Make sure to replace <username>
, </path/to/llmft>
, etc.
We make use of deepspeed ZeRO-3 to finetune large models. Below are estimates of the memory requirements needed to train all paramters of the largest OPT models. We assume that you have access to a machine with at least 4 GPUs.
Also, keep in mind that these estimates consider only the model and optimizer paramters. Batch size and sequence length will also have an impact on the memory consumption during fine-tuning and inference.
Estimated memory needed for params, optim states and gradients for a:
- HW: Setup with 1 node, 4 GPUs per node.
- SW: Model with 6658M total params, 205M largest layer params.
per CPU | per GPU | Options |
---|---|---|
167.43GB | 0.77GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1 |
167.43GB | 0.77GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0 |
148.83GB | 3.87GB | offload_param=none, offload_optimizer=cpu , zero_init=1 |
148.83GB | 3.87GB | offload_param=none, offload_optimizer=cpu , zero_init=0 |
4.60GB | 28.67GB | offload_param=none, offload_optimizer=none, zero_init=1 |
148.83GB | 28.67GB | offload_param=none, offload_optimizer=none, zero_init=0 |
Estimated memory needed for params, optim states and gradients for a:
- HW: Setup with 1 node, 8 GPUs per node.
- SW: Model with 6658M total params, 205M largest layer params.
per CPU | per GPU | Options |
---|---|---|
167.43GB | 0.77GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1 |
297.66GB | 0.77GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0 |
148.83GB | 2.32GB | offload_param=none, offload_optimizer=cpu , zero_init=1 |
297.66GB | 2.32GB | offload_param=none, offload_optimizer=cpu , zero_init=0 |
9.21GB | 14.72GB | offload_param=none, offload_optimizer=none, zero_init=1 |
297.66GB | 14.72GB | offload_param=none, offload_optimizer=none, zero_init=0 |
Estimated memory needed for params, optim states and gradients for a:
- HW: Setup with 1 node, 4 GPUs per node.
- SW: Model with 12853M total params, 257M largest layer params.
per CPU | per GPU | Options |
---|---|---|
323.21GB | 0.96GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1 |
323.21GB | 0.96GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0 |
287.30GB | 6.94GB | offload_param=none, offload_optimizer=cpu , zero_init=1 |
287.30GB | 6.94GB | offload_param=none, offload_optimizer=cpu , zero_init=0 |
5.75GB | 54.83GB | offload_param=none, offload_optimizer=none, zero_init=1 |
287.30GB | 54.83GB | offload_param=none, offload_optimizer=none, zero_init=0 |
Estimated memory needed for params, optim states and gradients for a:
- HW: Setup with 1 node, 8 GPUs per node.
- SW: Model with 12853M total params, 257M largest layer params.
per CPU | per GPU | Options |
---|---|---|
323.21GB | 0.96GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1 |
574.60GB | 0.96GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0 |
287.30GB | 3.95GB | offload_param=none, offload_optimizer=cpu , zero_init=1 |
574.60GB | 3.95GB | offload_param=none, offload_optimizer=cpu , zero_init=0 |
11.51GB | 27.89GB | offload_param=none, offload_optimizer=none, zero_init=1 |
574.60GB | 27.89GB | offload_param=none, offload_optimizer=none, zero_init=0 |
Estimated memory needed for params, optim states and gradients for a:
- HW: Setup with 1 node, 4 GPUs per node.
- SW: Model with 29974M total params, 360M largest layer params.
per CPU | per GPU | Options |
---|---|---|
753.73GB | 1.34GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1 |
753.73GB | 1.34GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0 |
669.98GB | 15.30GB | offload_param=none, offload_optimizer=cpu , zero_init=1 |
669.98GB | 15.30GB | offload_param=none, offload_optimizer=cpu , zero_init=0 |
8.05GB | 126.96GB | offload_param=none, offload_optimizer=none, zero_init=1 |
669.98GB | 126.96GB | offload_param=none, offload_optimizer=none, zero_init=0 |
Estimated memory needed for params, optim states and gradients for a:
- HW: Setup with 1 node, 8 GPUs per node.
- SW: Model with 29974M total params, 360M largest layer params.
per CPU | per GPU | Options |
---|---|---|
753.73GB | 1.34GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1 |
1339.97GB | 1.34GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0 |
669.98GB | 8.32GB | offload_param=none, offload_optimizer=cpu , zero_init=1 |
1339.97GB | 8.32GB | offload_param=none, offload_optimizer=cpu , zero_init=0 |
16.11GB | 64.15GB | offload_param=none, offload_optimizer=none, zero_init=1 |
1339.97GB | 64.15GB | offload_param=none, offload_optimizer=none, zero_init=0 |
See experiments/README.md for fine-tuning instructions.