Comments (3)
I've found what seems to be a solution, that enables Deepspeed inference on my trained gpt-neo model.
The issue seems to be that the trained gpt-neo has this line in its config.json
:
"use_cache": false,
I can see that the original model -- "EleutherAI/gpt-neo-2.7B", this value is set to true
https://huggingface.co/EleutherAI/gpt-neo-2.7B/blob/main/config.json#L77
So it seems that if, after training, I manually modify config.json
, in the folder of my trained model, and set this value to true
, then Deepspeed inference works as expected. I get about a 25% speedup on inference on an rtx 2070 super.
Closing this.
from finetune-gpt2xl.
After further testing, I am finding that inference with Deepspeed DOES offer the promised ~2x speedup, and this code ...
DOES work, but only if I use the not-finetuned, original model, from Huggingface -- "EleutherAI/gpt-neo-2.7B"..
so:
model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B").half().to("cuda")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")
works well, when prefaced by...
deepspeed.init_inference(
model, mp_size=1,
dtype=torch.half,
replace_method='auto'
)
But when I used my custom finetuned model, which I finetuned according to this guide... https://github.com/Xirider/finetune-gpt2xl/blob/main/README.md#finetune-gpt-neo-27-billion-parameters
....
I get the
IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)
error when I try inference with deepspeed.
Again, inference on my finetuned model, WITHOUT deepspeed, works well.
So in short it seems that something about the finetune process shown here produces a finetuned model that doesn't seem to work with Deepspeed inference, even through the original (not-finetuned) model can be shown to work will deepspeed inference.
from finetune-gpt2xl.
Thanks for the tip, it works!
from finetune-gpt2xl.
Related Issues (20)
- Freezing at "Using /home/user/.cache/torch_extensions as PyTorch extensions root..." HOT 4
- Ideal number of epochs? Number of examples meaning? HOT 1
- Can't change BOS token or EOS token for GPT Neo HOT 3
- Crashes with new Transformers version HOT 1
- New issue with Pandas
- AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' HOT 8
- Feeding the model separate examples instead of one continuous block of text HOT 1
- Out of memory with RTX3090 HOT 4
- subprocess.CalledProcessError: HOT 1
- Using [text,labels] instead of just [text] in Datasets HOT 1
- fine tuning GPT-J 6B? HOT 1
- IndexError: index out of bounds HOT 1
- TypeError: __init__() got an unexpected keyword argument 'no_args_is_help'
- Training on a larger dataset fails due to memory issues on faster GPUs HOT 3
- Multiple entries csv HOT 1
- Unable to proceed, no GPU resources available HOT 3
- Errors while trying to train with two GPUs HOT 4
- TypeError: unsupported operand type(s) for -: 'float' and 'str' on AWS g4dn.12xlarge
- Resume from checkpoint HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from finetune-gpt2xl.