Gpt-neo inference with Deepspeed: IndexError: Dimension out of range

[xirider]

Gpt-neo inference with Deepspeed: IndexError: Dimension out of range

[kingpalethe]

I've found what seems to be a solution, that enables Deepspeed inference on my trained gpt-neo model.

The issue seems to be that the trained gpt-neo has this line in its config.json:

"use_cache": false,

I can see that the original model -- "EleutherAI/gpt-neo-2.7B", this value is set to true
https://huggingface.co/EleutherAI/gpt-neo-2.7B/blob/main/config.json#L77

So it seems that if, after training, I manually modify config.json, in the folder of my trained model, and set this value to true, then Deepspeed inference works as expected. I get about a 25% speedup on inference on an rtx 2070 super.

Closing this.

[kingpalethe]

After further testing, I am finding that inference with Deepspeed DOES offer the promised ~2x speedup, and this code ...

https://github.com/Xirider/finetune-gpt2xl/blob/main/README.md#generate-text-with-a-gpt-neo-27-billion-parameters-model

DOES work, but only if I use the not-finetuned, original model, from Huggingface -- "EleutherAI/gpt-neo-2.7B"..

so:

model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B").half().to("cuda")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")

works well, when prefaced by...

    deepspeed.init_inference(
        model, mp_size=1, 
        dtype=torch.half, 
        replace_method='auto'
        )

But when I used my custom finetuned model, which I finetuned according to this guide... https://github.com/Xirider/finetune-gpt2xl/blob/main/README.md#finetune-gpt-neo-27-billion-parameters
....
I get the

IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

error when I try inference with deepspeed.

Again, inference on my finetuned model, WITHOUT deepspeed, works well.

So in short it seems that something about the finetune process shown here produces a finetuned model that doesn't seem to work with Deepspeed inference, even through the original (not-finetuned) model can be shown to work will deepspeed inference.

[martingajek]

Thanks for the tip, it works!