Throwing error for longer textual data like 9599

[Ameet1992]

Throwing error for longer textual data like 9599

[rasbt]

Hi there,
this is expected because the GPT model only supports context lengths of up to 1024 tokens. I will add an info box to chapter 6 to make it more explicit. As a workaround, you can set max_length=1024 in the data loader.

[rasbt]

Yes, you can feed the input data in chunks. There are also alternative methods like LongLora to extend contexts windows. But these are techniques that are currently out of scope for this book. I am planning some day to also reimplement Llama 3 with a 8,192 tokens length, but this will probably be down the road in a few months. Also, not all readers have the necessary GPU to support that bit of a context length for finetuning.

So, I think the best way here would be to use models that natively support larger context lengths, like the Mixtral-8x7B model you suggested, if your hardware (GPU RAM) supports it,

[rasbt]

Added a warning about it via #176 . Thanks for reporting that.

[Ameet1992]

Hi,
GPT model only supports up to 1024 tokens, So we can use up to 1024 tokens only, Can't we use/implement for more than 1024 tokens using something like chunks/step-by-step feeding on the data loader side?
If not then, I need to explore fine-tuning for other models which support more than 10k tokens like Mixtral-8x7B...