Comments (4)
Hi there,
this is expected because the GPT model only supports context lengths of up to 1024 tokens. I will add an info box to chapter 6 to make it more explicit. As a workaround, you can set max_length=1024
in the data loader.
from llms-from-scratch.
Yes, you can feed the input data in chunks. There are also alternative methods like LongLora to extend contexts windows. But these are techniques that are currently out of scope for this book. I am planning some day to also reimplement Llama 3 with a 8,192 tokens length, but this will probably be down the road in a few months. Also, not all readers have the necessary GPU to support that bit of a context length for finetuning.
So, I think the best way here would be to use models that natively support larger context lengths, like the Mixtral-8x7B model you suggested, if your hardware (GPU RAM) supports it,
from llms-from-scratch.
Added a warning about it via #176 . Thanks for reporting that.
from llms-from-scratch.
Hi,
GPT model only supports up to 1024 tokens, So we can use up to 1024 tokens only, Can't we use/implement for more than 1024 tokens using something like chunks/step-by-step feeding on the data loader side?
If not then, I need to explore fine-tuning for other models which support more than 10k tokens like Mixtral-8x7B...
from llms-from-scratch.
Related Issues (20)
- 2.7 Creating token embeddings - Text in bold print HOT 2
- Inconsistent vocabulary sizes mentioned in the same section (2.3 Converting tokens into token IDs) HOT 1
- Ch-07 instruction tuning HOT 3
- The GPT 2 Model url link needs to be corrected HOT 3
- Duplicated line in the Listing 2.6 (2.6 Data sampling with a sliding window) HOT 1
- Detaching a Tensor from the graph doesn't have effect (Embedding Layers and Linear Layers) HOT 3
- (Enhancement) Applying mask to attention in one operation (3.5 Hiding future words with causal attention) HOT 2
- Inconsistency in cases for applying dropout in the book and in the notebook (3.5.2 Masking additional attention weights with dropout) HOT 3
- Question of training on conversational datasets HOT 1
- generate_model_scores function bug? HOT 3
- lower validation&train loss with poorer performance
- Several typos/questions (Sections 4.1-4.2) HOT 1
- Edge case: Gradient accumulation HOT 3
- Typo of figure labeling? (5.1.2 Calculating the text generation loss) HOT 1
- An unusual link in the pdf version (5.1.2 Calculating the text generation loss) HOT 1
- Different figures in the book and jupyter notebook for Figure 5.9 (5.1.3 Calculating the training and validation set losses) HOT 1
- Inconsistencies in the book and Jupyter notebook (5.2 Training an LLM) HOT 4
- Missing word in a sentence (5.3.1 Temperature scaling) HOT 4
- Output and code cells are in the wrong order (5.3.1 Temperature scaling) HOT 1
- Incorrect formatting of the text as code (5.3.1 Temperature scaling) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llms-from-scratch.