Comments (6)
Thanks for reporting! And I could swear I wrote a response to this earlier ... not sure what happened to it. In any case, I think this could be a PyTorch bug.
If you change it to
checkpoint = torch.load("model_and_optimizer.pth")
model = GPTModel(GPT_CONFIG_124M)
model.load_state_dict(checkpoint["model_state_dict"])
model.to(device) # NEW
optimizer = torch.optim.AdamW(model.parameters(), lr=0.0004, weight_decay=0.1)
optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
model.train();
could you check whether it solves it for you? It did the trick for me.
from llms-from-scratch.
Just tested, I have the same issue (see ch05/01_main-chapter-code/exercise-solutions.ipynb)
As the model will already be moved on GPU (when there is one that is CUDA-supported), you could also move the states of the optimizer to the GPU as well:
# Iterate over all states in the optimizer
for state in optimizer.state.values():
# Iterate over all key-value pairs in the current state
for param_name, param_value in state.items():
# Check if the value is a torch.Tensor
if isinstance(param_value, torch.Tensor):
# Move the tensor to the specified device (e.g., GPU or CPU)
state[param_name] = param_value.to(device)
from llms-from-scratch.
At least for me, it worked. And I agree, seems like a bug.
Two notes on the exercises:
- There is still a duplicated
model.to(device)
- The GPT2-XL model might be too heavy, it used all of my RAM (32GB) and froze my computer for some seconds.
from llms-from-scratch.
At least for me, it worked. And I agree, seems like a bug.
Two notes on the exercises:
- There is still a duplicated
model.to(device)
- The GPT2-XL model might be too heavy, it used all of my RAM (32GB) and froze my computer for some seconds.
tbh,i did gpt2-3 pre-trained model from scratch recently in another version,compared with different GPU clusters,24G VRAM like RTX4090 or A10G will be OOM when i added the hellaswag evaluation part.IMHO,i did some test, if you trained with the HuggingFaceFW/fineweb-edu datasets on HF, i mean the minimum edu_fineweb10B, then you need to have at least around 48G VRAM like the L20 8xGPU, but it will be slower than 8xGPU A100, around half FLOPS computation disadvantages.Anyway nowadays i thought 8x 80G A100 is a standard requirements for LLM pre_training model
from llms-from-scratch.
There is still a duplicated model.to(device)
Thanks for the note, I'll clean that up. But it should not hurt, PyTorch is smart enough to detect if the model is already on the target device and then skips this op.
The GPT2-XL model might be too heavy, it used all of my RAM (32GB) and froze my computer for some seconds.
Yeah. Pretraining can be expensive. We trained TinyLlama on 64xA100s 😅. Besides lowering context length and batch size, you can try to train in lower precision if your GPU supports it:
model.to(device=device, dtype=torch.bfloat16)
I can also add some FSDP code some time in the future
Btw does the added model.to(device)
call fix things for you too?
from llms-from-scratch.
I think this issue is resolved now. But please feel free to reopen & respond if you are still having issues. Thanks again for reporting!
from llms-from-scratch.
Related Issues (20)
- Ch-07 instruction tuning HOT 3
- The GPT 2 Model url link needs to be corrected HOT 3
- Duplicated line in the Listing 2.6 (2.6 Data sampling with a sliding window) HOT 1
- Detaching a Tensor from the graph doesn't have effect (Embedding Layers and Linear Layers) HOT 3
- (Enhancement) Applying mask to attention in one operation (3.5 Hiding future words with causal attention) HOT 2
- Inconsistency in cases for applying dropout in the book and in the notebook (3.5.2 Masking additional attention weights with dropout) HOT 3
- Question of training on conversational datasets HOT 1
- generate_model_scores function bug? HOT 3
- lower validation&train loss with poorer performance
- Several typos/questions (Sections 4.1-4.2) HOT 1
- Edge case: Gradient accumulation HOT 3
- Typo of figure labeling? (5.1.2 Calculating the text generation loss) HOT 1
- An unusual link in the pdf version (5.1.2 Calculating the text generation loss) HOT 1
- Different figures in the book and jupyter notebook for Figure 5.9 (5.1.3 Calculating the training and validation set losses) HOT 1
- Inconsistencies in the book and Jupyter notebook (5.2 Training an LLM) HOT 4
- Missing word in a sentence (5.3.1 Temperature scaling) HOT 4
- Output and code cells are in the wrong order (5.3.1 Temperature scaling) HOT 1
- Incorrect formatting of the text as code (5.3.1 Temperature scaling) HOT 3
- Different function names for the same function (5.3.3 Modifying the text generation function) HOT 1
- Duplicated code cell (Exercise 5.3: Deterministic behavior in the decoding functions) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llms-from-scratch.