Comments (9)
Thanks for considering the issue and quick fix. This repo is really great and keeps updating it with awesome stuff!
from tez.
it should be fixed in main branch now. could you please confirm?
from tez.
This issue still exists in main branch as well as I did not see any fix for it.
from tez.
hmm... then i might be missing something. could you please share more information/code? i see that the zero grads as fine. or i missed something?
from tez.
I just run the latest tez code and same issue is still happening.
Here, you can see when batch_index = 0, it will first zero_grad and then forward pass will run and it is fine but when batch_index = 1, this condition will not run so forward pass will run without doing zero_grad first as in below snapshot.
So, one solution is to either remove if condition on zero_grad
in _step()
on line # 336 OR remove the self.batch_index == 0
on line # 299 OR you can come with some other fix which you know better.
Hope it helps.
from tez.
it seems like one of us is confused. when batch_index = 1 or any value greater than zero, zero_grad is happening here:
Line 336 in fd2d85e
when batch_index = 0, _step
function does the zero grad. this zero_grad is before the very first forward pass.
from tez.
I tried to log the forward pass and zero_grad (wherever it is written in code) like this,
You can point out if I am printing logs in wrong way.
and you will see logs like this,
ofcourse zero_grad is happening for batch_index = 1 but after forward pass completes for batch_index = 1.
from tez.
Also, I just recap myself about zero_grad and actually it has nothing to do with forward pass instead it must happen before backward pass (apologies for mistyping) so I logged backward pass as well but still same issue as you can see below.
zero_grad before backward pass:
https://discuss.pytorch.org/t/should-we-perform-optimizer-zero-grad-before-we-do-forward-pass-or-after-that/14254
from tez.
ohh! thanks for the code. i got it now and I've fixed it in main branch. many thanks for looking deep into the codebase 🙏🏽
from tez.
Related Issues (20)
- Issue while using Auc metric on imbalanced dataset like melanoma(ValueError: Only one class present in y_true. ROC AUC score is not defined in that case) HOT 2
- Add citation link.
- How do you pass the parameters when using the "reducelrplateon" scheduler?
- Applying metrics after the epoch HOT 2
- logging in text file
- Can I set the device id of GPUs?
- Is it possible to set variable Lr per epoch HOT 3
- how can i use multi gpus for training? HOT 2
- I want this recommender training model please.
- AttributeError: module 'tez' has no attribute 'Model' HOT 2
- 🔥 Require Tez + Apple M1 GPU Support
- ValueError: only one element tensors can be converted to Python scalars
- How can we access the input_ids/attention mask in each train batch loop?
- Saving after training an epoch HOT 3
- Saving validation score
- Getting error while importing enums from tez. HOT 3
- run example code error
- .
- NOTE: This is old Model class and is deprecated. It will no longer be maintained! Please use version > 0.5.1. Its much better and supports multi-gpu training too! HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tez.