Comments (2)
Hi, I noticed that there maybe some issues when resuming the latest checkpoints. In the training script, only the 'EMA' checkpoint is saved each time and the 'model' checkpoint is not saved. If the running job is interrupted, the 'EMA' checkpoint of recent training steps is loaded to initialize the model.state_dict. I am not sure if it is correct, because generally the 'model' checkpoint should be loaded for model.state_dict. In addition, when to re-run the scrip for resuming training, the 'EMA' is again initialized with random model state_dict. I am wondering if the 'EMA' should be initialized with the 'EMA' checkpoint of recent training steps?
So, I would like to make sure if there are differences between the whole training (without interruption) and resuming the training due to the interruption? Many thanks
Hi, ema
is not initialized randomly; it synchronizes the parameters of the model. Please see here.
from latte.
Hi, I noticed that there maybe some issues when resuming the latest checkpoints. In the training script, only the 'EMA' checkpoint is saved each time and the 'model' checkpoint is not saved. If the running job is interrupted, the 'EMA' checkpoint of recent training steps is loaded to initialize the model.state_dict. I am not sure if it is correct, because generally the 'model' checkpoint should be loaded for model.state_dict. In addition, when to re-run the scrip for resuming training, the 'EMA' is again initialized with random model state_dict. I am wondering if the 'EMA' should be initialized with the 'EMA' checkpoint of recent training steps?
So, I would like to make sure if there are differences between the whole training (without interruption) and resuming the training due to the interruption? Many thanksHi,
ema
is not initialized randomly; it synchronizes the parameters of the model. Please see here.
Sorry, I mean the ema do not load the saved ema checkpoint
resume_from_checkpoint
is a TODO function, which is not perfect yet. So check it carefully if you want to use it.
from latte.
Related Issues (20)
- What is the difference between Latte and ViViT? HOT 2
- RuntimeError: "LayerNormKernelImpl" not implemented for 'Half' HOT 2
- RuntimeError: "slow_conv2d_cpu" not implemented for 'Half' HOT 2
- image_size = [256,512] HOT 4
- CUDA out of memory HOT 6
- Evaluate the FVD? HOT 5
- Some weights of AutoencoderKL were not initialized from the model checkpoint at /path/to/Latte/t2v_required_models/ and are newly initialized because the shapes did not match: HOT 3
- FaceForensics数据集 HOT 3
- No positional embeddings in LatteT2V?
- Is autoregression possible? HOT 3
- Questions about the *0.18215 and /0.18215 operation HOT 2
- About video VAE HOT 2
- 有关时长的问题。 HOT 4
- question on t2v model training HOT 2
- 视频帧率 HOT 2
- Question: model code and design choices HOT 2
- how to place and preprocess these datasets HOT 7
- the code of variant 4 HOT 1
- Question: evaluate the FVD HOT 6
- Error once speed up training HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from latte.