Comments (2)
Someone linked you were looking for help, can provide some insights perhaps.
You can cache the latents of your training images ahead of time to disk or sys ram, but the VAE isn't that large either. Caching too much to sys ram can run your sys ram usage up a lot. I purposely do not cache latents in EveryDream because it breaks crop jitter, but if your absolute goal is minimum vram use it makes sense. Caching them to disk is the safest, save as .pt (torch tensors) and read on the fly, maybe a hash dictionary of the filename and tensor. Keep in mind creating the cache ahead will take some time for larger datasets, depends on your goals.
I've found text encoder training makes a big difference when doing normal unfrozen unet training. Not sure if you intend to unfreeze text encoder, but if not, you could cache the embedding outputs of the text encoder as well. tokenize them, encode them, then cache those values same as above. Obviously means you cannot train the text encoder at all.
Extra flag in your zero_grad (optimizer.zero_grad(set_to_none=True)
) will save a small amount of vram assuming you're on a new enough version of Torch.
from controlnet.
HuggingFace Diffusers ControlNet training script https://huggingface.co/docs/diffusers/training/controlnet uses different performance optimizations
min 8GB on Linux with DeepSpeed (which is only available on Linux right now), and 12GB on Windows
from controlnet.
Related Issues (20)
- How to apply Controlnet to OpenAI CLIP?
- Path of trained weights using control net HOT 2
- an error about the "parameterization"
- when will the tile for sdxl come out?
- How can i get safetensors model after training for openpose.
- "replace 50% text prompts with empty strings"?
- What is the meaning of guidance scale and unconditional_guidance_scale in code?
- How do I employ my model to generate images and saving them?
- Comtrolnet as a decensoring engine
- How many patches are divided from 512*512 image for position embedding?
- Question about Figure 3 in the paper HOT 2
- The site cannot be reached issue HOT 2
- pure code? HOT 1
- tool_transfer_control.py
- Feature Request: Mask painting precision assistance
- How to use unet
- a
- How long it takes to test an image HOT 3
- Unable to install gradio=3.40.1 HOT 3
- Training a ControlNet to generate furnished room -> empty room (and vice versa). Improvement plateau...
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from controlnet.