Comments (15)
Hi Kaiyang, I think with a slight modification, the model could run on ImageNet if using more than one graphic cards.
For CoOp, just change the code on line 257 of coopy.py from "self.model = nn.DataParallel(self.model)" to "self.model.text_encoder = nn.DataParallel(self.model.text_encoder)". I tested it on 1-shot ImageNet with 4 graphic cards, and found that each card only consumed around 8GB memory. Compared with my previous experiments on a single card, the accuracy even increases a little bit :)
The image_encoder does not need to be run parallelly at least for CoOp because it does not require backward propagation and only has a small batch size.
reduce the batch size thenjust realize mainly the text encoder causes the problem :(
hmm, it's a tough question then
ideally you would need a GPU with 32GB memory
for 16 GB GPUs you might need to modify the source code, somehow (no idea what to do, yet)
from coop.
reduce the batch size then
just realize mainly the text encoder causes the problem :(
hmm, it's a tough question then
ideally you would need a GPU with 32GB memory
for 16 GB GPUs you might need to modify the source code, somehow (no idea what to do, yet)
from coop.
Yes, it exceeds 16GB even using batch-size=1
. Unfortunately, I do not have a GPU with 32GB memory 😂
Maybe I should think other ways to reduce the memory consumption.
from coop.
@KaiyangZhou 麻烦问一下论文中res50在imagenet上的实验,需要几张卡,memory需要多大?一般训练多久?
from coop.
- 1 gpu card
- 32gb memory
- few hours
from coop.
@KaiyangZhou
Hi, I have encountered the same problem.
Can you kindly provide the checkpoints of CoOp on ImageNet?
from coop.
The pre-trained weights have just been released. Please see the readme file
from coop.
Thank you for sharing the code,
please ask, is it possible for one RTX3090 24Gb?
from coop.
is it possible for one RTX3090 24Gb?
basically the memory consumption increases with the number of classes (for prompt learning methods)
24gb might be sufficient for most datasets except imagenet
from coop.
I had to say the way using DataParallel on CustomCLIP cannot save GPU memory on each card at all...
from coop.
No. DataParallel won't help.
The problem for imagenet is that the 1,000 classes would create a huge memory consumption for the text encoder. So for smaller datasets with fewer classes, the problem is gone. And the tricky thing is, you can't split the classes into two gpus because in doing so the attention in the transformer model won't work properly. Hmm ...
from coop.
No. DataParallel won't help.
The problem for imagenet is that the 1,000 classes would create a huge memory consumption for the text encoder. So for smaller datasets with fewer classes, the problem is gone. And the tricky thing is, you can't split the classes into two gpus because in doing so the attention in the transformer model won't work properly. Hmm ...
In fact, DataParallel could save memory if used correctly. To be short, it is not the class tensors in the last layer themselves caused the problem, but the intermediate tensors, gradients and the computation graph.
If the code is modified as mentioned above, all the intermediate tensors and most of the gradients could be scattered to different GPUs. And finally, when the model on each card get the final text features, we can collect them to a single card then.
from coop.
Emm, I am not sure whether you noticed. I indeed have run experiments and verified the method's effectiveness...
I tested it on 1-shot ImageNet with 4 graphic cards, and found that each card only consumed around 8GB memory.
from coop.
No. DataParallel won't help.
The problem for imagenet is that the 1,000 classes would create a huge memory consumption for the text encoder. So for smaller datasets with fewer classes, the problem is gone. And the tricky thing is, you can't split the classes into two gpus because in doing so the attention in the transformer model won't work properly. Hmm ...
I do not understand why the attention in the transformer model would have problems. I think the attention mechanism only work within each sample, and there is no attention between samples in a batch.
from coop.
Oh, my bad. I thought you were talking about another approach. The attention thing doesn't matter then.
from coop.
Related Issues (20)
- RuntimeError: "slow_conv2d_cpu" not implemented for 'Half' HOT 3
- Why the test read on imagnet is empty HOT 3
- Stanford Cars dataset download HOT 2
- cifar100 dataset
- Question regarding the number of runs
- How to visualize classification results? HOT 1
- CoOp on cifar-100
- Question for the code x = x[torch.arange(x.shape[0]), tokenized_prompts.argmax(dim=-1)] @ self.text_projection
- error when running coop.
- `lock.acquire()` and cannot exit training
- CSC (class-specific context) CoOp/CoCoOp in "base-new" and "cross-dataset".
- Base-to-novel generalization
- I would like to ask how long is the experimental period for coop training on imagenet?
- Running zero shot for new classes
- FloatingPointError: Loss is infinite or NaN HOT 1
- About PromptLearner
- train.py --root argument error
- 论文中他、
- t中的V是如何学习得到的
- Using BLIP instead of CLIP
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from coop.