google-research / syn-rep-learn Goto Github PK
View Code? Open in Web Editor NEWLearning from synthetic data - code and models
License: Apache License 2.0
Learning from synthetic data - code and models
License: Apache License 2.0
Hi,
Thanks for the contribution and updated code on Supervised Contrastive Learning.
My question is related to this part of the loss:
https://github.com/google-research/syn-rep-learn/blob/main/StableRep/models/losses.py#L79
local_batch_size = feats.size(0)
...
# Create label matrix, since in our specific case the
# label matrix in side each batch is the same, so
# we can just create it once and reuse it. For other
# cases, user need to compute it for each batch
if local_batch_size != self.last_local_batch_size:
etc....
My understanding is that, for a given batch in distributed setting, the label tensor (after all_gather) will be identical across all the gpus so no need to compute it multiple times. Just once per batch is enough.
My question is then on the condition local_batch_size != self.last_local_batch_size:
: Why the check is done on the batch size and not on the tensors values, isn't the batch size pretty much the same during training ?
Thank you !
Hi, thanks for your great work. As for SynCLR, you first generate prompts, then generate the image based on the prompt. There are examples for some datasets in Folder SYN_TEXT, such as ImageNet, Food101. Are there examples for other datasets, such as STL-10 and CIFAR-100?
I currently run the linear probing evaluation for Imagenet distributed on 2 NVIDIA GTX 1080. It currently takes 4 hours for one epoch with 90 epochs to do. Which durations do you have experienced while your experiments?
Line 40 environment.yml
should be - pillow==9.4.0
not - pillow=9.4.0
Dear Authors,
I appreciate for your exceptional contributions to the field, particularly regarding your works on syn-rep-learn.
From your "stablerep" paper, particularly the scaling effects of linear probing shown in Figure 6, I discovered these do not seem to be consistent in ImageNet zero-shot classification using provided CLIP-based checkpoints.
(CLIP shows better accuracy than StableRep++ for larger pretraining samples)
Below, I've included the code I employed to generate these results, adapted from your repositories ("StableRep" and "Scaling").
The command and its corresponding output are as follows:
# model checkpoints are automatically downloaded from dropbox
# CLIP ViT-B-16
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion3m:CLIP_vitb16
[laion3m:CLIP_vitb16] ImageNet zero-shot accuracy: 21.83
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion10m:CLIP_vitb16
[laion10m:CLIP_vitb16] ImageNet zero-shot accuracy: 40.732
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion20m:CLIP_vitb16
[laion20m:CLIP_vitb16] ImageNet zero-shot accuracy: 45.754
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion50m:CLIP_vitb16
[laion50m:CLIP_vitb16] ImageNet zero-shot accuracy: 49.564
# StableRep-pp
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion3m:StableRep-pp_vitb16
[laion3m:StableRep-pp_vitb16] ImageNet zero-shot accuracy: 31.71
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion10m:StableRep-pp_vitb16
[laion10m:StableRep-pp_vitb16] ImageNet zero-shot accuracy: 40.86
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion20m:StableRep-pp_vitb16
[laion20m:StableRep-pp_vitb16] ImageNet zero-shot accuracy: 43.614
python eval_imagenet_clip.py --data-path /home/appuser/datasets/imagenet --model laion50m:StableRep-pp_vitb16
[laion50m:StableRep-pp_vitb16] ImageNet zero-shot accuracy: 44.886
Given these observations, I am curious to understand whether there might be an oversight on my part or if this phenomenon reflects the model's behavior.
Could you possibly give any idea on this discrepancy?
Best regards,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.