Comments (2)
KaimingHe
commented on June 2, 2024
12
A batch size of 1024 should work well: we have verified this for ViT-L. We have not tried 256.
As we can use accumulate gradient in MAE (thanks to the lack of BN or InfoNCE etc.), it is safer to keep the effective batch size as 4096 so you don't need to tweak other hyper-parameters when the batch size changes.
from mae.
HobbitLong
commented on June 2, 2024
Cool, thanks for sharing!
from mae.
Related Issues (20)
- Ask for segmentation finetune code
- Confusion in The Loss Function Implementation.
- param_groups_lrd for layer decay HOT 1
- Loss is considerably worse on custom data set with different mean and standard deviation HOT 2
- Error in loading pretrained weight for 'mae_vit_base_patch16' HOT 2
- About the gan-loss HOT 2
- patchify and unpatchify HOT 1
- I found both LLAMA and MAE used smaller beta2 in ADAMW optimizer during pre-training. Is that any intuition behind such setting? HOT 1
- How to obtain the reconstructed image for inference and masked
- model.fc_norm is not trained in linear probing
- visualization attention map.
- Could you provide the pretrained checkpoints of both encoder and decoder in MAE? HOT 2
- Is the training procedure result normal? Masked regions do not improve and appear to be random noise. HOT 2
- Two different checkpoints for each ViT type HOT 5
- Code: Compatible to any channels for function patchify and unpatchify HOT 2
- collab notebook error HOT 2
- How to obtain the complete reconstructed image?
- Can run interactive visualization demo with GPU?
- 训练的代码用最新的timm跑不通
- Reconstruction using normalized pixel values to get unnormalized pixel values?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mae.