Comments (10)
Hi @owmohamm and sorry for the late reply.
AFAIK weight decay does not have significant regularization effect in the modern networks that have normalization layers (e.g. LayerNorm) or standardized weights. I can recommend to look into this two references to understand this effect:
- L2 Regularization versus Batch and Weight Normalization by Laarhoven: https://arxiv.org/abs/1706.05350.
- An Exponential Learning Rate Schedule for Deep Learning by Li and Arora - https://arxiv.org/abs/1910.07454.
Hope it helps.
from vision_transformer.
Hello!
- The checkpoints published in this repository were pretrained for 90 epochs.
- Same data augmentation and model initialization
- We use sigmoid cross entropy (see e.g. tf.nn.softmax_cross_entropy_with_logits) - you could implement this as follows:
def sigmoid_xent(*, logits, labels):
log_p = jax.nn.log_sigmoid(logits)
log_not_p = jax.nn.log_sigmoid(-logits)
nll = -jnp.sum(labels * log_p + (1. - labels) * log_not_p, axis=-1)
return jnp.mean(nll)
Best!
from vision_transformer.
Thanks for the info!!
Just to confirm, you were referring to tf.nn.sigmoid_cross_entropy_with_logits and not tf.nn.softmax_cross_entropy_with_logits, right?
In regards to the weight decay, what value was used and was it applied to full model or were the bias and norm terms excluded? When I try to pretrain the model the bias terms tend to grow much larger than those in the released weights
from vision_transformer.
You're 100% right, that was an unfortunate typo on my side!
As for weight decay, we used weight_decay=0.1
and we apply the following function to all kernel
weights:
def decay_weight_fn(w, lr, weight_decay):
return (1.0 - lr * decay) * w if decay else w
As for bias terms, we initialized the head biases to -10 :
params["head"]["bias"] = jnp.full_like(params["head"]["bias"], -10.)
HTH
from vision_transformer.
Thanks for your help!!, that resolves any questions I had.
from vision_transformer.
Correction : Actually, for the relevant pretraining run, we used flax.optim.Adam(weight_decay=0.1)
instead of the manual version that I described above (so this would indeed apply to the weight decay to all the weights).
from vision_transformer.
That's interesting, was this done to combat overfitting?
In my experiments (not this architecture but in general) I've noticed that adding weight decay for the bias terms usually gives an inferior model, but in this case i.e. pretraining ViT-L/16 on imagenet-21k the model tends to overfit quite easily.
from vision_transformer.
Hello, everyone, @owmohamm @andsteing @akolesnikoff . I meet some same questions when pretrain the ViT-B/16 architecture on imagenet-21k:
- I also find that the bias in our weight in larger than the released weights.
As you says, do I need to apply the weight decay to all weight items including bias and norm? Then, should all the biases be initialized to -10 or is it just the bias of the final classifier?
params["head"]["bias"] = jnp.full_like(params["head"]["bias"], -10.)
- About the multilabel.
Where should I download multilabel information? Or does it mean that the same image in the 21k dataset of 14M already belongs to different classes? I'm not sure. - When I use softmax cross entropy loss, the value of loss after pre-training is about 1.75. However, when using sigmoid cross-entropy loss, the 10th epoch is only about 0.0003. I am not sure if something is wrong.
I have been troubled by these for a long time and hope to get help, thank u very mch!
from vision_transformer.
To your questions:
- We used
flax.optim.Adam(weight_decay=0.1)
during pre-training for these models. Only final classifier head biases were initialized. - As for ImageNet21k dataset multi-labeledness, see e.g. ImageNet-21K Pretraining for the Masses. Note that JFT is also multi-label, but ImageNet-1k is not (note that we also use sigmoid loss for ImageNet-1k though, see #62)
- With the implementation above you should get loss curves that resemble what I posted in #62
from vision_transformer.
Ok, thank u very much!
from vision_transformer.
Related Issues (20)
- Fine-Tuning HOT 3
- ERROR: Could not find a version that satisfies the requirement tensorflow_text (from vit-jax) (from versions: none) ERROR: No matching distribution found for tensorflow_text HOT 1
- If the weights of vit-base trained with dropout available? HOT 1
- How do I download the vit_base_patch8_384.pth
- Package versions' confliction [Windows] HOT 2
- Question about commercial usage of LiT model checkpoints
- Shouldn't accumulate_gradient pass rng_key?
- GPU Requirement to use vision transformer HOT 1
- flax.errors.CallCompactUnboundModuleError
- ViT
- can export the pretrained model to onnx or pytorch? HOT 2
- Vision transformer
- Import error in Jax (colab) HOT 3
- fine-tune imagenet21k_ViT-B_16.npz with pre_logits? HOT 2
- KeyError: 'embedding/kernel is not a file in the archive' HOT 1
- Hyperparameter issues HOT 1
- All attempts to get a Google authentication bearer token failed, returning an empty token.
- Where is ViT-22B?
- Is it possible to provide the checkpoint?
- How can I use imagenet21k+imagenet2012/ViT-B_16.npz to inference
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vision_transformer.