question about standardization

[liuzhuang13]

question about standardization

[liuzhuang13]

Hi @yuffon I think we just subtract the training means and stds in all testing, this is standard in machine learning.

[yuffon]

Hi @yuffon I think we just subtract the training means and stds in all testing, this is standard in machine learning.

Yes, this is common in ML. What if the input image is not from the 10 categories in cifar10 dataset?
If I want to do transfer learning or other tasks, how should I preprocess the data?
I think it is more convenient with per image standardization.
So I replaced the whole-batch standardization with per image standardization, the acc doesn't converge, untill that I add an BN layer without scale and shift variables at the start point of the densenet. I have not checked the final accuracy results in different scenarios.

[liuzhuang13]

I don't think one should preprocess the images by its own mean and stds, that causes different input to be changed by different amounts.

CIFAR is not a good dataset to transfer from, I think if you do transfer learning it's better to start with ImageNet models. In this case, you still subtract imageNet training data's mean and stds.

[yuffon]

In TensorFlow's official tutorial Resnet repository, they use per image standardization.
https://github.com/tensorflow/models/tree/master/official/resnet

[yuffon]

I don't think one should preprocess the images by its own mean and stds, that causes different input to be changed by different amounts.

CIFAR is not a good dataset to transfer from, I think if you do transfer learning it's better to start with ImageNet models. In this case, you still subtract imageNet training data's mean and stds.

If I make a new network that trained on cifar10, but I want to test the model on image out of the cifar10 dataset and observe the network behavior on the out-of-distribution data. How should I normalize the data?

[liuzhuang13]

I think the common practice is to just normalize the data as the way it is normalized during training (and using the training stats).