Could not train well on small dataset

[bearpaw]

Could not train well on small dataset

[linghai06]

To be more concrete, modified codes and training scripts are attached as follows:

self.train, self.valid = self.train[:4], self.valid[:4]

is added in datasets/mpii.py to select only 4 images

And here is the scripts

[bearpaw]

I think 4 images are not sufficient to train a pose estimator.

[linghai06]

Well, literally it is not training in normal sense but testing if the gradient flow could drive the network to memorize such a simple setting. If not, then probably there are some issues with the network design or the optimizer or anything else. I think it is a verification procedure before any further step.

There is a similar discussion on reddit: https://www.reddit.com/r/MachineLearning/comments/5pidk2/d_is_overfitting_on_a_very_small_data_set_a/

[bearpaw]

Interesting. What is the batchsize you used during training and testing? It is possible that the BN is not trained properly.

[linghai06]

Only 4 images were used and the batch-size was set to 4. Currently I only pay attention to the training loss for this 4 images and surprisingly found that it dropped slower than using full mpii and also stopped at a plateau higher.

Why should that be from the BN layers? Actually I also tried a small set with 500 images and the results were more or less the same with that from the set of 4 images. This might indicate it is not from the BN.