cmasch/zalando-fashion-mnist

Evaluation of fashion-MNIST with a simple cnn

Image classification on fashion-MNIST

I would like to share my results (93.43% accuracy on average) on the fashion-MNIST dataset. You can find further informations about the dataset on Zalando Research and Kaggle This dataset is a great option instead of using traditional handwritten MNIST.
Thanks to Han and Kashif!

Evaluation procedure

I splitted the training data randomly in train (80%) and validation (20%). The testset of 10k images are used for final evaluation. I created 5 models with the same architecture but with random train/validation data. I only saved the weights of every model with best loss. Finally I used the models to evaluate them on the testset. The average loss/accuracy of the 5 models is the final result.
Furthermore I evaluate the generalization of the network for classifying traditional MNIST at the end.

Results

Actually I'm focussed on a very simple architecture with less than 500,000 parameters which could run on CPU with 4GB memory. If you are interested in further results, you can find them on Zalando benchmark. If it fits in time, I will evaluate architectures e.g. DenseNet.

I'm just using two convolutional layers, batchnorm, dropout and three fully connected layers. For a detailed implementation check the model definition in the jupyter notebook.
This is the summary of the current model:

It's easy to reach an accuracy over 90% but around 92% it stucks very fast. If we take a look on the confusion matrix, we will find out that distinguishing between Shirt and T-Shirt/top is very difficult:

Here's an example where my model fails. I'm not a fashion expert. Maybe thats the reason why I don't see any difference. If you could explain why this is a shirt and not a t-shirt I would appreciate it:

Another two plots which illustrate the accuracy / loss of training and validation over the time:

And a detailed plot of iteration 0:

Scores for training: 0.1066 loss / 95.99% accuracy.
Scores for validation: 0.1245 loss / 95.64% accuracy.
Scores for test: 0.2149 loss / 93.43% accuracy.

MNIST

I used the same architecture of neural network to train on traditional MNIST.
Scores for training: 0.078 loss / 99.74% accuracy.
Scores for validation: 0.161 loss / 99.61% accuracy.
Scores for test: 0.0248 loss / 99.43% accuracy.

In the plot below you can track training / validation for MNIST:

Overview

Fashion-MNIST test accuracy	Fashion-MNIST train accuracy	Fashion-MNIST validation accuracy	MNIST test accuracy	Add. Settings	Trainable Params
93.43%	95.99%	95.64%	99.43%	BatchSize : 250 Epochs : 80 Data augmentation (2x)	493,772

Requirements

• TensorFlow 2.x
• TensorFlow-Datasets

If you have any questions or hints contact me through an issue. Thanks!

References

[1] Fashion-MNIST
[2] Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Author

Christopher Masch