fastai/imagenette

A smaller subset of 10 easily classified classes from Imagenet, and a little more French

Imagenette

🎶 Imagenette, gentille imagenette,

Imagenette, je te plumerai. 🎶

(Imagenette theme song thanks to Samuel Finlayson)

---

NB:

• Versions of Imagenette and Imagewoof with noisy labels are now available as CSV files that come with the dataset.
• The Imagenette and Imagewoof datasets changed on Dec 6th 2019. They now have a 70/30 train/valid split.
• The old versions (which have a much smaller validation set) are still available with the same URLs, but the URLs below point to the new versions.
• We've also added the Image网 dataset (see below for details). The leaderboards below been updated using the new datasets, using a strong. Can you beat it?...

---

The Datasets

Imagenette

Imagenette is a subset of 10 easily classified classes from Imagenet (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute).

'Imagenette' is pronounced just like 'Imagenet', except with a corny inauthentic French accent. If you've seen Peter Sellars in The Pink Panther, then think something like that. It's important to ham up the accent as much as possible, otherwise people might not be sure whether you're refering to "Imagenette" or "Imagenet". (Note to native French speakers: to avoid confusion, be sure to use a corny inauthentic American accent when saying "Imagenet". Think something like the philosophy restaurant skit from Monty Python's The Meaning of Life.)

The '320 px' and '160 px' versions have their shortest side resized to that size, with their aspect ratio maintained.

• Full size download
• 320 px download
• 160 px download.

The dataset also comes with a CSV file with 1%, 5%, 25%, and 50% of the labels randomly changed to an incorrect label. More information about the noisy labels are provided in the "noisy_labels" folder. Leaderboards for 5% noise and 50% noise are maintained below.

Too easy for you? In that case, you might want to try Imagewoof.

Imagewoof

Imagewoof is a subset of 10 classes from Imagenet that aren't so easy to classify, since they're all dog breeds. The breeds are: Australian terrier, Border terrier, Samoyed, Beagle, Shih-Tzu, English foxhound, Rhodesian ridgeback, Dingo, Golden retriever, Old English sheepdog. (No we will not enter in to any discussion in to whether a dingo is in fact a dog. Any suggestions to the contrary are un-Australian. Thank you for your cooperation.)

• Full size download;
• 320 px download;
• 160 px download.

The dataset also comes with a CSV file with 1%, 5%, 25%, and 50% of the labels randomly changed to an incorrect label. More information about the noisy labels are provided in the "noisy_labels" folder.

Imagewoof too easy for you too?!? Then get your hands on Image网.

Image网

Image网 is pronounced "Imagewang"; 网 means "net" in Chinese! Image网 contains Imagenette and Imagewoof combined, but with some twists that make it into a tricky semi-supervised unbalanced classification problem:

• The validation set is the same as Imagewoof (i.e. 30% of Imagewoof images); there are no Imagenette images in the validation set (they're all in the training set)

• Only 10% of Imagewoof images are in the training set!

• The remaining are in the unsup ("unsupervised") directory, and you can not use their labels in training!

• It's even hard to type and hard to say!

• Full size download;

• 320 px download;

• 160 px download.

Why Imagenette?

I (Jeremy Howard, that is) mainly made Imagenette because I wanted a small vision dataset I could use to quickly see if my algorithm ideas might have a chance of working. They normally don't, but testing them on Imagenet takes a really long time for me to find that out, especially because I'm interested in algorithms that perform particularly well at the end of training.

But I think this can be a useful dataset for others as well.

Usage

If you are already using the fastai library, you can download and access these quickly with commands like:

path = untar_data(URLs.IMAGENETTE_160)

where path now stores the destination to ImageNette-160.

For researchers

• Try to create a classifier that's as accurate as possible under various constraints (we'll keep leaderboards below, submit your PR with a link to your repo or gist!), such as:
  • Within a certain number of epochs: 5, 20, 40, 160
  • Within a certain budget on AWS or GCP (use spot or interruptible instances to save money): $0.05, $0.10, $0.25, $0.50, $1.00, $2.00
• Experiment with other low resource problems like transfer learning from small datasets, using semi-supervised learning to help classify small datasets, etc
• Test the impact of using different sized images, either separately, or together as part of training (i.e. progressive resizing)
• Compare your algorithm on easy vs hard small datasets, which are otherwise very similar (Imagenette vs Imagewoof)
• Ensure that you start from random weights - not from pretrained weights.

For students

• Practice your modeling skills on a dataset that's very similar to Imagenet, but much less expensive to deal with
• Do send me a PR with your other applications for this dataset!

Tips

• Because there are only 10 categories, the usual "top 5 accuracy" isn't so interesting. So you should generally report top 1 accuracy when using Imagenette
• The best approaches to 5 epoch training often don't scale well to more epochs
• Data augmentation like mixup tends to only help for 80+ epochs

Leaderboard

Generally you'll see +/- 1% differences from run to run since it's quite a small validation set. So please only send in contributions that are higher than the reported accuracy >80% of the time. Here's the rules:

• No inference time tricks, e.g. no: TTA, validation size > train size
• Must start with random weights
• Must be one of the size/#epoch combinations listed in the table
• If you have the resources to do so, try to get an average of 5 runs, to get a stable comparison. Use the "# Runs" column to include this (note that train_imagenette.py provides a --runs flag to make this easy)
• In the URL column include a link to a notebook, blog post, gist, or similar which explains what you did to get your result, and includes the code you used (or a link to it), including the exact commit, so that others can reproduce your result.

Imagenette Leaderboard

Size (px)	Epochs	URL	Accuracy	# Runs
128	5	fastai2 train_imagenette.py 2020-10 + MaxBlurPool + tuned hyperparams	87.43%	5, mean
128	20	fastai2 train_imagenette.py 2020-01 + MaxBlurPool	91.57%	5, mean
128	80	fastai2 train_imagenette.py 2020-01	93.55%	1
128	200	fastai2 train_imagenette.py 2020-01	94.24%	1
192	5	fastai2 train_imagenette.py 2020-01 + MaxBlurPool	86.76%	5, mean
192	20	fastai2 train_imagenette.py 2020-01 + MaxBlurPool	92.50%	5, mean
192	80	fastai2 train_imagenette.py 2020-01	94.50%	1
192	200	fastai2 train_imagenette.py 2020-01	95.03%	1
256	5	fastai2 train_imagenette.py 2020-01 + MaxBlurPool	86.85%	5, mean
256	20	fastai2 train_imagenette.py 2020-01 + MaxBlurPool	93.53%	5, mean
256	80	fastai2 train_imagenette.py 2020-01	94.90%	1
256	200	fastai2 train_imagenette.py 2020-01	95.11%	1

---

Imagenette w/Label Noise = 5%

Size (px)	Epochs	URL	Accuracy	# Runs
128	5	baseline	83.44%	1
128	20	baseline	89.53%	1
128	80	baseline	89.30%	1
128	200	baseline	90.04%	1
192	5	baseline	84.13%	1
192	20	baseline	90.65%	1
192	80	baseline	91.01%	1
192	200	baseline	91.08%	1
256	5	SESEMI	88.87% ± 0.67	5,mean±std
256	20	baseline	91.39%	1
256	80	SESEMI	92.95% ± 0.12	3,mean±std
256	200	SESEMI	93.96% ± 0.23	3,mean±std

---

Imagenette w/Label Noise = 50%

Size (px)	Epochs	URL	Accuracy	# Runs
128	5	baseline	66.60%	1
128	20	baseline	79.36%	1
128	80	baseline	50.80%	1
128	200	baseline	52.18%	1
192	5	baseline	67.54%	1
192	20	baseline	79.34%	1
192	80	baseline	52.51%	1
192	200	baseline	53.71%	1
256	5	SESEMI	76.72% ± 0.83	5,mean±std
256	20	baseline	79.21%	1
256	80	SESEMI	57.76% ± 0.39	3,mean±std
256	200	SESEMI	61.48% ± 0.33	3,mean±std

---

Imagewoof Leaderboard

Size (px)	Epochs	URL	Accuracy	# Runs
128	5	depthwise(x6)	76.61%	5, mean
128	20	depthwise(x4)	86.27%	5, mean
128	80	depthwise(x4)	87.83%	1
128	200	fastai2 train_imagenette.py 2020-01	87.20%	1
192	5	depthwise(x4)	81.15%	5, mean
192	20	depthwise(x4)	88.37%	5, mean
192	80	depthwise(x2)	90.30%	1
192	200	fastai2 train_imagenette.py 2020-01	89.54%	1
256	5	Resnet Trick + Mish + Sa + MaxBlurPool	78,84%	5, mean
256	20	Resnet Trick + Mish + Sa + MaxBlurPool	88,58%	5, mean
256	80	fastai2 train_imagenette.py 2020-01	90.48%	1
256	200	fastai2 train_imagenette.py 2020-01	90.38%	1

---

Image网 Leaderboard

Size (px)	Epochs	URL	Accuracy	# Runs
128	5	SwAV	72.94%	5,mean
128	20	SwAV	72.18%	3,mean
128	80	SwAV	69.53%	1
128	200	SwAV	66.04%	1
192	5	SwAV	77.07%	5,mean
192	20	SwAV	77.81%	3,mean
192	80	SwAV	74.9%	1
192	200	SwAV	71.77%	1
256	5	SwAV	79.56%	5,mean
256	20	SwAV	79.2%	3,mean
256	80	SESEMI	78.41% ± 0.39	5,mean±std
256	200	SESEMI	79.27% ± 0.20	3,mean±std

imagenette's Issues

InPainting results has default pre-trained flag turned on

While trying to repro the in-painting notebook (really cool by the way), I noticed u-net is loading a frozen pre-trained imagenet stem:
learn = unet_learner(dbunch, partial(m, sa=sa), opt_func=opt_func, metrics=[], loss_func=MSELoss())

In this case, the default pretrained flag needs to be specifically turned off:
https://github.com/fastai/fastai2/blob/master/fastai2/vision/learner.py#L183

From my initial tests, it seems to matter.

@JoshVarty

ResNet 18 - 92% accuracy

I couldn't decide whether to make a PR but resnet18 baseline gives 92+% top1 accuracy on imagenette with out of the box fastai get_transform, size=160, TTA within n_epoch=40.

• best model epoch: 36
• non TTA top1: 91%
• TTA top1: 92.2%

github repo: https://github.com/KeremTurgutlu/imagenette_experiments/blob/master/starter.ipynb

everthing is seeded so that it can be reproduced.

Keep-Alive-Actions

How to cite you work？

Paper or link?

Accuracy of resnet18 with imagewoof

Hi,
I tried training Resnet-18(no-pretraining) with imagewoof, since it was missing from the shared results table. Using 160px I managed to go till 75% accuracy without any bells and whistles. I only used data augmentation(random horiz. flip, random rotation, random cropping). I was wondering if you have a baseline measure against which I can compare my results. The difference appears to be significant from xresnet + modifications.

How to get class names from folder names?

How can I get the names of the dog breeds for the imagewoof dataset from the folder names?

Comparison to CIFAR-10

I think the main competitor to this set is something standard like CIFAR-10. So, how does this compare? A short comparison in the readme would be useful. To me, it looks like the main difference is using higher resolution images instead of the tiny 32x32 images that are hard even for humans.

Leaderboards for 128/192/256 image size but datasets are full/320/160

Not sure if this is a typo in the leaderboard or just me not understanding the leaderboard.

The leaderboard shows results for 128/192/256 images, however the datasets available are only available in "full size", shortest side of 320 and shortest side of 160.

Does this mean the leaderboard should be changed to full/320/160? Or the results reported are supposed to be from myself resizing the images to either 128/192/256?

Rating problems

Hi friends of Imagenette,

I tried my luck with the evaluation on Imagenette with a modified Resnet 50 (more parameters) and I got these results:
"model": "resnet50",
"top1": 93.2739,
"top1_err": 6.7261,
"top5": 98.9299,
"top5_err": 1.0701,
"param_count": 30.23,
"img_size": 224,
"crop_pct": 0.875,
"interpolation": "bicubic"

What do you think? Very good or standard?
(For a modiefied ResNet50 with Bottleneck arcitecture.)

Would appreciate an opinion, thanks,
André

Imagewoof - Resnet18/34 Training

Hey,
I am trying to produce a baseline model for imagewoof as it is hard to find a pretrained model.
I trained both resnet-18/34 by using Adam optimizer. I have reached 81.55/82.62% for top-1 accuracy.

I found some papers reporting that imagenette and imagewoof are 13k samples for training, where I expect this is would be the first version.

1. Could anyone post a link for the 1st version of them? I want to reproduce their results.

2. Could we use both datasets to fine-tune the hyperparameters for any training algorithm to generalize it on ImageNet?

3. Is Imagenette and imagewoof training sets contains some samples from the training set? I checked the files content and I found some but I need someone to confirm this.

Ref: Data Efficient Stagewise Knowledge Distillation

Add Self-Supervised baselines for ImageNette

Hi, we just recently released some baseline benchmarks using self-supervised models such as MoCo, SimCLR, BYOL here: https://github.com/lightly-ai/lightly/releases/tag/v1.1.12

We used the 160px version of the dataset.

The code we used for benchmarking is here: https://github.com/lightly-ai/lightly/blob/master/docs/source/getting_started/benchmarks/imagenette_benchmark.py

Could we add the baselines to the readme? I'm sure others would be interested in seeing how well self-supervised models perform.

NSFW images

Are you aware of the presence of NSFW images in the dataset? (image 3764 for example, parachute category)

Noisy Imagenette/Woof Proposal

Noisy Imagenette/Woof

Introduction

Most of the time, dataset labels are actually quite noisy as the humans generating those labels are error-prone. This is especially the case for labels generated through crowdsourcing. Recently, there has been significant research in dealing with noisy labels in datasets and training deep learning models that are robust to noise (ex: here, here, here, here, here, and here). It would be great to be able to implement some of these techniques in fastai (some work has already been done on this front, ex: here) and test them on a benchmarking dataset.

Proposal

I propose to add to this dataset/repository a corrupted version of Imagenette and ImageWoof where the training labels are switched randomly at varying probabilities to simulate datasets with different levels of noise. This dataset is currently available here. The images themselves are the same as Imagenette, but the labels are instead provided in a CSV file. There are 4 noise levels: 1%, 5%, 25%, 50% noise. The generation of these labels are provided here. A baseline based on the original Imagenette baseline is provided as well.

Note that in this imagenette repository there are currently 24 leaderboards (12 for Imagenette and 12 for ImageWoof). Since there are 4 noise levels, there is a total of 96 possible leaderboards. I have run my baseline for all 96 possibilities (done automatically on 4 TITAN RTXs with this bash script) and provided an extended leaderboard over here. I have also selected 16 leaderboards that I have kept on the README. It is up to the maintainers (Jeremy and Hamel) to decide which leaderboards they want to keep.

What needs to be done

These are the things I believe needs to be done for this dataset to be added to Imagenette:

• Add URLs for Noisy Imagenette/Woof (same as Imagenette but with the extra CSV file)
• Select the leaderboards to include
• Add the leaderboards to the repository README
• Potentially add the label generation, benchmarks, extended LB in another folder on this repository? Alternatively, I could keep it hosted in my repository.

@jph00 and @hamelsmu Please review my repository and let me know if you think there are any additional tasks that I would need to do.

Train- / Validation-Split of Imagenette

I am using Imagenette for fine-tuning of an Imagenet pre-trained VGG-16 from the PyTorch model zoo.
Is the validation-set of Imagenette build from the validation- / test-set of Imagenet? Or are there some Imagenet training examples in the Imagenette validation-set?

Because after fine-tuning the pre-trained VGG for 1 epoch (on Imagenette), I reach a Top-1 Accuracy of 98.4% on the validation-set. Am I dealing with some data leakage here?

89% accuracy. 160px 25 epochs

I was able to create a classifier with a resnet like architecture and was able to achieve an accuracy of 89% with 25 epochs for 160px dataset.
github repo: https://github.com/jinudaniel/imagenette

Are the leaderboard entries using accuracy or top k accuracy?

I am asking because the entry:

256 | 20 | fastai2 train_imagenette.py 2020-01 + MaxBlurPool | 93.53% | 5, mean

Related to the following ipynb

https://github.com/ducha-aiki/imagewoofv2-fastv2-maxpoolblur/blob/master/fastai2-imagenette-train-maxblurpool.ipynb

Appears with accuracy "93.53%" in the leader board, but in the ipynb the accuracy never reaches 86%.
What does reach >90% is the top_k_accuracy.

URLs.IMAGEWOOF train-val

Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastai.vision import *
>>> path = untar_data(URLs.IMAGEWOOF)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/fastai-fork/fastai/datasets.py", line 233, in untar_data
    tarfile.open(fname, 'r:gz').extractall(dest.parent)
  File "/home/ubuntu/anaconda3/lib/python3.7/tarfile.py", line 2002, in extractall
    numeric_owner=numeric_owner)
  File "/home/ubuntu/anaconda3/lib/python3.7/tarfile.py", line 2044, in extract
    numeric_owner=numeric_owner)
  File "/home/ubuntu/anaconda3/lib/python3.7/tarfile.py", line 2114, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/home/ubuntu/anaconda3/lib/python3.7/tarfile.py", line 2163, in makefile
    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
  File "/home/ubuntu/anaconda3/lib/python3.7/tarfile.py", line 247, in copyfileobj
    buf = src.read(bufsize)
  File "/home/ubuntu/anaconda3/lib/python3.7/gzip.py", line 276, in read
    return self._buffer.read(size)
  File "/home/ubuntu/anaconda3/lib/python3.7/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/home/ubuntu/anaconda3/lib/python3.7/gzip.py", line 482, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
>>> path = untar_data(URLs.IMAGEWOOF_320)
>>> (path/"train").ls()
[PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02093754'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02088364'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02089973'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02087394'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02111889'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02099601'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02105641'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02086240'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02115641'), PosixPath('/home/ubuntu/.fastai/data/imagewoof-320/train/n02096294')]

Getting this error when untaring IMAGEWOOF full size. Other sizes untar without any error.

Keep-Alive-Actions

Question regarding the LICENSE

I have a question regarding the license of this. I see this is redistributed with Apache2.0 - I was wondering were you able to gain the rights for that subset of imagenet to be distributed under Apache2.0 or do the same license terms as per the regular imagenet apply?

Bonjour!

Any format to follow for sending a PR?

I have uploaded a naive implementation of training a model on your dataset in this repo.

Kaggle Dataset

Hey,
thank you for the work in creating these datasets!
Have you ever considered publishing a kaggle competition / dataset?

Thanks :-)

Keep-Alive-Actions

BibTeX citation

Is there a preferred BibTeX citation for Imagenette, Imagewoof and Imagewang?

Bounding box for this dataset

Hi, how can I access the bounding box for this one?