barisozmen/deepaugment

Discover augmentation strategies tailored for your dataset

deepaugment's Issues

Loss turns into 'nan'

I'm experimenting with wrn-16-8 (WideResNet) at this repo. During training, loss suddenly turned into nan. I guess it's a numerical calculation problem.

Use black code formetter

https://github.com/ambv/black

black formats code in PEP 8 style:

Black ignores previous formatting and applies uniform horizontal and vertical whitespace to your code. The rules for horizontal whitespace can be summarized as: do whatever makes pycodestyle happy. The coding style used by Black can be viewed as a strict subset of PEP 8.

Visualize tf models with tensorboard

Tensorboard result would be something like this:

"notebook.csv" format is different

Hi, thank you for sharing this code!
I have training on my own data with the following code:

---

cnn_config = {"model":model,
"child_batch_size": 32,
"child_epochs": 50}
deepaug = DeepAugment(images=x_train, labels=y_train, config=cnn_config)
best_policies = deepaug.optimize(100)

---

The columns names in the CSV file are different from the names of the columns that you display on "notebooks/result-analyses/*".

My columns names are:
acc | loss | val_acc | val_loss | trial_no | A_aug1_type | A_aug1_magnitude | A_aug2_type | A_aug2_magnitude | B_aug1_type | B_aug1_magnitude | B_aug2_type | B_aug2_magnitude | C_aug1_type | C_aug1_magnitude | C_aug2_type | C_aug2_magnitude | D_aug1_type | D_aug1_magnitude | D_aug2_type | D_aug2_magnitude | E_aug1_type | E_aug1_magnitude | E_aug2_type | E_aug2_magnitude | sample_no | mean_late_val_acc | epoch

while yours are:
acc | loss | val_acc | val_loss | trial_no | aug1_type | aug1_magnitude | aug2_type | aug2_magnitude | aug3_type | aug3_magnitude | portion | sample_no | mean_late_val_acc

Do I have to run more code lines to get the same file as yours?

.

Explore raw data and report it here:
/notebooks/explore-raw-data.ipynb

Explore raw data

Do an explorative analysis on jupyter notebook and put it to /notebooks/explore-raw-data

Notebook should iterate followings:

1. Explore raw training data from DOTA, and report followings:

   • Distribution of ground sample distances (gsd) of images
   • Distribution of image sources (GoogleEarth or others)
   • Frequency of each object type across all images
   • Number of objects per image
   • Co-occurrences and mutual exclusivity of object types across images
   • Are images augmented?

2. Show some image samples

3. Write an overall summary of explorative analysis. And add necessary information from DOTA paper in it.

Required Dependencies

Hello my friend,

Wich version of Tensorflow is needed?
(For GPU support)
Wich Python Version works best?

pip install doesn't work

DeepAugment for Semantic Segmentation Task

Hi @barisozmen

How can one use deepaugment for encoder-decoder network architectures like semantic and instance segmentation.

Thanks and Regards,
Deeksha.
Research Scholar (Data Science)
IIIT, Bangalore, India.

datasets for showcasing deepaugmenter

Fashion-MNIST (https://hanxiao.github.io/2018/09/28/Fashion-MNIST-Year-In-Review/) (https://medium.com/@lukaszlipinski/fashion-mnist-with-keras-in-5-minuts-20ab9eb7b905)

Build object detection model v0.1

An SSD model pre-trained by DOTA team would be a good start. It can be downloaded following links here (github.com/ringringyi/DOTA_models#training)

augmented images output

Its unclear for me if this project produces the images or if it also does the training - outputting the final score. Is it possible to use this project like keras ImageDataGenerator?

Dropout Cause significant performance change between each trainning

Using Dropout in child_model shows great works on prevent overfitting, however it also cause the final performance on model change significantly during each training with same hyper-params. It is too random that cause that we need using more sampling times to estimate final performance on one hyper-params which is very time consuming. Any ideal for solving this problem.

Explore 16 data transformation with PIL

They are listed here:

'''
def ShearX(img, v): # [-0.3, 0.3]
return img.transform(img.size, PIL.Image.AFFINE, (1, v, 0, 0, 1, 0))

def ShearY(img, v): # [-0.3, 0.3]
return img.transform(img.size, PIL.Image.AFFINE, (1, 0, 0, v, 1, 0))

def TranslateX(img, v): # [-150, 150] => percentage: [-0.45, 0.45]
v = v*img.size[0]
return img.transform(img.size, PIL.Image.AFFINE, (1, 0, v, 0, 1, 0))

def TranslateY(img, v): # [-150, 150] => percentage: [-0.45, 0.45]
v = v*img.size[1]
return img.transform(img.size, PIL.Image.AFFINE, (1, 0, 0, 0, 1, v))

def Rotate(img, v): # [-30, 30]
return img.rotate(v)

def AutoContrast(img, _):
return PIL.ImageOps.autocontrast(img)

def Invert(img, _):
return PIL.ImageOps.invert(img)

def Equalize(img, _):
return PIL.ImageOps.equalize(img)

def Flip(img, _): # not from the paper
return PIL.ImageOps.mirror(img)

def Solarize(img, v): # [0, 256]
return PIL.ImageOps.solarize(img, v)

def Posterize(img, v): # [4, 8]
v = int(v)
return PIL.ImageOps.posterize(img, v)

def Contrast(img, v): # [0.1,1.9]
return PIL.ImageEnhance.Contrast(img).enhance(v)

def Color(img, v): # [0.1,1.9]
return PIL.ImageEnhance.Color(img).enhance(v)

def Brightness(img, v): # [0.1,1.9]
return PIL.ImageEnhance.Brightness(img).enhance(v)

def Sharpness(img, v): # [0.1,1.9]
return PIL.ImageEnhance.Sharpness(img).enhance(v)

def Cutout(img, v): # [0, 60] => percentage: [0, 0.2]
w, h = img.size
v = v*img.size[0]
x0 = np.random.uniform(w-v)
y0 = np.random.uniform(h-v)
xy = (x0, y0, x0+v, y0+v)
color = (127, 127, 127)
img = img.copy()
PIL.ImageDraw.Draw(img).rectangle(xy, color)
return img

def SamplePairing(imgs): # [0, 0.4]
def f(img1, v):
i = np.random.choice(len(imgs))
img2 = PIL.Image.fromarray(imgs[i])
return PIL.Image.blend(img1, img2, v)
return f
'''

Build data processing pipeline v0.1

Data preprocess should be like:

1. Remove images having width or height less than 608*
2. Split images using SplitImg.py module of DOTA_devkit, where subsize=608 and gap=0.
3. Remove any image whose after-split dimensions are not order of 608
4. Convert oriented bounding boxes (OBB) to horizontal bounding boxes (HBB)

For pipeline v0.1, only use 20 images for training set, where 10 of them having "planes" in it. All images from test set. MVP targets only to detect planes.

• The rationale of choosing 608 as size is that pre-trained model, which I will use in v0.1, was trained by 608x608 images (https://github.com/ringringyi/DOTA_models#training).

Unable to run with fashion_mnist

Hi,

Could you please explain about sample script that use fashion_mnist dataset
As I know, fashion_mnist is gray dataset, how convert to to 3 channel images as requirement ?

#below is code i used:

from keras.datasets import fashion_mnist
from deepaugment.deepaugment import DeepAugment
my_config = {
"model": "basiccnn",
"method": "bayesian_optimization",
"train_set_size": 2000,
"opt_samples": 3,
"opt_last_n_epochs": 3,
"opt_initial_points": 10,
"child_epochs": 50,
"child_first_train_epochs": 0,
"child_batch_size": 64
}
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
X_train = x_train.reshape(x_train.shape[0], x_train.shape[1], x_train.shape[2], 1)
deepaug = DeepAugment(images=X_train, labels=y_train, config=my_config)
best_policies = deepaug.optimize(300)

I have tried to reshape X_train = x_train.reshape(x_train.shape[0], x_train.shape[1], x_train.shape[2], 3), but it cann't
The error I faced

0, 0.1327777779367234, ['rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0]
trial: 1
['gamma-contrast', 0.8442657485810175, 'coarse-salt-pepper', 0.8472517387841256, 'brighten', 0.38438170729269994, 'translate-y', 0.056712977317443194, 'translate-y', 0.47766511732135, 'add-to-hue-and-saturation', 0.47997717237505744, 'emboss', 0.8360787635373778, 'sharpen', 0.6481718720511973, 'emboss', 0.9571551589530466, 'rotate', 0.8700872583584366]
/home/kaka/PycharmProjects/DeepAugment /venv/lib/python3.6/site-packages/imgaug/augmenters/color.py:448: UserWarning: Received an image with shape (H, W, C) and C=1 in ChangeColorspace._augment_image(). Expected C to usually be 3 -- any other value will likely result in errors. (Note that this function is e.g. called during grayscale conversion and hue/saturation changes.)
"changes.)" % (image.shape[2],)

deepAugment for regression

Is it possible to add simply add ChildCNN for regression - using MSE instead of accuracy? Or will that also require change to the Controller?

Reinforcement Learning resources

Many resources referenced here (https://www.manifold.ai/blog/exploration-exploitation-reinforcement-learning)

UC Berkeley RL course (http://rail.eecs.berkeley.edu/deeprlcourse/)

'Fast AutoAugment' by my team, KakaoBrain

30x-250x efficient method to find augmentation policies automaticallty, relative to AutoAugment by Google.

Arxiv : https://arxiv.org/abs/1905.00397
Code : https://github.com/KakaoBrain/fast-autoaugment

We don't train child-networks like AutoAugment or deepaugment, and that is the key reason of the speed. But I really appreciate your work and I hope we can influence each other, in a good way. I also want to make my repo easy to use like yours.

Using deepaugment with large custom dataset (using generator)?

According to my observation, I don't see deepaugment support big dataset (which I cannot load all images and labels at a time and have to use data generator)? If I'm missing something, can you show how to use this repo with custom dataset which I have to use data generator?

AssertionError when do augment policy

I write a simple script like this:

import os
from deepaugment import DeepAugment
from keras.datasets import cifar10

(x_train, y_train), (x_test, y_test) = cifar10.load_data()
deepaug = DeepAugment(x_train, y_train)
best_policies = deepaug.optimize(300)

after run it, about one minute, I got a AssertionError:

...
...
Epoch 48/50
 - 1s - loss: 0.2101 - acc: 0.9382 - val_loss: 2.6119 - val_acc: 0.5540
Epoch 49/50
 - 1s - loss: 0.2101 - acc: 0.9347 - val_loss: 2.7725 - val_acc: 0.5430
Epoch 50/50
 - 1s - loss: 0.2075 - acc: 0.9388 - val_loss: 1.9880 - val_acc: 0.5510
fit()'s runtime:  55.3912 sec.
0, 0.567111111190584, ['rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0, 'rotate', 0.0]
('trial:', 1, '\n', ['gamma-contrast', 0.8442657485810175, 'coarse-salt-pepper', 0.8472517387841256, 'brighten', 0.38438170729269994, 'translate-y', 0.056712977317443194, 'translate-y', 0.47766511732135, 'add-to-hue-and-saturation', 0.47997717237505744, 'emboss', 0.8360787635373778, 'sharpen', 0.6481718720511973, 'emboss', 0.9571551589530466, 'rotate', 0.8700872583584366])
Traceback (most recent call last):
  File "test.py", line 29, in <module>
    best_policies = deepaug.optimize(300)
  File "/data/ansheng/cv_strategy/autoML/deep_augment/deepaugment-master/deepaugment/deepaugment.py", line 151, in optimize
    f_val = self.objective_func.evaluate(trial_no, trial_hyperparams)
  File "/data/ansheng/cv_strategy/autoML/deep_augment/deepaugment-master/deepaugment/objective.py", line 44, in evaluate
    self.data["X_train"], self.data["y_train"], *trial_hyperparams
  File "/data/ansheng/cv_strategy/autoML/deep_augment/deepaugment-master/deepaugment/augmenter.py", line 166, in augment_by_policy
    ), "first transform is unvalid"
AssertionError: first transform is unvalid

The code that throw Error is:

X_portion_aug = transform(hyperparams[i], hyperparams[i+1], X_portion)  # first transform
assert (
    X_portion_aug.min() >= -0.1 and X_portion_aug.max() <= 255.1
), "first transform is unvalid"

It seems the code after data-augmentation is out of range [0,255].

So if the function augment_by_policy() in augmenter.py has some bug?

List of relevant resources

Journal papers

• AutoAugment: Learning Augmentation Policies from Data (link)
• Smart Augmentation - Learning an Optimal Data Augmentation Strategy (link)
• Adaptive data augmentation for image classification (link)
• Learning to Compose Domain-Specific Transformations for Data Augmentation (link)
  • Cited by AutoAugment paper

AutoAugment implementations

• Official (github)
  • Tensorflow used
  • does not have controller (RL) part
• Unofficial (github)
  • Keras used
  • Have the controller part
• Unofficial exploration on jupyter notebook (github)
  • Very good for learning
  • Pytorch used
  • Doesn't implement the whole
• In tensorflow-hub (link)

Blogs

• https://towardsdatascience.com/how-to-improve-your-image-classifier-with-googles-autoaugment-77643f0be0c9
• Comparing accuracy increase of various augmentation methods (https://medium.com/nanonets/how-to-use-deep-learning-when-you-have-limited-data-part-2-data-augmentation-c26971dc8ced)
  • NanoNets
• A bag of tricks for image classification (https://towardsdatascience.com/a-big-of-tricks-for-image-classification-fec41eb28e01)
  • Occlusion (putting grey rectangle to some areas of image) as a data augmentation method.
• Deep neuro-evolution (DL + GA) (https://eng.uber.com/deep-neuroevolution/) (github)
  • Ekin Dogus suggested at this video (12:17)
  • Uber AI team hybridizing deep learning with genetic algorithms

Videos

• Seminar presentation (Youtube)
  • distills the situation and the model well!
  • discussion part gives a good critic, data augmentation might overfit for the validation set

Libraries

• Tanda Learning to Compose Domain-Specific Transformations for Data Augmentation
• Automold road augmentation library for self-driving cars
• Albumentations general data augmentation library
• Autosat Semantic segmentation on aerial and satellite imagery. Extracts features such as: buildings, parking lots, roads, water, clouds
• NAS
• autokeras

Ideas

• How to make AutoAugment scalable?
  • By training models with 50-100 epochs in the beginning with original data. And then training it with suggested augmentations!
    • Training times of various models and datasets (https://dawn.cs.stanford.edu/benchmark/)
    • The DAWNbench paper (https://dawn.cs.stanford.edu/benchmark/papers/nips17-dawnbench.pdf)
• Transfer learning (TL) for decreasing computation time
• Transfer learning and genetic algorithms
• A case for using Random Search instead of RL:
  • *"Typically random search algorithms sacrifice a guarantee of optimality for finding a good solution quickly with convergence results in probability." *(Random Search Algorithms)
  • Also see: "Simple random search provides a competitive approach to reinforcement learning" (https://arxiv.org/abs/1803.07055)
  • Data augmentation guided by cooccurrences of objects
  • (https://www.researchgate.net/publication/328899557_Learning_data_augmentation_policies_using_augmented_random_search) paper claims that they get better results than AutoAugment by tweaking its parameters

monitor progress

HI @barisozmen thanks for sharing the code for deepaugment
I would like to try this on my dataset.
which value would you recommend to monitor on?
have you considered to implement tensorboard/ tensorboardX in the code for easy validate of the process?

thanks!

Determine project rules

Some suggestions:

• SOLID rules (https://hackernoon.com/solid-principles-made-easy-67b1246bcdf)
• Robert Martin rules (https://gist.github.com/wojteklu/73c6914cc446146b8b533c0988cf8d29)

Resources on Bayesian optimization

https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f

http://josh-tobin.com/assets/pdf/troubleshooting-deep-neural-networks-01-19.pdf (Last slides are about Bayesian hyperparameter optimization)

Explore experiment results from 2019-1-30_19-27

Experiment data is at: https://github.com/barisozmen/deepaugmenter/tree/master/reports/experiments/2019-1-30_19-27

Should child models trained with X_aug only, or with X + X_aug?

X : data as it is
X_aug: augmented version of X

Current plan:
Make an initial training (200 epochs) with X of the child model, then using trained weights:

1. train 60 epochs with X_aug
2. train 60 epochs with X + X_aug

Make an experiment for options 1 and 2, and see which one is better.

PyPl 's typo(sorry, Im good at PR)

https://pypi.org/project/deepaugment/

In the Advanced usage

deepaug = DeepAugment(iamges=x_train, labels=y_train, config=my_config)

!! images typo 'iamges' !!
the right patten is I thought

deepaug = DeepAugment(images=x_train, labels=y_train, config=my_config)

skopt resources

A nice usage example of skopt: https://geekyisawesome.blogspot.com/2018/07/hyperparameter-tuning-using-scikit.html

How to get the parrot.jpg?

How to get the parrot.jpg?

Compatibily with Tensorflow 1.13.x

I've just discovered your projet and I would like to add it in my ML pipeline using Tensorflow 1.13.1 but the requirement of my projet doesn't match yours. Is it planned to support an upper version of Tensorflow ?

Thanks !

list index out of range when using child_first_train_epochs

When setting child_first_train_epochs (I tried with 15 and 20), the following error occurs after training 'child_first_train_epochs' number of epochs:

Traceback (most recent call last):
  File "run_deepaugment.py", line 51, in <module>
    deepaug = DeepAugment(images=x_train, labels=y_train.reshape(TRAIN_SET_SIZE, 1), config=my_config)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/deepaugment/lib/decorators.py", line 106, in wrapper
    return func(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/deepaugment/deepaugment.py", line 120, in __init__
    self._do_initial_training()
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/deepaugment/deepaugment.py", line 202, in _do_initial_training
    -1, ["first", 0.0, "first", 0.0, "first", 0.0, 0.0], 1, None, history
  File "/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/deepaugment/notebook.py", line 38, in record
    new_df["B_aug2_magnitude"] = trial_hyperparams[7]
IndexError: list index out of range

Here is my config used:

my_config = {
    'model': 'wrn_16_2',
    'train_set_size': int(TRAIN_SET_SIZE*0.75),
    'child_epochs': 60,
    'child_batch_size': 64,
    'child_first_train_epochs': 20,
    'opt_samples': 1,
}

Where TRAIN_SET_SIZE is a custom dataset of 3000 examples
The code runs fine if I omit the child_first_train_epochs setting

Which object detection model should be chosen? RetinaNet or SSD

deepaugment with object detection datasets

I have a custom dataset to be used for object detection.

Can deepaugment be applied to this dataset?

I would be very grateful if you could reply.