evizero/augmentor.jl

A fast image augmentation library in Julia for machine learning.

augmentor.jl's Issues

Apply same transformation to mask?

I have boolean masks for my images for semantic segmentation that I would like to transform identically to the raw image so that I can augment my training set.

Is there any way to deterministically apply the same transformations to both the raw image and the mask?

Run femtocleaner

add documentation settings

The last thing to do before registering v0.6.0 in the General registry is to update the documentation.

@Evizero Looks like this repository haven't set up auto-deployment CI for documentation, right? Can you add a deploy key and a secret key (with name DOCUMENTER_KEY) following the instruction of https://juliadocs.github.io/Documenter.jl/stable/man/hosting/#travis-ssh-1 I plan to set the documentation via Github action

configure ssh key for tagbot

Here's the motivation:

One GitHub action could not trigger another GitHub action if only configured with token: ${{ secrets.GITHUB_TOKEN }}. This limitation means release/tag created by tagbot could trigger the docs CI.

@Evizero could you help set up an ssh key and corresponding github secrets for this? It should be ssh key with write permission in https://github.com/Evizero/Augmentor.jl/settings/keys (public, write permission) and secrets in https://github.com/Evizero/Augmentor.jl/settings/secrets/new (private)

The quick command to generate ssh key:

ssh-keygen -t rsa -b 4096 -C "[email protected]" -f tagbot

then copy tagbot.pub as keys and tagbot as secrets (let's say with name "TagBot").

References:

• https://juliadocs.github.io/Documenter.jl/dev/man/hosting/#GitHub-Actions

Do not show `NoOp`?

pl = Either(NoOp(), Rotate90(), Rotate180(), Rotate270()) |>
            Either(0.5=>NoOp(), 0.25=>FlipX(), 0.25=>FlipY())
2-step Augmentor.ImmutablePipeline:
 1.) Either: (25%) No operation. (25%) Rotate 90 degree. (25%) Rotate 180 degree. (25%) Rotate 270 degree.
 2.) Either: (50%) No operation. (25%) Flip the X axis. (25%) Flip the Y axis.

vs

pl = Either(NoOp(), Rotate90(), Rotate180(), Rotate270()) |>
            Either(0.5=>NoOp(), 0.25=>FlipX(), 0.25=>FlipY())
2-step Augmentor.ImmutablePipeline:
 1.) Either: (25%) Rotate 90 degree. (25%) Rotate 180 degree. (25%) Rotate 270 degree.
 2.) Either: (25%) Flip the X axis. (25%) Flip the Y axis.

I prefer the latter since NoOp works more like a placeholder for randomness.

Is there a way to see the generated displacement field?

Hello,
I am working on an image registration method and Augmentor would be a nice way for me to test it. Ideally, however, I would like to see the displacement field that maps the original to the augmented image.
Is there a way to do this in Augmentor?

For example is there a way to see a displacement vector for every pixel in img that map to the pixels' location in img_proccessed for every a specific augment call?

img = testpattern()
pl = FlipX() |> Zoom(0.9:0.1:1.2) |> CropSize(64,64)
img_processed = augment(img, pl)

I hope this makes sense. Thanks in advance.

Unify the augment interface

tl;dr Remove augmentbatch! and have augment and augment! support batch inputs. I think it's doable but there are some issues.

---

Currently we have

• augment(imgs, pl),
• augment!(outs, imgs, pl),
• augmentbatch!(out_batch, img_batch, pl),

and we would be also interested in

• augmentbatch(img_batch, pl).

Should we drop the "batch" versions, and keep just augment and augment!? And if so, can we do it? I tend to think yes and yes.

Why

It simplifies the interface. The user does not have to care if they work with batches or single images, the same function will work for both.

Having only a single function would help us solve #33 easily, too.

How

I believe the main issue is to tell if an input is a batch or not based on its type. Once we are done with that, we can just employ dispatching to call appropriate code.

In my limited view of the world, an image is a 3-dimensional array of shape (H, W, C), where H, W, C is the height, width, and number of channels. When C=1, the image can as well be a 2-dimensional array of shape (H, W).

A batch is a 4-dimensional array of shape (N, H, W, C), where N is the number of images. Again, when C==1, the batch can be a 3-dimensional array of shape (N, H, W).

In Julia, the situation is a little bit more complicated, as the array elements tend to be structure instances (like RGB) and the "channel" dimension is omitted.

Also, the dimensions can be permuted (e.g., (H, W, C, N)), but Augmentor already deals with that to some extent.

The following table maps the input types to the decision batch/image. Unfortunatelly, there is one ambiguity (typed in bold).

Type	Shape	Element	Example	Decision
AbstractArray{<:Number, 4}	(N, H, W, C)	intensity in a channel	batch of RGB images	batch
AbstractArray{<:Color{T, 3}, 3}	(N, H, W)	3-d value for all channels	batch of RGB images	batch
AbstractArray{<:Color{T, 1}, 3}	(N, H, W)	pixel intensity	batch of grayscale images	batch
AbstractArray{<:Number, 3}	(N, H, W)	pixel intensity	batch of grayscale images	image
AbstractArray{<:Number, 3}	(H, W, C)	intensity in a channel	one RGB image	image
AbstractArray{<:Color{T, 3}, 2}	(H, W)	3-d value for all channels	one RGB image	image
AbstractArray{<:Color{T, 1}, 2}	(N, H, W)	pixel intensity	one grayscale image	batch
AbstractArray{<:Number, 2}	(H, W)	pixel intensity	one grayscale image	image

I came up with two ways to resolve the ambiguity, both of which are sort-of bad:

1. Just go with one of the two options and document it.
2. Support arrays of colors only, not numbers.

I could see this ambiguity take down the whole proposal. Yet, I would love if we could figure it out.

Mixing batches and images

Now, it is possible to do

augment((img1, img2), pl)

which applies the same operations to both img1 and img2. On the contrary, this

augmentbatch(batch, pl)

applies possibly different operations to different images in batch (assuming the non-mutating version of augmentbatch! exists). Consequently, the following would not be well-defined:

augment((img1, img2, batch), pl)

There is a contradiction in it: "apply the same operations to img1, img2, and batch, and also apply different operations to images in batch".

For this reason, I think we should disallow mixing batches and images on the input.

Dealing with semantic wrappers

We would require the following to work:

# Augment an image
augment(img, pl)
# Augment an image and its mask (same operations applied to both)
augment(img => mask, pl)
# An alternative to the previous
augment((img, Mask(mask)), pl)
# Generalization of the previous for more inputs
augment((img, Mask(mask), KeyPoints(kp)), pl)

# Augment a batch of images
augment(batch, pl)
# Augment a batch of images and their masks 
# (same operations applied to corresponding image-mask pairs)
augment(batch => masks, pl)
# An alternative to the previous
augment((batch, Mask(masks)), pl)
# Generalization of the previous
# Probably length(batch) == length(masks) == length(kps)
augment((batch, Mask(masks), KeyPoints(kps)), pl)

Questions for discussion

I am interested in any opinion of yours, but here I list two questions to start a discussion:

1. Does it look desirable to you?
2. How do you think we should deal with the input ambiguity?

Change `ImageCore.showcoloranttype` to `ColorTypes.showcoloranttype`

I just noticed this upstream change when running the tests. looks like an easy fix.

Semantic wrapper: `Label`

Introduce a new semantic wrapper, Label, which represents a target class label of an instance. No operations are applied to this wrapper.

unnecessary type promotion

Hi, playing around with the package I just saw that it is often not type stable, e.g. putting in an image containing Float32 it returns a Float64 image. Affected functions I have tested:

Resize
Rotate
Shear
Zoom

I think working on this issue would increase the performance even more.

Edit: Example:

raw = rand(Float32,8,100,100)
img = view(reinterpret(SVector{8,Float32}, raw), 1,:,:)  # now its an AbstractMatrix
imgcp = copy(img)  # for demo purposes
pl = Rotate(123)
aug = augment(imgcp, pl)
typeof(aug) == typeof(imgcp)

CI fails

Hello,

Currently CI is falling, it seems it was caused by the ImageTransformations v0.8.8 release. All tests have passed when I downgraded it to v0.8.7. I guess the commit JuliaImages/ImageTransformations.jl@a4f2190 has changed the result of invwarpedview.

julia> square = [0.1 0.2 0.3; 0.4 0.5 0.6; 0.7 0.6 0.9]
julia> x = Augmentor.prepareaffine(square)
julia> m = Augmentor.toaffinemap(Rotate90(), square)

With ImageTransformations v0.8.7:

julia> Augmentor.invwarpedview(x, m)
3×3 InvWarpedView(extrapolate(interpolate(::Array{Float64,2}, BSpline(Interpolations.Linear())), Flat()), AffineMap([6.123233995736766e-17 -1.0; 1.0 6.123233995736766e-17], [4.0, 0.0])) with eltype Float64 with indices 1:3×1:3:
0.3 0.6 0.9
0.2 0.5 0.6
0.1 0.4 0.7

With ImageTransformations v0.8.8:

julia> Augmentor.invwarpedview(x, m)
4×4 InvWarpedView(extrapolate(interpolate(::Array{Float64,2}, BSpline(Interpolations.Linear())), Flat()), AffineMap([6.123233995736766e-17 -1.0; 1.0 6.123233995736766e-17], [4.0, 0.0])) with eltype Float64 with indices 1:4×1:4:
 0.3  0.6  0.9  0.9
 0.2  0.5  0.6  0.6
 0.1  0.4  0.7  0.7
 0.1  0.4  0.7  0.7

Adding non-geometric augmentations

Hi

I wanted to ask whether non-geometric augmentations would fit into Augmentor.jl

Especially things like brightness, contrast, white balance, etc.
https://github.com/aleju/imgaug has a quite wide array of image augmentations. All of them is probably a bit overkill, but a few would be useful I think.

If you think this would fit into the package I would start on adding a few additional augmentations.

Update reference files

The merge of JuliaLang/METADATA.jl#11972 will break some existing files

tweight specialization on FixedPoint

some tests in tst_rotation.jl are broken because the inferred weight type is incorrect. This might be due to recent upgrades in Interpolation and FixedPointNumbers.

julia> img = Gray{N0f8}[0.1 0.2 0.3; 0.4 0.5 0.6; 0.7 0.6 0.9];

julia> Augmentor.applyeager(Rotate(90), img); # eltype Gray{N0f8}

julia> Augmentor.applyeager(Rotate(90), N0f8.(img)); # eltype Float64 # not expected

A quick fix can be found in JuliaImages/ImageTransformations.jl#89 , but in the meantime, we need another patch in Augmentor to keep backward compatibility to old ImageTransformation versions.

random failure OffsetArray tests for Scale and Zoom

This bug is identified during the upgrade of Augmentor #31 Since that PR is growing bigger and becoming hard to review, it's better to open an issue here and fix it separately.

julia> using Augmentor, ImageCore, OffsetArrays, ImageDistances

julia> img = rand(Gray{N0f8}, 64, 64);

julia> out1 = Augmentor.applyeager(Scale(1.5), img);

julia> out2 = Augmentor.applyeager(Scale(1.5), OffsetArray(img, -1, -1));

julia> out1 == out2
false

julia> euclidean(out1, out2)
0.044884406f0

This bug might be related to upstream packages: Interpolation and ImageTransformation.jl

Update: downscale (e.g., Scale(0.2) ) is not affected.

rewrite the examples and simplify README.md

I would like to simplify README.md a bit and move The performance aspect to the documentation.

More examples are needed to explain how this package is integrated with Flux and MLDataUtils.

Interact.jl is not under very active development; actually, I've never successfully built it in my mac. We'd probably need to rewrite https://evizero.github.io/Augmentor.jl/generated/mnist_elastic/ with something that's runnable.

Edit:

The README is simplified a lot with #71

integration with Flux

Hi @Evizero @johnnychen94 ,
I was wondering how to integrate Augmentor in a Flux pipeline. I can think of two options:

1. once the DataLoader in FluxML/Flux.jl#1051 is merged, we can extend it with a transforms option taking an Augmentor's pipeline, and have Augmentor as a Flux dependency.
2. keep Augmentor as a separate entity and just showcase its use in model-zoo examples, e.g. Cifar10 standard data augmentation.

I think the first option would make for a more simple and streamlined user experience. What do you think?

Is the example here
https://github.com/Evizero/Augmentor.jl/blob/master/examples/mnist_knet.jl
still a valid template?

Can not install on Julia 1.0.1

v1.0) pkg> add Augmentor
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Augmentor [0612f1b9]:
 Augmentor [0612f1b9] log:
 ├─possible versions are: [0.0.1-0.0.2, 0.1.0, 0.2.0, 0.3.0-0.3.1, 0.4.0, 0.5.0] or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions [0.0.1-0.0.2, 0.1.0, 0.2.0, 0.3.0-0.3.1, 0.4.0, 0.5.0]
 └─restricted by julia compatibility requirements to versions: uninstalled — no versions left
(v1.0) pkg> dev Augmentor
   Cloning git-repo `https://github.com/Evizero/Augmentor.jl.git`
  Updating git-repo `https://github.com/Evizero/Augmentor.jl.git`
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ShowItLikeYouBuildIt [9966252f]:
 ShowItLikeYouBuildIt [9966252f] log:
 ├─possible versions are: [0.0.1, 0.1.0-0.1.1, 0.2.0] or uninstalled
 ├─restricted to versions 0.0.0-* by Augmentor [0612f1b9], leaving only versions [0.0.1, 0.1.0-0.1.1, 0.2.0]
 │ └─Augmentor [0612f1b9] log:
 │   ├─possible versions are: 0.5.0 or uninstalled
 │   └─Augmentor [0612f1b9] is fixed to version 0.5.0+
 └─restricted by julia compatibility requirements to versions: uninstalled — no versions left

Clean up `gh-pages` branch

I noticed that the URL https://evizero.github.io/Augmentor.jl/, which is in the About section of the repository takes users to an obsolete documentation. The correct URL is either https://evizero.github.io/Augmentor.jl/stable/ or https://evizero.github.io/Augmentor.jl/dev/.

I investigated further and noticed that the gh-pages branch contains even more obsolete content. I listed the author dates for each top-level file/directory using git ls-tree --name-only HEAD | while read filename; do echo "$(git log -1 --format="%ad" -- $filename) $filename"; done:

Sun Feb 4 20:34:49 2018 +0100 LICENSE
Mon Jul 9 09:22:54 2018 +0200 assets
Sun Feb 4 20:34:49 2018 +0100 background
Sun Aug 8 06:48:07 2021 +0000 dev
Mon Jul 9 09:33:08 2018 +0200 generated
Sun Feb 4 20:34:49 2018 +0100 gettingstarted
Sun Feb 4 20:34:49 2018 +0100 images
Sun Apr 8 21:05:15 2018 +0200 index.html
Sun Apr 8 21:05:15 2018 +0200 indices
Mon Jul 9 09:22:54 2018 +0200 interface
Mon Jul 9 09:22:54 2018 +0200 operations
Sun Feb 4 20:34:49 2018 +0100 search
Mon Jul 9 09:22:54 2018 +0200 search_index.js
Sun Aug 8 02:16:43 2021 +0000 stable
Sun Aug 8 02:16:43 2021 +0000 v0.6
Fri Sep 18 11:59:14 2020 +0000 v0.6.2
Mon May 17 13:17:17 2021 +0000 v0.6.3
Mon Jul 12 20:12:00 2021 +0000 v0.6.4
Sun Aug 8 02:16:43 2021 +0000 v0.6.5
Fri Sep 18 11:57:43 2020 +0000 versions.js

These items have not been updated in 2021:

LICENSE
assets
background
generated
gettingstarted
images
index.html         # keep this one
indices
interface
operations
search
search_index.js
v0.6.2             # keep this one
versions.js        # keep this one

I believe they (with the three exceptions) are not needed anymore, and we are free to dispose them.

I propose:

1. Change the top-level index.html so that it redirects to https://evizero.github.io/Augmentor.jl/stable/
2. rm -rf LICENSE assets background generated gettingstarted images indices interface operations search search_index.js

CI: Documentation build fails

Documentation build fails in a recent PR (see log). The issue comes from MLDatasets.jl, which uses Conda to install scipy. This is done by calling pyimport_conda from the PyCall.jl package. It seems that PyCall does not install conda on linux-based systems by default, and so the invocation of pyimport_conda results in the error that we see in our log (probably from here).

I could reproduce this error in a fresh Julia environment on my computer (with a linux-based system without conda installed).

(@v1.6) pkg> generate Test
  Generating  project Test:
    Test/Project.toml
    Test/src/Test.jl

(@v1.6) pkg> activate Test
  Activating environment at `/tmp/Test/Project.toml`

(Test) pkg> add MLDatasets
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
    Updating `/tmp/Test/Project.toml`
  [eb30cadb] + MLDatasets v0.5.9
    Updating `/tmp/Test/Manifest.toml`
  [9e28174c] + BinDeps v1.0.2
  [b99e7846] + BinaryProvider v0.5.10
  [a74b3585] + Blosc v0.7.0
  [e1450e63] + BufferedStreams v1.0.0
  [944b1d66] + CodecZlib v0.7.0
  [3da002f7] + ColorTypes v0.11.0
  [34da2185] + Compat v3.32.0
  [8f4d0f93] + Conda v1.5.2
  [124859b0] + DataDeps v0.7.7
  [53c48c17] + FixedPointNumbers v0.8.4
  [92fee26a] + GZip v0.5.1
  [f67ccb44] + HDF5 v0.15.6
  [cd3eb016] + HTTP v0.9.13
  [83e8ac13] + IniFile v0.5.0
  [692b3bcd] + JLLWrappers v1.3.0
  [682c06a0] + JSON v0.21.1
  [23992714] + MAT v0.10.1
  [eb30cadb] + MLDatasets v0.5.9
  [1914dd2f] + MacroTools v0.5.6
  [739be429] + MbedTLS v1.0.3
  [69de0a69] + Parsers v1.1.1
  [21216c6a] + Preferences v1.2.2
  [438e738f] + PyCall v1.92.3
  [189a3867] + Reexport v1.1.0
  [ae029012] + Requires v1.1.3
  [3bb67fe8] + TranscodingStreams v0.9.5
  [30578b45] + URIParser v0.4.1
  [5c2747f8] + URIs v1.3.0
  [81def892] + VersionParsing v1.2.0
  [0b7ba130] + Blosc_jll v1.21.0+0
  [0234f1f7] + HDF5_jll v1.12.0+1
  [5ced341a] + Lz4_jll v1.9.3+0
  [458c3c95] + OpenSSL_jll v1.1.10+0
  [3161d3a3] + Zstd_jll v1.5.0+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [83775a58] + Zlib_jll
  [8e850ede] + nghttp2_jll
  [3f19e933] + p7zip_jll
Precompiling project...
  1 dependency successfully precompiled in 3 seconds (37 already precompiled)

julia> using MLDatasets
ERROR: InitError: PyError (PyImport_ImportModule

The Python package scipy could not be imported by pyimport. Usually this means
that you did not install scipy in the Python version being used by PyCall.

PyCall is currently configured to use the Python version at:

/usr/bin/python3

and you should use whatever mechanism you usually use (apt-get, pip, conda,
etcetera) to install the Python package containing the scipy module.

One alternative is to re-configure PyCall to use a different Python
version on your system: set ENV["PYTHON"] to the path/name of the python
executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.

Another alternative is to configure PyCall to use a Julia-specific Python
distribution via the Conda.jl package (which installs a private Anaconda
Python distribution), which has the advantage that packages can be installed
and kept up-to-date via Julia.  As explained in the PyCall documentation,
set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,
To install the scipy module, you can use `pyimport_conda("scipy", PKG)`,
where PKG is the Anaconda package that contains the module scipy,
or alternatively you can use the Conda package directly (via
`using Conda` followed by `Conda.add` etcetera).

) <class 'ModuleNotFoundError'>
ModuleNotFoundError("No module named 'scipy'")

Stacktrace:
 [1] pyimport(name::String)
   @ PyCall ~/.julia/packages/PyCall/BD546/src/PyCall.jl:550
 [2] pyimport_conda(modulename::String, condapkg::String, channel::String)
   @ PyCall ~/.julia/packages/PyCall/BD546/src/PyCall.jl:708
 [3] pyimport_conda
   @ ~/.julia/packages/PyCall/BD546/src/PyCall.jl:707 [inlined]
 [4] __init__()
   @ MLDatasets ~/.julia/packages/MLDatasets/KMScX/src/MLDatasets.jl:62
 [5] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base ./loading.jl:696
 [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base ./loading.jl:782
 [7] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1020
 [8] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:936
 [9] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:923
during initialization of module MLDatasets

`ConvertEltype(Gray)` not working as expect anymore

It seems that for some reason or another, leaving the inner type open for Gray now causes fatal issues with inference. Will either have to force user to be specific with ConvertEltype(Gray{INNER}) or see if I can fix this.

support functor API

A simple use case:

pl = ElasticDistortion(6, scale=0.3, border=true) |>
            Rotate([10, -5, -3, 0, 3, 5, 10]) |>
            ShearX(-10:10) * ShearY(-10:10) |>
            CropSize(28, 28) |>
            Zoom(0.9:0.1:1.2)

pl(img) # instead of augment(img, pl)

I find this syntax easier to memorize and use, so I plan to do this when this package got updated.

Screenshot Storage

just images/gifs I don't want to have in my git history (or can't autogenerate with Documeter)

Rotation without changing size

Hi, I want to rotate the image using Augmentor.jl to do that, I took an sample image from the net. and applied following commands:

julia> using Augmentor
julia> using Images  # I use it to load jpg file, if it is possible with Augmentor I can use it too
julia> img = load("test-input.jpg")
julia> size(img)
(1600,2400)
julia> pl = Rotate(45)
julia> img_new = augment(img,pl)
julia> size(img_new)
(2830,2830)

Below, I put both the original image (img) and rotated one (img_new). I have two problems here. First of all, when I rotate it, the size of the images changes. Is it possible to keep it constant ? Secondly, why the image brokes at some points like under the keyboard? If possible I also want to resize the image before rotating it. How could it be integrated in this flow ?

Original Image:

Rotated Image:

Add translation operation

Add `RCropSize` for a randomly placed `CropSize`

for a bit more variation

Does Augmentor.jl support multispectral images?

Hi, I am frequently working with multispectral images (say 4-8 bands) and need to augment them. These usually come in the form of 3D arrays. In the source code I saw that the augmentation functions require img::AbstractMatrix - so at the first glance I cannot plug in plain arrays. Did I overlook something? Or how can I mangle my arrays into a form that can be processed? Tell me if you think this is better moved to discourse :)

`Either` may not work properly with `Pipeline`

For the following three seeming "equivalent" operations, the first one randomly changes the image size...

For img = testimage("cameraman"):

• augment(img, Either(Rotate90()) |> FlipY()) returns image of size (512, 512) or (513, 512)
• augment(img, Rotate90() |> FlipY()) returns image of size (512, 512)
• augment(augment(img, Either(Rotate90())), FlipY()) returns image of size (512, 512)

Segmentation tasks: augmentation of GT images

In the case of segmentation tasks, an input consists of an image and a ground-truth segmentation. To augment both the image and GT segmentation using the same operations, we can use

augment((img, gt), pipeline)

However, this cannot be used if the pipeline includes an operation that adjust the colors of images because it would damage the color encoding in the GT segmentations. As of now, we do not have such operations but non-geometric operations (such as changing contrast) were requested in #16, and PR #84 introduces first such operations.

Notice that we cannot just use

augment(img, pipeline)

because if the pipeline contains some affine operations, we want them applied on the GT segmentation too.

---

Generally, some operations (e.g., all affine operations) are desired to be applied on both the image and GT, while others (e.g., contrast adjustment) should be applied only on the image (and not GT). I think it raises two questions:

1. How to distinguish the image-only operations from image-and-gt operations?
2. How to adjust the API so it is clear whether the user passes only images, or images with GT segmentations?

I think that (1) could be resolved by introducing a new abstract type ImageOnlyOperation <: Operation. All operations that would damage GT segmentations would be subtypes of this abstract type. The existing type ImageOperation would denote operations that should be applied on everything (images and segmentations).

For (2), I think the API could be

augment(img, gt, pipeline)
augment(imgs, gts, pipeline) # length(imgs) == length(gts)

augment modes supported by different operations

This issue is for reference usage

Operation

Name	eager	lazy	affinemap	permute	view	stepview	affine	affineview
Augmentor.ImageOperation	✅	❌	❌	❌	❌	❌	❌	❌
AggregateThenMapFun	✅	✅	❌	❌	❌	❌	❌	❌
CombineChannels	❌	✅	❌	❌	❌	❌	❌	❌
ConvertEltype	✅	✅	❌	❌	❌	❌	❌	❌
MapFun	✅	✅	❌	❌	❌	❌	❌	❌
PermuteDims	✅	✅	❌	❌	❌	❌	❌	❌
Reshape	❌	✅	❌	❌	❌	❌	❌	❌
SplitChannels	❌	✅	❌	❌	❌	❌	❌	❌

ImageOperation

Name	eager	lazy	affinemap	permute	view	stepview	affine	affineview
Augmentor.AffineOperation	✅	✅	✅	❌	❌	❌	✅	✅
Augmentor.CacheImageInto	✅	✅	❌	❌	❌	❌	❌	❌
CacheImage	✅	❌	❌	❌	❌	❌	❌	❌
Crop	❌	✅	✅	❌	✅	✅	❌	✅
CropNative	❌	✅	✅	❌	✅	✅	❌	✅
CropRatio	❌	✅	✅	❌	✅	✅	❌	✅
CropSize	❌	✅	✅	❌	✅	✅	❌	✅
Either	✅	❌	❌	❌	❌	❌	❌	❌
ElasticDistortion	❌	✅	❌	❌	❌	❌	❌	❌
RCropRatio	❌	✅	✅	❌	✅	✅	❌	✅
Resize	✅	✅	✅	❌	❌	❌	❌	✅
Zoom	❌	✅	✅	❌	❌	❌	❌	✅

AffineOperation

Name	eager	lazy	affinemap	permute	view	stepview	affine	affineview
FlipX	✅	✅	✅	❌	❌	✅	✅	✅
FlipY	✅	✅	✅	❌	❌	✅	✅	✅
NoOp	❌	✅	✅	❌	✅	✅	✅	✅
Rotate	❌	✅	✅	❌	❌	❌	✅	✅
Rotate180	✅	✅	✅	❌	❌	✅	✅	✅
Rotate270	✅	✅	✅	✅	❌	❌	✅	✅
Rotate90	✅	✅	✅	✅	❌	❌	✅	✅
Scale	❌	✅	✅	❌	❌	❌	✅	✅
ShearX	❌	✅	✅	❌	❌	❌	✅	✅
ShearY	❌	✅	✅	❌	❌	❌	✅	✅

script

using Augmentor

check_list = ["lazy",
              "uses_affinemap",
              "eager",
              "permute",
              "view",
              "stepview",
              "affine",
              "affineview",
              ]

op_list = [AggregateThenMapFun,
           CombineChannels,
           ConvertEltype,
           MapFun,
           PermuteDims,
           Reshape,
           SplitChannels,
           # ImageOperation
           Augmentor.ImageOperation,
           Augmentor.CacheImageInto,
           CacheImage,
           Crop,
           CropNative,
           CropRatio,
           CropSize,
           Either,
           ElasticDistortion,
           RCropRatio,
           Resize,
           Zoom,
           # AffineOperation
           Augmentor.AffineOperation,
           FlipX,
           FlipY,
           NoOp,
           Rotate,
           Rotate180,
           Rotate270,
           Rotate90,
           Scale,
           ShearX,
           ShearY,
]

bool2emoji(x) = if x "✅" else "❌" end


open("out.md", "w") do io
    # header
    print(io, "| Name")
    for fn_name in check_list
        print(io, " | ", fn_name)
    end
    println(io, " |")

    print(io, "| ---")
    for _ in check_list
        print(io, " | --- ")
    end
    println(io, " |")

    for op in op_list
        print(io, "| ", op)
        for fn_name in check_list
            if isdefined(Augmentor, Symbol("supports_"*fn_name))
                fn_name = Symbol("supports_"*fn_name)
            else
                fn_name = Symbol(fn_name)
            end
            @show fn_name

            rst = @eval ($fn_name)($op)
            rst = bool2emoji(rst)
            print(io, " | ", rst)
        end
        println(io, " |")
    end
end

document the performance

The only piece of words on performance is only available in README.md and that's pretty old.

make `Crop(indicies(A))` work for `Base.OneTo`

Integrate DataLoaders with `augment`

https://github.com/lorenzoh/DataLoaders.jl has a very nice implementation to support image loading/augmentation with background threads, so I think we can maybe take advantage of it, and maybe deprecate augment_batch in favor of this.

Inconsistent behaviour for `applyeager` concerning indices

I just noticed sometimes that the fallback gets rid of arbitrary indices while some implementations do not (and instead return a OffsetArray). This should be streamlined

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Applying `ConvertEltype` lazily after affine operations can cause misaligned images

Just making a note to myself. Will have to investigate

Add size preserving rotate (i.e. rotate + crop)

In similar spirit to Zoom. I originally felt it would be redundant, but chaining them manually gets old pretty quick

the function in MapFun should be a function that operates an image

Currently, we need to pass a function that operates on single pixel to MapFun, many operations become invalid because of this; many available image processing algorithms accept only image (not pixel).

Pixel operations can be just broadcasted to become valid, e.g.,

- augment(img, MapFun(px -> px - RGBA(0.5, 0.3, 0.7, 0.0)))
+ augment(img, MapFun(px -> px .- RGBA(0.5, 0.3, 0.7, 0.0)))

A usage of this could be:

pl = ConvertEltype(Gray{Float32}) |>
     MapFun(AdditiveWhiteGaussianNoise(0.1)) |> # ImageNoise.jl
     Either(NoOp(), Rotate90(), Rotate180(), Rotate270()) |>
     Either(0.5=>NoOp(), 0.25=>FlipX(), 0.25=>FlipY())

3D Data Augmentation

This is either a question, or a feature request:

Does Augmentor.jl allow for 3D data augmentation? I tried to pass a 3D array but got an error message.
If this is not possible right now: Would it be complicated to add support for that?

Thanks,

Tobi

Info about upcoming removal of packages in the General registry

As described in https://discourse.julialang.org/t/ann-plans-for-removing-packages-that-do-not-yet-support-1-0-from-the-general-registry/ we are planning on removing packages that do not support 1.0 from the General registry. This package has been detected to not support 1.0 and is thus slated to be removed. The removal of packages from the registry will happen approximately a month after this issue is open.

To transition to the new Pkg system using Project.toml, see https://github.com/JuliaRegistries/Registrator.jl#transitioning-from-require-to-projecttoml.
To then tag a new version of the package, see https://github.com/JuliaRegistries/Registrator.jl#via-the-github-app.

If you believe this package has erroneously been detected as not supporting 1.0 or have any other questions, don't hesitate to discuss it here or in the thread linked at the top of this post.

Mind interpolation type when resizing / rotating masks

This issue is more a reminder to myself, haven't tested yet. But with the new feature for masks we have to keep in mind that masks could come as e.g. color coded and each exact value has a specific meaning. Therefore we must use nearest interpolation when rotating / resizing etc. I think, so far eg imresize uses linear, which could introduce label noise.

Proper Attribution

Because this package makes heavy use of CoordinateTransformations.jl, ImageTransformations.jl, IdentityRanges.jl, and PermutedDimsArray, I thought it would be polite to make you aware of this package's existence (which I am to register soon).

I hope the way I attribute the packages at the end of the readme is to everyone's satisfaction. If not please let me know.

@timholy, @andyferris, @c42f

Add `channel_dims` argument to `CombineChannels`/`SplitChannels`

Internally, CombineChannels uses ImageCore.colorview to do the work, which requires the data layout to be CHW. When deal with deep learning frameworks, it's often the case that we need to handle HWC/WHC layout.

Previously, CombineChannels(RGB) interpret rand(3, 4, 4) as a 4x4 RGB image and errors for rand(4, 4, 3). And SplitChannels() unconditionally interpret rand(RGB, 4, 4) as 3x4x4 numerical array.

So the idea is to let CombineChannels(RGB, 3) interpret rand(4, 4, 3) as a 4x4 RGB image, and to let SplitChannels(3) interpret rand(RGB, 4, 4) as 4x4x3 numerical array.

Edit: When I opened this issue I didn't realize that PermuteDims is also an Augmentor operation; there are two very similar names in Base permutedims and PermutedDimsArray. PermuteDims is no doubt more generic than channel_dims argument proposed here. Thus this proposal is only to make things more convenient for a very specific usage; it doesn't fill any missing functionalities.

Resize does not seem to work with augmentbatch!

Maybe I just don't understand how to use it...

Makes most sense for me as a user to only specify heigh and width:

julia> augmentbatch!(zeros(6,6,3,2), ones(4,4,3,2), Resize(6,6))
ERROR: MethodError: no method matching applyaffine(::Resize{2}, ::ImageTransformations.InvWarpedView{Float64,3,Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}},CoordinateTransformations.AffineMap{UniformScaling{Float64},SArray{Tuple{3},Float64,1,3}},Tuple{UnitRange{Int64},UnitRange{Int64},UnitRange{Int64}},CoordinateTransformations.AffineMap{UniformScaling{Bool},SArray{Tuple{3},Float64,1,3}},Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}}}, ::Nothing)
Closest candidates are:
  applyaffine(::Augmentor.AffineOperation, ::AbstractArray, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:77
  applyaffine(::Either, ::AbstractArray, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\either.jl:203
  applyaffine(::Augmentor.Operation, ::AbstractArray) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60  
  ...
Stacktrace:
 [1] applyaffineview(::Resize{2}, ::ImageTransformations.InvWarpedView{Float64,3,Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}},CoordinateTransformations.AffineMap{UniformScaling{Float64},SArray{Tuple{3},Float64,1,3}},Tuple{UnitRange{Int64},UnitRange{Int64},UnitRange{Int64}},CoordinateTransformations.AffineMap{UniformScaling{Bool},SArray{Tuple{3},Float64,1,3}},Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}}}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:72
 [2] applylazy(::Resize{2}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\resize.jl:74
 [3] applylazy at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60 [inlined]
 [4] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\codegen.jl:76 [inlined]
 [5] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [6] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [7] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:126 [inlined]
 [8] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Tuple{Resize{2}}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:112
 [9] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Resize{2}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:109
 [10] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:105 [inlined]
 [11] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:94 [inlined] (repeats 2 times)     
 [12] augmentbatch!(::Array{Float64,4}, ::Array{Float64,4}, ::Resize{2}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:85
 [13] top-level scope at none:0

This is more consistent with Crop, but what should happen if dimension 3 is resized?

julia> augmentbatch!(zeros(6,6,3,2), ones(4,4,3,2), Resize(6,6,3))
ERROR: MethodError: no method matching +(::CartesianIndex{3}, ::CartesianIndex{2})
Closest candidates are:
  +(::Any, ::Any, ::Any, ::Any...) at operators.jl:529
  +(::CartesianIndex{N}, ::CartesianIndex{N}) where N at multidimensional.jl:110
Stacktrace:
 [1] applyaffineview(::Resize{3}, ::ImageTransformations.InvWarpedView{Float64,3,Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}},CoordinateTransformations.AffineMap{UniformScaling{Float64},SArray{Tuple{3},Float64,1,3}},Tuple{UnitRange{Int64},UnitRange{Int64},UnitRange{Int64}},CoordinateTransformations.AffineMap{UniformScaling{Bool},SArray{Tuple{3},Float64,1,3}},Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}}}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\resize.jl:83
 [2] applylazy(::Resize{3}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\resize.jl:74
 [3] applylazy at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60 [inlined]
 [4] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\codegen.jl:76 [inlined]
 [5] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [6] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [7] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:126 [inlined]
 [8] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Tuple{Resize{3}}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:112
 [9] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Resize{3}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:109
 [10] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:105 [inlined]
 [11] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:94 [inlined] (repeats 2 times)     
 [12] augmentbatch!(::Array{Float64,4}, ::Array{Float64,4}, ::Resize{3}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:85
 [13] top-level scope at none:0

This does not make sense at all, but I tried it just in case:

julia> augmentbatch!(zeros(6,6,3,2), ones(4,4,3,2), Resize(6,6,3,2))
ERROR: MethodError: no method matching applyaffine(::Resize{4}, ::ImageTransformations.InvWarpedView{Float64,3,Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}},CoordinateTransformations.AffineMap{UniformScaling{Float64},SArray{Tuple{3},Float64,1,3}},Tuple{UnitRange{Int64},UnitRange{Int64},UnitRange{Int64}},CoordinateTransformations.AffineMap{UniformScaling{Bool},SArray{Tuple{3},Float64,1,3}},Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}}}, ::Nothing)
Closest candidates are:
  applyaffine(::Augmentor.AffineOperation, ::AbstractArray, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:77
  applyaffine(::Either, ::AbstractArray, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\either.jl:203
  applyaffine(::Augmentor.Operation, ::AbstractArray) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60  
  ...
Stacktrace:
 [1] applyaffineview(::Resize{4}, ::ImageTransformations.InvWarpedView{Float64,3,Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}},CoordinateTransformations.AffineMap{UniformScaling{Float64},SArray{Tuple{3},Float64,1,3}},Tuple{UnitRange{Int64},UnitRange{Int64},UnitRange{Int64}},CoordinateTransformations.AffineMap{UniformScaling{Bool},SArray{Tuple{3},Float64,1,3}},Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}}}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:72
 [2] applylazy(::Resize{4}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\resize.jl:74
 [3] applylazy at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60 [inlined]
 [4] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\codegen.jl:76 [inlined]
 [5] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [6] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [7] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:126 [inlined]
 [8] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Tuple{Resize{4}}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:112
 [9] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Resize{4}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:109
 [10] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:105 [inlined]
 [11] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:94 [inlined] (repeats 2 times)     
 [12] augmentbatch!(::Array{Float64,4}, ::Array{Float64,4}, ::Resize{4}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:85
 [13] top-level scope at none:0

Ambiguity with RotMatrix

Hi

I suddenly become the following error when using Augmentor

MethodError: Rotations.RotMatrix{2,Float64,4}(::StaticArrays.SArray{Tuple{2,2},Float64,2,4}) is ambiguous. Candidates:
  (::Type{Rotations.RotMatrix{N,T,L}})(x::AbstractArray) where {N, T, L} in Rotations at /home/paethon/.julia/v0.6/Rotations/src/core_types.jl:78
  (::Type{SA})(a::StaticArrays.StaticArray) where SA<:StaticArrays.StaticArray in StaticArrays at /home/paethon/.julia/v0.6/StaticArrays/src/convert.jl:4
Possible fix, define
  (::Type{Rotations.RotMatrix{N,T,L}})(::StaticArrays.StaticArray)

Reason is rotations.jl line 324

recenter(RotMatrix(deg2rad(Float64(safe_rand(op.degree)))), center(img))

Unfortunately I am not yet confident enough to fix this myself

let Either also accept pipeline

The following doesn't work right now

Either(
    NoOp(),
    FlipY(),
    Rotate90() |> FlipY(),
)

Flip does not work with augmentbatch

julia> augmentbatch!(zeros(4,4,3,2), ones(4,4,3,2), FlipX())
ERROR: MethodError: no method matching toaffinemap(::NoOp, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true})
Closest candidates are:
  toaffinemap(::Augmentor.Operation, ::Any, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:48     
  toaffinemap(::NoOp, ::AbstractArray{T,2} where T) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\noop.jl:17
  toaffinemap(::Either, ::Any, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\either.jl:171
Stacktrace:
 [1] prepareaffine(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:34
 [2] applylazy_fallback(::FlipX, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:108
 [3] _applylazy(::FlipX, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:100
 [4] applylazy(::FlipX, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:85
 [5] applylazy at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60 [inlined]
 [6] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\codegen.jl:76 [inlined]
 [7] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [8] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [9] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:126 [inlined]
 [10] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Tuple{FlipX}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:112
 [11] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::FlipX) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:109
 [12] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:105 [inlined]
 [13] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:94 [inlined] (repeats 2 times)
 [14] augmentbatch!(::Array{Float64,4}, ::Array{Float64,4}, ::FlipX) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:85
 [15] top-level scope at none:0

With an implementation for the ND identity affine map the same issue appears for Flip:

julia> using LinearAlgebra

julia> using StaticArrays

julia> Augmentor.toaffinemap(::NoOp, img::AbstractArray{T,N}) where {T,N} = Augmentor.AffineMap(I, @SVector(zeros(N)))

julia> augmentbatch!(zeros(4,4,3,2), ones(4,4,3,2), FlipX())
ERROR: MethodError: no method matching toaffinemap(::FlipX, ::ImageTransformations.InvWarpedView{Float64,3,Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}},CoordinateTransformations.AffineMap{UniformScaling{Float64},SArray{Tuple{3},Float64,1,3}},Tuple{UnitRange{Int64},UnitRange{Int64},UnitRange{Int64}},CoordinateTransformations.AffineMap{UniformScaling{Bool},SArray{Tuple{3},Float64,1,3}},Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}}})
Closest candidates are:
  toaffinemap(::Augmentor.Operation, ::Any, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:48     
  toaffinemap(::NoOp, ::AbstractArray{T,N}) where {T, N} at none:1
  toaffinemap(::FlipX, ::AbstractArray{T,2} where T) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\flip.jl:59
  ...
Stacktrace:
 [1] toaffinemap at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:48 [inlined]
 [2] applyaffine(::FlipX, ::ImageTransformations.InvWarpedView{Float64,3,Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}},CoordinateTransformations.AffineMap{UniformScaling{Float64},SArray{Tuple{3},Float64,1,3}},Tuple{UnitRange{Int64},UnitRange{Int64},UnitRange{Int64}},CoordinateTransformations.AffineMap{UniformScaling{Bool},SArray{Tuple{3},Float64,1,3}},Interpolations.Extrapolation{Float64,3,Interpolations.BSplineInterpolation{Float64,3,SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Interpolations.BSpline{Interpolations.Linear},Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Interpolations.BSpline{Interpolations.Linear},Interpolations.Flat{Nothing}}}, ::Nothing) at 
E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:77
 [3] applylazy_fallback(::FlipX, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:108
 [4] _applylazy(::FlipX, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:100
 [5] applylazy(::FlipX, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:85 
 [6] applylazy at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60 [inlined]
 [7] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\codegen.jl:76 [inlined]
 [8] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [9] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [10] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:126 [inlined]
 [11] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Tuple{FlipX}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:112
 [12] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::FlipX) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:109
 [13] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:105 [inlined]
 [14] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:94 [inlined] (repeats 2 times)     
 [15] augmentbatch!(::Array{Float64,4}, ::Array{Float64,4}, ::FlipX) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:85
 [16] top-level scope at none:0

With an ND implementation for flip it works:

julia> Augmentor.toaffinemap(::FlipX, img::AbstractArray{T,N}) where {T,N} = Augmentor.recenter(SMatrix{N,N}(Matrix{typeof(0.)}(I, N,N) .* [1, -1, ones(N-2)...]), Augmentor.center(img))

julia> augmentbatch!(zeros(4,4,3,2), ones(4,4,3,2), FlipX())
4×4×3×2 Array{Float64,4}:
...

It still needs an ND implementation of applystepview to work inside an Either:

julia> augmentbatch!(zeros(4,4,3,2), ones(4,4,3,2), FlipX(0.5))
ERROR: MethodError: no method matching applystepview(::FlipX, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Nothing)
Closest candidates are:
  applystepview(::NoOp, ::AbstractArray, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\noop.jl:27  
  applystepview(::FlipX, ::AbstractArray{T,2} where T, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\flip.jl:65
  applystepview(::Crop, ::AbstractArray, ::Any) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\crop.jl:80  
  ...
Stacktrace:
 [1] applystepview at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60 [inlined]
 [2] applystepview(::Either{2,Tuple{FlipX,NoOp}}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Int64) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\either.jl:204
 [3] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\either.jl:146 [inlined]
 [4] applylazy at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operations\either.jl:146 [inlined]
 [5] applylazy at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\operation.jl:60 [inlined]
 [6] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\codegen.jl:76 [inlined]
 [7] macro expansion at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [8] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:130 [inlined]
 [9] _augment_avoid_eager at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:126 [inlined]
 [10] augment!(::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true}, ::Tuple{Either{2,Tuple{FlipX,NoOp}}}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:112
 [11] augment! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augment.jl:109 [inlined]
 [12] augmentbatch!(::CPU1{Nothing}, ::MLDataPattern.ObsView{SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Array{Float64,4},LearnBase.ObsDim.Last}, ::MLDataPattern.ObsView{SubArray{Float64,3,Array{Float64,4},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},Int64},true},Array{Float64,4},LearnBase.ObsDim.Last}, ::Either{2,Tuple{FlipX,NoOp}}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:105
 [13] augmentbatch! at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:94 [inlined] (repeats 2 times)     
 [14] augmentbatch!(::Array{Float64,4}, ::Array{Float64,4}, ::Either{2,Tuple{FlipX,NoOp}}) at E:\Programs\julia\.julia\packages\Augmentor\kkcKe\src\augmentbatch.jl:85
 [15] top-level scope at none:0

This seems to work:

julia> Augmentor.applystepview(::FlipX, img::AbstractArray{T,N}, param) where {T,N} = Augmentor.indirect_view(img, (1:1:size(img,1), size(img,2):-1:1, (1:1:size(img,i) for i in 3:N)...))

julia> augmentbatch!(zeros(4,4,3,2), ones(4,4,3,2), FlipX(0.5))
4×4×3×2 Array{Float64,4}:
...