keras-team/keras-cv

Industry-strength Computer Vision workflows with Keras

keras-cv's Issues

GaussianBlur preprocessing layer

Write up style guide

End to end model.fit() performance testing w/ COCO metrics

Some comments on this:

• we can use a Keras callback to track the runtimes of each step
• we must omit the first batch, as this will include the Keras compilation step
• we can test the model runtime with/without each metric, this will give us a reference as to how expensive the metrics are. No metric should increase runtime by more than a factor of 2x

RandomColorAffine preprocessing layer

https://keras.io/examples/vision/semisupervised_simclr/

Efficientnet block/layer

Channel/Spatial/Element wise Attention Modules

In the timm package, it provides some soft attention modules to building network blocks and I think it's a good fit here, for example:

• Squeeze-and-Excitation (SE): https://arxiv.org/abs/1709.01507 Cited by 10411
• CBAM: https://arxiv.org/abs/1807.06521 Cited by 3530
• Residual Attention https://arxiv.org/abs/1704.06904 Cited by 2177
• Dual Attention https://arxiv.org/abs/1809.02983 Cited by 1937
• Selective Kernel (SK) https://arxiv.org/abs/1903.06586 Cited by 667
• GCNet https://arxiv.org/abs/1904.11492 Cited by 598
• SAN https://ieeexplore.ieee.org/document/8954252 Cited by 566
• ECA-Net https://arxiv.org/abs/1910.03151 Cited by 471
• BAM https://arxiv.org/abs/1807.06514 Cited by 365
• A2-Nets https://arxiv.org/abs/1810.11579 Cited by 278
• Bottleneck Transformer https://arxiv.org/abs/2101.11605 Cited by 147
• Lambda Layer: https://arxiv.org/abs/2102.08602 Cited by 67
• SGENet https://arxiv.org/abs/1905.09646 Cited by 51
• HaloNet https://arxiv.org/abs/2103.12731 Cited by 51
• CoAtNet https://arxiv.org/abs/2106.04803 Cited by 35 ( < 50 current)

and many others.

Look into some quickstart guides

for example:
"Looking to preprocess image data for classification? Try this CutMix + RandAugment pipeline"

Random Erasing as a KPL

Random Erasing [1] is another essential augmentation transform for controlling the amount of regularization when training models on ImageNet. It's also used by the recent line of papers [2, 3]. Short explanation: https://paperswithcode.com/method/random-erasing.

[1] Random Erasing: https://arxiv.org/abs/1708.04896
[2] ResNet Strikes Back: https://arxiv.org/abs/2110.00476
[3] A ConvNet for the 2020s: https://arxiv.org/abs/2201.03545

Can you tell me the road map of keras-cv？

I'm very interested in this warehouse. Is it an upgraded version of keras app? Like torchvision?

Simple Copy-Paste Augmentation (Instance Segmentation)

Demo:

Ref: https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/copy_paste

ResNet34 architecture

AutoAugment image augmentation

Paper: https://arxiv.org/abs/1805.09501 Cited by 832
Ref code. https://github.com/tensorflow/models/blob/ded32f0500604928e52e27fd3f678e694e5133b7/official/vision/image_classification/augment.py#L731

Create nightly PyPi release

The nightly release will be published every commit to master via a GitHub action

Document weight management for KerasCV

This includes:

• default values
• code locations
• GCS upload
• how to reproduce weights
• where to store reproduction scripts

Anchor free models

Please evaluate to extract common components/utils that could support Anchor free models:
tensorflow/hub#424

See also:
google-ai-edge/mediapipe#495 (comment)
google-coral/edgetpu#51

Implement numerical correctness test for COCO metrics

This can be done by hard coding some predictions, running through cocoeval.py, then testing the known values against the Keras COCO metrics

Document "Auxiliary Tasks"

These are not the focus of KerasCV, but are in scope of creating state of the art models for one of the focus tasks.

A great example of this is model visualization

Create documentation for KerasCV API

We can likely autogenerate this

Standardized / preferred way to implement blocks and models.

Given the models requirements are being gathered in the discussion, is there a preferred way to implement them?

There are multiple ways, to implement blocks and models:

1. keras.applications way -> Models and blocks are functional Example.
2. Model Garden way -> Model is functional, but blocks are layer subclasses. (even though the model is a direct subclass, it does not override call method).
3. Blocks are layer subclasses, and models are model subclasses of keras.layers.Layer and keras.Model respectively. Both implement call method.

Each way has it's own benefits and drawbacks. Is one of the above preferred? Or maybe something entirely different?

RandomErasing preprocessing layer

Explore model visualization tools

e.x.

https://tf-explain.readthedocs.io/en/latest/

TF Addons CV components

Are you interested to cover or migrate some of the CV components in the TF Addons namespace?

I suppose the starting point could be to review tf.addons.image as we are already duplicated in this repo things like coutout/randomcoutout etc..:
https://www.tensorflow.org/addons/api_docs/python/tfa/image

P.s. This is a parallel ticket of keras-team/keras-nlp#11

Reorder channel layer for smooth native RGB - BGR

This is a commonly performed operation in image processing making it a good fit for keras-cv. While this is not a super complicated operation (it's really just a lambda into a tf.gather) the resulting code is more readable. Let's include this in keras-cv.

Originally discussed here: keras-team/keras#15705

Re the design of the layer signature...

The current proposal is to implement a layer that looks like this:

rgb2bgr_layer = tf.keras.layers.ReorderChannel(order=[2, 1, 0], axis=-1)

Perhaps we actually may want to use a einsum inspired syntax, such as:

rgb2bgr_layer = tf.keras.layers.ReorderChannel('rgb->bgr', axis=-1)

This would be really readable to anyone stumbling upon a new codebase, and generalizes quite well to any number of channels.

Please feel free to comment below with any additional thoughts

AugMix image augmentation

Paper: https://arxiv.org/abs/1912.02781 Cited by 308
Code:

• src 1: https://github.com/google-research/augmix
• src 2: https://github.com/AakashKumarNain/AugMix_TF2

Implement profiling for COCORecall

Understanding which for loops are costing the most will guide the optimization/vectorization effort. Profiling can be done using the TensorFlow profiler:

https://www.tensorflow.org/guide/profiler

Purpose, scope and road-map of this repository

I have explained my questions in more detail here. I would appreciate if you could answer them for this repository (i.e. keras-cv) as well.

Vectorize/Optimize the COCORecall metric

Lots of the for loops in the COCORecall update_state method could be reduced to tf.einsum() calls. Additionally, we should be able to vectorize area computation in the iou function, along with some other operations in the overall computation of the metric. tf.TensorArrays are quite slow, so we should look to remove those if possible.

RandAug preprocessing layer

https://keras.io/examples/vision/randaugment/

RandomCutMix, RandomMixUp augmentation

These two well-known and very effective augmentation methods are widely used among ML practitioners. In keras, we can find some general augmentation . And these two augmentation methods can be included as advanced augmentation layers.

CutMix - Paper - Cited by ~ 865
MixUp - Paper - Cited by ~ 2675

Let's add these to
keras_cv.layers.RandomCutMix and keras_cv.layers.RandomMixUp.

CutOut augmentation

Add `ResNeXt [50, 101]` to keras.applications

System information.

TensorFlow version (you are using): 2.5
Are you willing to contribute it (Yes/No) : Yes

Describe the feature and the current behavior/state.

ResNeXt is a well-known classification model that is weirdly missing from the keras.applications. The idea that is presented here is great and I think it's fit for the model member in the keras package.

Will this change the current api? How?
Yes.

from tensorflow.keras.applications.resnext50 import ResNeXt50
from tensorflow.keras.applications.resnext101 import ResNeXt101

Who will benefit from this feature?
ML engineers and researcher who uses tf.keras.

Others
Others implementation: https://github.com/qubvel/classification_models

Integrate pre-commit hooks

based on tensorflow/tensorflow#48371

fill_utils tests

This module came about due to some shared functionality that was originally tested within the layers. As long as this is shared, it would be good to unit test it.

ResNeXT block/layer

SaltAndPepper preprocessing layer

CutMix augmentation for Object Detection task.

[Reposting from here as a future reference for potential contributor].

CutMix - Paper - Cited by ~ 865
MixUp - Paper - Cited by ~ 2675

---

Currently, this aug can be applied to classification tasks but as the kerascv sets the target to general vision task, that's why to support vision tasks like object detection, we may need to add utility to accept the bbox_params argument too.

example of mixup-object-detection - region=full-images

example of mixup-object-detection - region=random

example of cutmix-object-detection

ref: https://www.kaggle.com/ankursingh12/data-augmentation-for-object-detection
ref: https://www.kaggle.com/shonenkov/oof-evaluation-mixup-efficientdet

Add Swin-Transformer to keras.applications

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with the documentation (for small docs fixes please send a PR instead).
The form below must be filled out.

Here's why we have that policy:.

Keras developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information.

TensorFlow version (you are using): 2.6
Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

Describe the feature clearly here. Be sure to convey here why the requested feature is needed. Any brief description of the use-case would help.

Paper: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Original Code: https://github.com/microsoft/Swin-Transformer?utm_source=catalyzex.com

It's a variant of the transformer model and achieves state-of-the-art performance or comparable performance with the best CNN-based models. It also contains enough citations (~250 at this moment) for addition to the package.

On ImageNet-1K and 22K, below is the comparable results with EfficientNet (CNN) models.

-	Img Size	Top 1K acc	-	Img Size	Top 1K acc	Top 22K acc
E3	300	81.6	EfficientNetV2-S	-	83.9	84.9
E5	456	83.6	EfficientNetV2-M	-	85.1	86.2
E7	600	84.3	EfficientNetV2-L	-	85.7	86.8
-	-	-	EfficientNetV2-XL	-	-	87.3
Swin-T	224	81.3	Swin-B	224	-	85.2
Swin-S	224	83.0	Swin-B	384	-	86.4
Swin-B	224	83.5	Swin-L	384	-	87.3
Swin-B	384	84.5	-	-	-	-

Will this change the current api? How?
Yes. It will change as follows

tensorflow.keras.applications.SwinT
tensorflow.keras.applications.SwinS
tensorflow.keras.applications.SwinB
tensorflow.keras.applications.SwinL

Who will benefit from this feature?
Keras users.

Contributing

• Do you want to contribute a PR? (yes/no): yes.
• If yes, please read this page for instructions
• Briefly describe your candidate solution(if contributing):

Including All Components to Train State of the Art Imagenet-1k models

https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives

Two papers to reference:

• ResNet-RS
• ResNet Strikes Back

ResNet-RS block/layer

WideResNet architecture

Best resnet for Cifar10/Cifar100

https://arxiv.org/abs/1605.07146

GridMask Augmentation

Paper: GridMask Data Augmentation
Citation: ~70
Code: https://github.com/google/automl/blob/master/efficientdet/aug/gridmask.py

Demo:

---

@LukeWood Is it possible to add a Discussion tab, like.

Create a CONTRIBUTING.MD guide

From a PR review:

Do we want to have a paper citation treshold or any other evalustion metric?

We want an adoption metric of some kind, and a citation threshold (~50) is a convenient metric.

What about the component maintainership policiy? Can we scale over the community maintainership/ownership of the components?

We'll try. Someone who contributes a component will be called to fix issues with it if any arise.

I think also that we could not go to only accumulate components over time how we are going to evaluate and handle deprecations?

Deprecation decisions are always the result of cost/benefits analysis. The picture varies from component to component and over the lifetime of the repo. It's a case by case basis.

I suppose that we could partially use the README.MD for this and partially CONTRIBUTING.MD

Yes, this is mostly information that should go in the contributors' guide.

ConvNeXt Architecture

Sayak is working on this

COCORecall should support XLA compilation

Current error when running in mirrored mode:

ValueError: SyncOnReadVariable does not support assign_add in cross-replica context when aggregation is set to tf.VariableAggregation.SUM.

If we can compile to XLA we can also run on TPUs.

Make internal utilities private

We don’t want people depending on, for example, fill_utils. This is intended for internal use only

ResNet18

Note: Owen is already working on this.

SLIC Layer: Superpixel

Image Augment layer using SLIC
https://ieeexplore.ieee.org/document/6205760 cited by 7880
Implementation in skimage https://github.com/scikit-image/scikit-image/blob/v0.19.0/skimage/segmentation/slic_superpixels.py#L110-L385

Sharpen images by using a unsharp mask or something better I am unaware of.

Re-evaluate the need for `label_smoothing` in CutMix

I'm pretty sure we can remove this and rely on keras.losses.CategoricalCrossentropy's label_smoothing. I don't think it makes any numerical difference to do it earlier in the pipeline.

Mosaic Augmentation for Object Detection

Mosaic Augmentation

• Paper: YOLOv4: Optimal Speed and Accuracy of Object Detection (Figure 3).
• Yolo-V4 Citation: 3718

What is it?
The idea is to take 4 random samples and create a single sample of mosaic fashion with them. For example:

Starter

• PyTorch Implementation:
• TensorFlow Implementation: google/automl/efficientdet/aug/mosaic.py

COCOMeanAveragePrecision metric