tensorflow/addons

Useful extra functionality for TensorFlow 2.x maintained by SIG-addons

⚠️ ⚠️ ⚠️

TensorFlow Addons (TFA) has ended development and introduction of new features.

TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024. Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP)

For more information see: #2807

---

---

Continuous Build Status

Build	Status
Ubuntu/macOS
Ubuntu GPU custom ops

TensorFlow Addons is a repository of contributions that conform to well-established API patterns, but implement new functionality not available in core TensorFlow. TensorFlow natively supports a large number of operators, layers, metrics, losses, and optimizers. However, in a fast moving field like ML, there are many interesting new developments that cannot be integrated into core TensorFlow (because their broad applicability is not yet clear, or it is mostly used by a smaller subset of the community).

Addons Subpackages

• tfa.activations
• tfa.callbacks
• tfa.image
• tfa.layers
• tfa.losses
• tfa.metrics
• tfa.optimizers
• tfa.rnn
• tfa.seq2seq
• tfa.text

Maintainership

The maintainers of TensorFlow Addons can be found in the CODEOWNERS file of the repo. This file is parsed and pull requests will automatically tag the owners using a bot. If you would like to maintain something, please feel free to submit a PR. We encourage multiple owners for all submodules.

Installation

Stable Builds

TensorFlow Addons is available on PyPI for Linux & macOS (Windows support was dropped due to inconsistent TF2.15 whl packaging). To install the latest version, run the following:

pip install tensorflow-addons

To ensure you have a version of TensorFlow that is compatible with TensorFlow Addons, you can specify the tensorflow extra requirement during install:

pip install tensorflow-addons[tensorflow]

Similar extras exist for the tensorflow-gpu and tensorflow-cpu packages.

To use TensorFlow Addons:

import tensorflow as tf
import tensorflow_addons as tfa

Python Op Compatility

TensorFlow Addons is actively working towards forward compatibility with TensorFlow 2.x. However, there are still a few private API uses within the repository so at the moment we can only guarantee compatibility with the TensorFlow versions which it was tested against. Warnings will be emitted when importing tensorflow_addons if your TensorFlow version does not match what it was tested against.

Python Op Compatibility Matrix

TensorFlow Addons	TensorFlow	Python
tfa-nightly	2.12, 2.13, 2.14	3.9, 3.10, 3.11
tensorflow-addons-0.22.0	2.12, 2.13, 2.14	3.9, 3.10, 3.11
tensorflow-addons-0.21.0	2.11, 2.12, 2.13	3.8, 3.9, 3.10, 3.11
tensorflow-addons-0.20.0	2.10, 2.11, 2.12	3.8, 3.9, 3.10, 3.11
tensorflow-addons-0.19.0	2.9, 2.10, 2.11	3.7, 3.8, 3.9, 3.10
tensorflow-addons-0.18.0	2.8, 2.9, 2.10	3.7, 3.8, 3.9, 3.10
tensorflow-addons-0.17.1	2.7, 2.8, 2.9	3.7, 3.8, 3.9, 3.10
tensorflow-addons-0.16.1	2.6, 2.7, 2.8	3.7, 3.8, 3.9, 3.10
tensorflow-addons-0.15.0	2.5, 2.6, 2.7	3.7, 3.8, 3.9
tensorflow-addons-0.14.0	2.4, 2.5, 2.6	3.6, 3.7, 3.8, 3.9
tensorflow-addons-0.13.0	2.3, 2.4, 2.5	3.6, 3.7, 3.8, 3.9
tensorflow-addons-0.12.1	2.3, 2.4	3.6, 3.7, 3.8
tensorflow-addons-0.11.2	2.2, 2.3	3.5, 3.6, 3.7, 3.8
tensorflow-addons-0.10.0	2.2	3.5, 3.6, 3.7, 3.8
tensorflow-addons-0.9.1	2.1, 2.2	3.5, 3.6, 3.7
tensorflow-addons-0.8.3	2.1	3.5, 3.6, 3.7
tensorflow-addons-0.7.1	2.1	2.7, 3.5, 3.6, 3.7
tensorflow-addons-0.6.0	2.0	2.7, 3.5, 3.6, 3.7

C++ Custom Op Compatibility

TensorFlow C++ APIs are not stable and thus we can only guarantee compatibility with the version TensorFlow Addons was built against. It is possible custom ops will work with multiple versions of TensorFlow, but there is also a chance for segmentation faults or other problematic crashes. Warnings will be emitted when loading a custom op if your TensorFlow version does not match what it was built against.

Additionally, custom ops registration does not have a stable ABI interface so it is required that users have a compatible installation of TensorFlow even if the versions match what we had built against. A simplification of this is that TensorFlow Addons custom ops will work with pip-installed TensorFlow but will have issues when TensorFlow is compiled differently. A typical example of this would be conda-installed TensorFlow. RFC #133 aims to fix this.

C++ Custom Op Compatibility Matrix

TensorFlow Addons	TensorFlow	Compiler	cuDNN	CUDA
tfa-nightly	2.14	GCC 9.3.1	8.6	11.8
tensorflow-addons-0.22.0	2.14	GCC 9.3.1	8.6	11.8
tensorflow-addons-0.21.0	2.13	GCC 9.3.1	8.6	11.8
tensorflow-addons-0.20.0	2.12	GCC 9.3.1	8.6	11.8
tensorflow-addons-0.19.0	2.11	GCC 9.3.1	8.1	11.2
tensorflow-addons-0.18.0	2.10	GCC 9.3.1	8.1	11.2
tensorflow-addons-0.17.1	2.9	GCC 9.3.1	8.1	11.2
tensorflow-addons-0.16.1	2.8	GCC 7.3.1	8.1	11.2
tensorflow-addons-0.15.0	2.7	GCC 7.3.1	8.1	11.2
tensorflow-addons-0.14.0	2.6	GCC 7.3.1	8.1	11.2
tensorflow-addons-0.13.0	2.5	GCC 7.3.1	8.1	11.2
tensorflow-addons-0.12.1	2.4	GCC 7.3.1	8.0	11.0
tensorflow-addons-0.11.2	2.3	GCC 7.3.1	7.6	10.1
tensorflow-addons-0.10.0	2.2	GCC 7.3.1	7.6	10.1
tensorflow-addons-0.9.1	2.1	GCC 7.3.1	7.6	10.1
tensorflow-addons-0.8.3	2.1	GCC 7.3.1	7.6	10.1
tensorflow-addons-0.7.1	2.1	GCC 7.3.1	7.6	10.1
tensorflow-addons-0.6.0	2.0	GCC 7.3.1	7.4	10.0

Nightly Builds

There are also nightly builds of TensorFlow Addons under the pip package tfa-nightly, which is built against the latest stable version of TensorFlow. Nightly builds include newer features, but may be less stable than the versioned releases. Contrary to what the name implies, nightly builds are not released every night, but at every commit of the master branch. 0.9.0.dev20200306094440 means that the commit time was 2020/03/06 at 09:44:40 Coordinated Universal Time.

pip install tfa-nightly

Installing from Source

You can also install from source. This requires the Bazel build system (version >= 1.0.0).

CPU Custom Ops

git clone https://github.com/tensorflow/addons.git
cd addons

# This script links project with TensorFlow dependency
python3 ./configure.py

bazel build build_pip_pkg
bazel-bin/build_pip_pkg artifacts

pip install artifacts/tensorflow_addons-*.whl

GPU and CPU Custom Ops

git clone https://github.com/tensorflow/addons.git
cd addons

export TF_NEED_CUDA="1"

# Set these if the below defaults are different on your system
export TF_CUDA_VERSION="12"
export TF_CUDNN_VERSION="8"
export CUDA_TOOLKIT_PATH="/usr/local/cuda"
export CUDNN_INSTALL_PATH="/usr/lib/x86_64-linux-gnu"

# This script links project with TensorFlow dependency
python3 ./configure.py

bazel build build_pip_pkg
bazel-bin/build_pip_pkg artifacts

pip install artifacts/tensorflow_addons-*.whl

Tutorials

See docs/tutorials/ for end-to-end examples of various addons.

Core Concepts

Standardized API within Subpackages

User experience and project maintainability are core concepts in TensorFlow Addons. In order to achieve these we require that our additions conform to established API patterns seen in core TensorFlow.

GPU and CPU Custom Ops

TensorFlow Addons supports precompiled custom ops for CPU and GPU. However, GPU custom ops currently only work on Linux distributions. For this reason Windows and macOS will fallback to pure TensorFlow Python implementations whenever possible.

The order of priority on macOS/Windows is:

1. Pure TensorFlow + Python implementation (works on CPU and GPU)
2. C++ implementation for CPU

The order of priority on Linux is:

1. CUDA implementation
2. C++ implementation
3. Pure TensorFlow + Python implementation (works on CPU and GPU)

If you want to change the default priority, "C++ and CUDA" VS "pure TensorFlow Python", you can set the environment variable TF_ADDONS_PY_OPS=1 from the command line or run tfa.options.disable_custom_kernel() in your code.

For example, if you are on Linux and you have compatibility problems with the compiled ops, you can give priority to the Python implementations:

From the command line:

export TF_ADDONS_PY_OPS=1

or in your code:

import tensorflow_addons as tfa
tfa.options.disable_custom_kernel()

This variable defaults to True on Windows and macOS, and False on Linux.

Proxy Maintainership

TensorFlow Addons has been designed to compartmentalize submodules so that they can be maintained by community users who have expertise, and a vested interest in that component. We heavily encourage users to submit sign up to maintain a submodule by submitting your username to the CODEOWNERS file.

Full write access will only be granted after substantial contribution has been made in order to limit the number of users with write permission. Contributions can come in the form of issue closings, bug fixes, documentation, new code, or optimizing existing code. Submodule maintainership can be granted with a lower barrier for entry as this will not include write permissions to the repo.

For more information see the RFC on this topic.

Periodic Evaluation of Subpackages

Given the nature of this repository, submodules may become less and less useful to the community as time goes on. In order to keep the repository sustainable, we'll be performing bi-annual reviews of our code to ensure everything still belongs within the repo. Contributing factors to this review will be:

1. Number of active maintainers
2. Amount of OSS use
3. Amount of issues or bugs attributed to the code
4. If a better solution is now available

Functionality within TensorFlow Addons can be categorized into three groups:

• Suggested: well-maintained API; use is encouraged.
• Discouraged: a better alternative is available; the API is kept for historic reasons; or the API requires maintenance and is the waiting period to be deprecated.
• Deprecated: use at your own risk; subject to be deleted.

The status change between these three groups is: Suggested <-> Discouraged -> Deprecated.

The period between an API being marked as deprecated and being deleted will be 90 days. The rationale being:

1. In the event that TensorFlow Addons releases monthly, there will be 2-3 releases before an API is deleted. The release notes could give user enough warning.

2. 90 days gives maintainers ample time to fix their code.

Contributing

TensorFlow Addons is a community-led open source project (only a few maintainers work for Google!). As such, the project depends on public contributions, bug fixes, and documentation. This project adheres to TensorFlow's code of conduct. By participating, you are expected to uphold this code.

Do you want to contribute but are not sure of what? Here are a few suggestions:

1. Add a new tutorial. Located in docs/tutorials/, these are a great way to familiarize yourself and others with TensorFlow Addons. See the guidelines for more information on how to add examples.
2. Improve the docstrings. The docstrings are fetched and then displayed in the documentation. Do a change and hundreds of developers will see it and benefit from it. Maintainers are often focused on making APIs, fixing bugs and other code related changes. The documentation will never be loved enough!
3. Solve an existing issue. These range from low-level software bugs to higher-level design problems. Check out the label help wanted. If you're a new contributor, the label good first issue can be a good place to start.
4. Review a pull request. So you're not a software engineer but you know a lot about a certain field a research? That's awesome and we need your help! Many people are submitting pull requests to add layers/optimizers/functions taken from recent papers. Since TensorFlow Addons maintainers are not specialized in everything, you can imagine how hard it is to review. It takes very long to read the paper, understand it and check the math in the pull request. If you're specialized, look at the list of pull requests. If there is something from a paper you know, please comment on the pull request to check the math is ok. If you see that everything is good, say it! It will help the maintainers to sleep better at night knowing that he/she wasn't the only person to approve the pull request.
5. You have an opinion and want to share it? The docs are not very helpful for a function or a class? You tried to open a pull request but you didn't manage to install or test anything and you think it's too complicated? You made a pull request but you didn't find the process good enough and it made no sense to you? Please say it! We want feedback. Maintainers are too much the head into the code to understand what it's like for someone new to open source to come to this project. If you don't understand something, be aware there are no people who are bad at understanding, there are just bad tutorials and bad guides.

Please see contribution guidelines to get started (and remember, if you don't understand something, open an issue, or even make a pull request to improve the guide!).

Community

• Public Mailing List
• SIG Monthly Meeting Notes
  • Join our mailing list and receive calendar invites to the meeting

License

Apache License 2.0

addons's Issues

Implement Conditional Random Field

Per the RFC, we need to move CRF from contrib to addons:

This will involve moving the code and modifying test cases to run in TF2.

Add new API type: Keras Callback

Per #58 and our monthly discussion in March, Keras Callbacks seem like they are within scope for TF Addons. This new addition will require:

• Adding a new directory in tensorflow_addons which will hold keras callbacks
• Some initial contribution guidelines for adding new callbacks
  • e.g. sub-class the abstract base class
• Addition to the projects central BUILD file
• Updating the README/CONTRIBUTINg docs

This will ideally include a new callback as an example, but not necessarily.

Release first version of tensorflow-addons

As we discussed in the February monthly meeting... we are targeting our first release by early next week. I wanted to get this issue up so we can discuss versioning and outline what else needs to get done.

Versioning

I think there are two workable methods here:

1. Release on our own cadence and mention in the release which TF version it was tested against.
   • This is how it's done for tensorflow_probability
   • We could monitor our nightly tests and catch things like #37 which require us to rebuild
2. Release in step with core TensorFlow
   • This is how it's done for tensorboard.
   • This is more intuitive for the end user IMO
   • Does require more of a commitment from the SIG

Remaining blockers

• Some examples (use tensorflow/examples as template)
• Decide on version naming

Implement Maxout Layer

Per the RFC, we need to move maxout from contrib to addons:

This will involve restructuring in an OO format by inheriting from base Keras Layer, modifying the code to match those APIs, and modifying test cases to run in TF2.

https://arxiv.org/abs/1302.4389

Feature request: add parametric ELU (PELU) activation function

Proposal

The exponential linear unit (ELU) is already in TensorFlow as tf.nn.elu which is great. The new parametric version (called PELU) shows very promising experimental results so I wonder if it could be added in to TensorFlow too in order to encourage more widespread experimentation with it by the deep learning community. One problem with it though is that it's stateful (e.g. tf.Variable), meaning it's not clear to me where in TensorFlow it fits in.

Implementation

Here's an implementation of the PELU that I've been using lately (I'm assuming batch_size is the first dimension in x):

def pelu(x):
  """Parametric Exponential Linear Unit (https://arxiv.org/abs/1605.09332v1)."""
  with tf.variable_scope(x.op.name + '_activation', initializer=tf.constant_initializer(1.0)):
    shape = x.get_shape().as_list()[1:]
    alpha = tf.get_variable('alpha', shape)
    beta = tf.get_variable('beta', shape)
    positive = tf.nn.relu(x) * alpha / (beta + 1e-9)
    negative = alpha * (tf.exp((-tf.nn.relu(-x)) / (beta + 1e-9)) - 1)
    return negative + positive

Reference

https://arxiv.org/abs/1605.09332v1

Implement InstanceNorm

Per the RFC, we need to move instance_norm from contrib to addons:

This will involve restructuring in an OO format by inheriting from base Keras Layer, modifying the code to match those APIs, and modifying test cases to run in TF2.

https://arxiv.org/abs/1607.08022

Request for function that randomly augments sets of images

Several operations for image data augmentation have been implemented in tensorflow, however most of these work in single examples.

As such, to achieve the same rotation between an input image and a set of different-sized masks (as in a feature pyramid network for region proposal, for example) you have to go through a process of conditionals (through tf.cond).

I have tried to do this to no avail (i.e. https://stackoverflow.com/questions/51951039/different-results-with-image-flipping-in-tensorflow) and was wondering whether any solution had been implemented for this and, if not, to request a function for data augmentation in sets of images

Implement DenseToSparse

Per the RFC, we need to move dense_to_sparse from contrib to addons:

This will involve restructuring in an OO format by inheriting from base Keras Layer, modifying the code to match those APIs, and modifying test cases to run in TF2.

Discussion: Does it make more sense to consider this a low level nn block instead of a Layer? Or just an array op?

[Discussion] When to utilize tf.function

Are there any best practices for when to decorate functions with tf.function? The TF2 documentation has a nice example where a keras Model is decorating the call method.

@karmel @martinwicke Should we use this extensively in our repo anytime there is a method without python side-effects? Is it safer to only utilize it when there are clear performance gains by exploiting parallelism or other optimizations? The latter seems like a vague criteria without specifying what patterns to look for.

Standardize formatting

Would like to standardize repository style using a code formatter. @facaiy suggested https://github.com/google/yapf which looks like it would work very nicely.

In the future this should be tied in with our CI testing

Set up tests scripts for integration with CI

Need to add scripts so that we have control over CI testing jobs

As an example... something similar to this:
https://github.com/tensorflow/tensorflow/blob/9590c4c32dd4346ea5c35673336f5912c6072bf2/tensorflow/tools/ci_build/linux/gpu/run_cc_core.sh

Implement PoincareNormalize

Per the RFC, we need to move poincare_normalize from contrib to addons:

This will involve restructuring in an OO format by inheriting from base Keras Layer, modifying the code to match those APIs, and modifying test cases to run in TF2.

https://arxiv.org/pdf/1705.08039.pdf

Implement External Optimizers

Per the RFC, we need to move external optimizers from contrib to addons:

This will involve inheriting from base Keras optimizer, modifying the code to match those APIs, and modifying test cases to run in TF2.

• ExternalOptimizerInferace
• ScipyOptimizerInterface

Implement MovingAverageOptimizer

Per the RFC, we need to move MovingAverageOptimizer from contrib to addons:

This will involve inheriting from base Keras optimizer, modifying the code to match those APIs, and modifying test cases to run in TF2.

Implement Image functions

Per the RFC, we will be moving several of the image modules from contrib to addons:

• dense_image_warp
• random_hsv_in_yiq
• adjust_hsv_in_yiq
• rotate
• translate
• angle_to_projective_transforms
• translations_to_projective_transforms
• transform
• compose_transforms
• flat_transforms_to_matricies
• connected_components
• sparse_image_warp

This will involve moving the code base and modifying test cases to run in TF2.

Implement Metric Losses

Per the RFC, we need to move metric_losses from contrib to addons:

• pairwise_distance
• contrastive_loss
• triplet_semihard_loss
• npairs_loss
• npairs_loss_multilabel
• lifted_struct_loss

This will involve inheriting from base Keras Loss, modifying the code to match those APIs, and modifying test cases to run in TF2.

ImageProjectiveTransform should be named ImageProjectiveTransformV2

Describe the bug

• It looks like ImageProjectiveTransform is based on ImageProjectiveTransformV2 present in tf.contrib in TF-1.x.

Describe the expected behavior

• Op names should not colide between 1.x and 2.x environments or saved models won't be shareable across them.

Implement Weight Decay Optimizers

Per the RFC, we need to move Weight Decay Optimizers from contrib to addons:

• DecoupledWeightDecayExtension
• AdamWOptimizer
• MomentumWOptimizer

This will involve inheriting from base Keras optimizer, modifying the code to match those APIs, and modifying test cases to run in TF2.

Rename transform_test.py to transform_ops_test.py for consistency

In the folder https://github.com/tensorflow/addons/tree/master/tensorflow_addons/image, rename transform_test.py to transform_ops_test.py to be consistent with naming of test files.

Add contribution guideline for moving from tf.contrib

@dynamicwebpaige wisely suggested in #38 that we put together a guideline on moving from tf.contrib to addons. A majority of our code is currently ported from tf.contrib and there are many open issues for bringing over the rest of our addons RFC.

Some things to include in the guideline:

• TF2.0 symbols mapping
• Guidelines on when to use tf.function
• Common code changes (remove variable_scope, etc.)
• Suggestions on adding aditional test cases

I imagine we would hold this document in the repo for a substantial amount of time, but ultimately will want to remove it as we grow away from our original codebase of ported tf.contrib code

Remove numpy implementation in test case?

I find, in most of test cases of Addons / tf.contrib, we are used to write a duplicated numpy function to verify the correction of our implementations. It's straightforward, but the drawbacks are:

1. both tf and numpy implications could be incorrect;
2. numpy might have a bug, or will introduce a bug in the future;
3. difficult to maintain test cases, and might also too complex to understand it.

Hence I propose to remove all numpy implementations, and replace them by explicit numerical value (and add necessary comments to describe how to figure it out, say intermediate result of calculation) in test cases.

# No
def np_add(a, b):
    return a + b

self.assertEqual(np_add(1, 2), tf_add(1, 2)) 

# Yes
# Add some comments if necessary:
# c = a + b = 1 + 2 = 3
self.assertEqual(3, tf_add(1, 2))

cc @seanpmorgan @karmel

Implement LazyAdamOptimizer

Per the RFC, we need to move LazyAdamOptimizer from contrib to addons:

This will involve inheriting from base Keras optimizer, modifying the code to match those APIs, and modifying test cases to run in TF2.

Feature request : Add more Scipy Optimizer

Hi,

Could it be possible to wrap the scipy.optimize.basinhopping and scipy.optimize.anneal the same way the tf.contrib.opt.ScipyOptimizerInterface allow us to use method as LBFGS with Tensorflow to minimize a function ?

Thanks

Dendrite morphological neural networks

Can you guys also provide an implementation for Dendrite morphological neural networks in tensorflow.

http://www.sciencedirect.com/science/article/pii/S0925231213010916

[feature] tf.contrib.image.invert

/CC @ringw

Along the lines of the other utility functions in tf.contrib.image, would be great to have a tf.contrib.image.invert that takes a transform and returns the inverse transform.

CI: output log of each test target

There seems only one target tensorflow_addons/ubuntu/py2/presubmit in the detail page, check out:
https://source.cloud.google.com/results/invocations/bcda13af-4a13-41c6-a7db-76ff8e3ec504/targets/tensorflow_addons%2Fubuntu%2Fpy2%2Fpresubmit/log

@av8ramit
Could we give all the details for each test target, like tensorflow/tensorflow? That would be greatly helpful for review.

Add sanity check for addons

Since #63 has introduced sanity check for addons, can we add it for CI?
cc @av8ramit @karmel @seanpmorgan

command:

make sanity-check
# or
bash tools/ci_build/install/install_ci_dependency.sh --quiet
bash tools/ci_build/ci_sanity.sh --incremental

Entity could not be transformed

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 2.0.0-dev20190317 (gpu nightly)
• TensorFlow Addons installed from (source, PyPi): source
• TensorFlow Addons version: master
• Python version and type (eg. Anaconda Python, Stock Python as in Mac, or homebrew installed Python etc): anaconda (conda-forge) python 3.7.1
• Bazel version (if compiling from source): 0.20.0
• GCC/Compiler version (if compiling from source): 7.3.0
• Is GPU used? (yes/no): yes
• GPU model (if used): RTX 2070

I'm seeing the following warning appear when trying to use the transform op. The simplest way to reproduce is to simply run the tests via python:

python ~/src/addons/tensorflow_addons/image/transform_test.py
[...]
W0317 18:03:00.753756 139906711668544 tf_logging.py:161] Entity <function image_projective_transform_v2 at 0x7f3e73a05a60> could not be transformed and will be staged without change. Error details can be found in the logs when running with the env variable AUTOGRAPH_VERBOSITY >= 1. Please report this to the AutoGraph team. Cause: Unexpected error transforming <function image_projective_transform_v2 at 0x7f3e73a05a60>. If you believe this is due to a bug, please set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output when filing the bug report. Caused by: Unable to locate the source code of <function image_projective_transform_v2 at 0x7f3e73a05a60>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code

export AUTOGRAPH_VERBOSITY=10
python ~/src/addons/tensorflow_addons/image/transform_test.py
[...]
I0317 18:11:38.742995 140338201401152 ag_logging.py:132] Error transforming <function image_projective_transform_v2 at 0x7fa2ea6b2a60>
Traceback (most recent call last):
  File "~/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/autograph/pyct/parser.py", line 51, in parse_entity
    source = tf_inspect.getsource_no_unwrap(entity)
  File "~/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/tf_inspect.py", line 408, in getsource_no_unwrap
    lines, lnum = _inspect.findsource(obj)
  File "~/miniconda3/envs/tf2/lib/python3.7/inspect.py", line 786, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

Is this expected? I expect this is likely an artifact of a broken source build on my end, but I imagine others may experience similar issues until the CI/CD system is building nightly versions for all platforms and python versions.

Do you accept tf.keras.callbacks?

Seeing as keras is going to become a first citizen inside of tensorflow, are you also accepting submission for callbacks? Are there guidelines for this (writing tests/docs)?.

Feature Request : PathNorm and PathSGD

PathSGD was introduced in this paper. Is there existing support for this? If not, this is a feature request for:

• PathNorm computation (Equation 5 in the paper)
• PathSGD using the PathNorm

For the first part, the interface can be to provide a function path_norm(a, b, p=2) where a and b are tensors, p is a scalar. The function returns the p-PathNorm for the "path" between the tensors a and b (assuming that b depends on a and some weights. If not, there can be an exception or simply return 0).

Replace ops.name_scope by tf.name_scope

Examples:

addons/tensorflow_addons/image/distort_image_ops.py

Line 134 in 8dad6d2

with tf.name_scope(name):

addons/tensorflow_addons/image/transform_ops.py

Line 74 in 8dad6d2

with ops.name_scope(name, "transform"):

We should use tf public API tf.name_scope.

Remove use of internal tensorflow API

The current implementations use the internal tensorflow.python interface. I know this was how it was done in tf.contrib. However, since addons are not an integrated part of tensorflow but rather decoupled from the main tensorflow module, using that internal API seams highly wrong.

Unpooling layer in tensorflow

It would be nice to have in TensorFlow also the unpooling layer as it is described in the paper on deconvolution networks: http://cvlab.postech.ac.kr/research/deconvnet/

I was googling a bit and I found that the added unpooling layer would be handful also for others:
http://stackoverflow.com/questions/36548736/tensorflow-unpooling

Add project description to package and pypi

As is done here: https://github.com/tensorflow/tensorflow/blob/cfbbbe3b2e034438437cf5c9fa46756ae4d0b2d5/tensorflow/tools/pip_package/setup.py#L248

We should include the description found in the README

Implement Sparsemax Activation and Loss

Per the RFC, we need to move sparsemax from contrib to addons:

• Moving sparsemax to addons/activations and modifying test cases to run in TF2.
• Restructuring sparsemax_loss in an OO format by inheriting from base Keras Loss, modifying the code to match those APIs, and modifying test cases to run in TF2.

https://arxiv.org/abs/1602.02068

[Bug] skip_gram_sample assert rank is 1

While converting the tests for skip_gram_ops, I noticed a test that was failing in TF2 eager execution, but I'm not sure why.

This test:

with self.assertRaises(ValueError):
    invalid_tensor = constant_op.constant([[b"the"], [b"quick"], [b"brown"]])
    text.skip_gram_sample(invalid_tensor)

Should raise an error because of this check in the op definition. When decorating with @test_util.run_deprecated_v1 the test correctly passes (error is raised), but otherwise it fails to raise an Error.

Is this related to inspecting the rank of EagerTensors? Or is something else at play?

Feature Request : Stochastic Depth

Stochastic Depth (aka layer dropout) has been shown to speed up and improve training in ResNets, as well as overall accuracy on testing sets. Essentially, every training step a random subset of residual layers are entirely removed from the network, and training proceeds on the remaining layers. Direct connections are made between the missing layers.

It is described in this paper: https://arxiv.org/pdf/1603.09382.pdf. (Deep Networks with Stochastic Depth by Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q. Weinberger)

I can't think of a way to implement this with the python API without reconstructing the model every training iteration, and I'm not familiar the with the C++ API / Cudnn to try to write the op myself.

Of course I'm willing to try any python-only suggestions.

Thanks in advance,

Alex

Automate Build Process

Currently we have no automated process for building Addons across python version and operating systems. Going forward we'll want this process to be automated.. but it may be challenging for us to start builds without access to the Google internal tooling.

We could conceivably use Travis... but if we can keep consistent CI that would be ideal.

Add a bot to email mailing list

We currently have an issue tag Discussion Needed. As @facaiy suggested, it would be great if those tags were sent to the mailing list [email protected] .

If we could create a bot which searched for these tags and emailed the listserv when they are created, I think that'd be a very helpful addition to our repo. A starting point could be here: https://github.com/probot/probot

Build out contributing guidelines

In an effort to encourage involvement, and standardize contributions we should make a concerted effort to build thorough docs on how to contribute for each sub-package (layers, crf, etc.)

For example:
Building from #19 dicussion. We should create standards for adding custom keras Layers. Some things to include:

• Testing with the Keras layer test
• Registering the Layer so a Model can be serialized / deserialized

Feature request: Keras CRF layer

It will be great if in the contrib.keras in the nearest future linear CRF layer (inside the Keras wrapper) will be added.

I think a lot of people will find this very useful.

Implement GroupNorm, LayerNorm & InstanceNorm

Per the RFC, we need to move layer_norm from contrib to addons:

This will involve restructuring in an OO format by inheriting from base Keras Layer, modifying the code to match those APIs, and modifying test cases to run in TF2.

https://arxiv.org/abs/1607.06450

EDIT: It will make more sense to implement GroupNorm, and then created interfaces for LayerNorm and InstanceNorm

Group Norm: https://arxiv.org/pdf/1803.08494.pdf

Generate API docs

As our repository matures it's important to have api docs to improve user experience. As discussed in #38 we will also be able to remove the table of contents off the main README.

Should we host on https://readthedocs.org/ or is there something else recommended @ewilderj @dynamicwebpaige @karmel ?

Packaged Addons don't run on tf-nightly-2.0

While packaging 0.2.0 release of addons, I realized that if the pip package was built against tensorflow==2.0.0-alpha0 the installed package would not correctly work on the tf2-nightly install.

The error is:

NotFoundError: /usr/local/lib/python3.6/dist-packages/tensorflow_addons/custom_ops/image/python/_distort_image_ops.so: undefined symbol: _ZN10tensorflow12OpDefBuilder10SetShapeFnEPFNS_6StatusEPNS_15shape_inference16InferenceContextEE

Digging in further I thought that something may have gone awry in the build for this version, but when I checked a previous version (0.1.1) I noticed it also failed to load on the current nightly. Typically I've seen this type of error when using gcc>=5 but the solution provided is to set D_GLIBCXX_USE_CXX11_ABI=0 which we do

Some things to note in the investigation:

• 0.1.1 and 0.1.0 were built against the tf2-nightly at the time of packaging and did import fine with that package at the time (It also works on tf2-alpha)
• If I build 0.2.0 against the current nightly then it does successful import against tf2-nightly (This is why our nightly builds succeed). However, it'll then fail to import against TF2-alpha

Has there been a recent change in how tf2-nightly is packaged compared to tf2-alpha? Going forward we can mention that addons should be used with tf2-alpha, and just use nightly for testing... but there seems to be an underlying issue that needs to be fixed.

Colab Notebooks:

Addons-0.1.1 + Alpha
Addons-0.1.1 + Nightly

Addons-0.2.0 + Alpha
Addons-0.2.0 + Nightly

cc @gunan @yifeif

Resize image for nd

Is there a functionality of resize image for n-d where n > 2? It seems currently tensorflow only support resize images for 2d images (4D tensor). Could you add the functionality?

Build Addons for MacOS

Currently our template for custom-ops is designed for linux builds. We'll need to adapt our build setup to build for all operating systems.

https://github.com/tensorflow/custom-op

@yifeif @karmel When time allows after the summit, could we discuss the steps needed.

Package Addons for python3.7

Using the current docker image, we're unable to install / build TensorFlow in python3.7 with the following error:

ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /root/miniconda2/envs/py37/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

It seems like we need to compile using Ubuntu 16.04 libs, which will require a bump in the custom-op docker image. @yifeif would it be possible to bump the base image in the container? If so we can rebuild our release as 0.1.1 and include py37.

Alternatively, we could create a new docker image for builds.

Create Issue Templates

It would be beneficial to the repo's maintianability if we had some templates for common issues. Possible templates

• Issue to request new addition to addons (option for migration from tf.contrib)
• Bug report

Build out examples with working colab notebooks

Would be nice to mirror the TF2 doc style of colab notebooks being first-class tutorials.

I think it's also a good idea to require colab notebooks for new contributions in order to showcase their benefits and best practices. Certainly won't be possible/necessary for everything.

Rename LazyAdamOptimizer to LazyAdam

All core optimizers in TensorFlow 2.0 dropped the "Optimizer" suffix: "Adam", "Adadelta", etc. (see the module overview). For consistency, optimizers defined in tensorflow/addons should probably follow the same naming convention.

So I propose to rename LazyAdamOptimizer to LazyAdam.