Make IndexedConv compatible with tensorflow models

[indexedconv]

Make IndexedConv compatible with tensorflow models

[mikael10j]

Hi @TjarkMiener, to use indexed convolution (or pooling) you also need at least neighbours_extraction and pool_index_matrix.
It would be very nice if you add the tf version directly in the IndexedConv package. So you wouldn't have to rewrite the utils functions, only the engine part, and you could benefit from unit tests. Moreover, the package already has a DOI and a "visibility" with the publication in the VISAPP proceedings.
To do this, I'll reorganize the engine folders with pytorch and tensorflow subfolders, and I'll take care of getting the utils functions agnostic to the framework used.
What do you think ?

[TjarkMiener]

That would be awesome! Thank you very much for writing the utils functions agnostic to the framework :). @nietootein and me are taking care of the tensorflow engine part.

[mikael10j]

@TjarkMiener, I've prepared IndexedConv for tf in the branch tf_support.

[TjarkMiener]

That was quick! Thanks, @mikael10j! I'm on it.

[TjarkMiener]

Hi @mikael10j! I lost a bit momentum in this issue due to urgent stuff I had to do. I sadly had to learn that the tensorflow array slicing and indexing aren't gpu supported. So far, I implemented a MaskedConv2D layer and MaskedAvgPool2D pooling class with functional interfaces (please see here), which I believe should run on gpu. I didn't run it with CTLearn yet, since the installation in the tf_support branch is failing. I pushed a commit that added the current version of tensorflow used by CTLearn and a missing comma. However, this doesn't solved the installation issue. It also need minor changes to the ImageMapper of DL1DH, which I will be pushing soon and set you as a reviewer.
I also started to implement an IndexedConv2D layer class, but I will focus on the Multitask approach in the upcoming weeks. @nietootein is advising @hrueda25, who is writing his Master's Thesis about hexagonal convolution with IACT data. Maybe we can use this as a starting point to close this issue.

[TjarkMiener]

Hi @mikael10j and @vuillaut,

@hrueda25 and me trying to finalize this issue soon. We will develop the tensorflow layer classes in my fork and once we completed the roadmap below we will open a PR.
Roadmap:

• Fix installation bug in tf_support branch. Maybe Mikael can assist here.
• Minor changes to the DL1DH ImageMapper for the masked convolution classes (I will take care of this)
• Implement masked convolution classes (MaskedConv2D, MaskedAvgPool2D and MaskedMaxPool2D). This is done for MaskedConv2D and MaskedAvgPool2D.
• Implement indexed convolution classes (IndexedConv2D, IndexedAvgPool2D and IndexedMaxPool2D). Check out tf.batch_gather for tensorflow indexing. However array slicing and indexing is not gpu supported in tensorflow.
• Test and verify the implementation for at least one dataset (cifar,mnist or aid)
• Test tf_support implementation with CTLearn
• Benchmark hexagonal convolution for IACT data

If time allows:

• Implement masked/indexed convolution classes for Keras/update to tensorflow v2.0

[mikael10j]

@TjarkMiener , could you please provide the error trace you get when trying to install ?

[TjarkMiener]

@mikael10j It took me awhile to reproduced the issue I saw long time ago. So the installation is not failing, but when I try to import indexedconv it can't find the module.
(I added a missing comma after "tensorflow" in line 19 of the setup.py)

(myenv) Tjarks-MacBook-Pro:tests tmiener$ python
Python 3.7.3 | packaged by conda-forge | (default, Jul 1 2019, 14:38:56)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import indexedconv
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/anaconda3/envs/myenv/lib/python3.7/site-packages/indexedconv/__init__.py", line 2, in <module>
import indexedconv.engine
ModuleNotFoundError: No module named 'indexedconv.engine'

[mikael10j]

@TjarkMiener , I've fixed the installation and import issues (112bbe9).

[TjarkMiener]

Thank you very much @mikael10j! I checked the fix and it solved the problem.

[rockdeme]

@TjarkMiener is this project abandoned?

[TjarkMiener]

@rockdeme The project was carried out here with CTLearn v0.5.X and TensorFlow 1.15.X. Sadly, the code devs were never submitted to the main repo. We would need to update/translate the code to Keras in order to make it compatible with CTLearn v0.6.X and TensorFlow 2.8.X. I'm sorry, but I won't have time to do it myself anytime soon. Would you like to contribute?