marl/autopool

Adaptive pooling operators for multiple instance learning

autopool's Issues

Restructuring for multiple backends

Tagging @justinsalamon for some input on this, should be easy.

The reference implementation of autopool is tied to old-style Keras, and that's cool. PR #12 updates this to work with v2.2 and up. However, it is recommended at this point that keras-dependent code target the tf.keras backend instead of the standalone package.

I think we should still provide the old-style interface (for historical reasons), but we should also have a tf.keras-facing implementation.

At the same time, we may as well provide a pytorch implementation as well.

I think the sensible way to approach this is by having different submodules for each backend, eg autopool.keras, autopool.tf, autopool.torch from which you would import the AutoPool/SoftMaxPool classes.

The only hiccup here is that the import structure changes. Instead of the previous

from autopool import AutoPool1D

you would have

from autopool.keras import AutoPool1D

or

from autopool.torch import AutoPool1D

(not unlike from tqdm.notebook import tqdm).

What do you think?

Images for readme

This issue contains all images needed for the README file.

Equations

Why normalizing the scaled feature map?

I don't fully grasp the need of this normalization that you employ in your code:
softmax = tf.keras.backend.exp(scaled - max_val)

I'm tempted to remove this line from my implementation.
Any ideas on that? Does it help training?

Jordi.

Evaluation

Hi @justinsalamon
i have a doubt regarding the evaluation implementation . during evaluation(testing) do we remove the aggregator layer to get instance level tagging. in that case the Alpha of the adaptive pooling layer is used only for training, pls clarify.

thanks
yuvaram

paremeter question

hi, I want to figure out if the instance_pred in the code bag_pred = AutoPool(axis=1, kernel_constraint=keras.constraints.max_norm(alpha_max, axis=0))(instance_pred) is embedded feature maps of multi-instances? thanks! By the way it would more convenient if a demo script can be offer.

Explanation on the call function

Hi ,
i am trying to understand the implementation by comparing it with the paper. currently i am not able to understand a particular code

def call(self, x, mask=None):
        scaled = self.kernel * x
        max_val = K.max(scaled, axis=self.axis, keepdims=True)
        softmax = K.exp(scaled - max_val)
        weights = softmax / K.sum(softmax, axis=self.axis, keepdims=True)
        return K.sum(x * weights, axis=self.axis, keepdims=False)

in this call function, we are subtracting max value from scaled softmax = K.exp(scaled - max_val). but , i am not able to find this part on the paper . pls explain.

Update for modern keras/tf.keras, tensorflow 2.0

Issue title says it - this should be a fairly easy update.

Autopool2D

I guess the image folks could benefit from a 2D autopool implementation. Does anyone feel like taking a crack at generalizing our implementation?

Implementation for Pytorch

Hi Justin,
i like to try this for a SED work. i am looking to Implementation this on pytorch. i like to know if there exist any pytorch Implementation .
thanks
yuvaram

How to use Autopool for SED

Hi @justinsalamon ,
i am currently trying to implement the MIL approach for SED based on your adaptive pooling paper. https://arxiv.org/pdf/1804.10070.pdf. currently my implementation is not giving the intended results . i like to get your opinion on my implementation and correct me if i have missed any.

1.split the single track into 4 instance of 5sec each
2, converted them into melspec and passed to feature extractor made of conv layer followed by fully connected layer .
3, at the end of the previous step, the model will give sigmoid output for 4instance X n_classes.
4, this is passed on to my implememtation of Autopooling layer and will get a single vector containing n_classes weighted sigmoid values.
5, i am using BCEWithLogitsLoss in pytorch which is equivalent to BinaryCrossentropy with sigmoid layer in keras

pls share your thoughts on this.
thanks
yuvaram