nairouz/dynae

Deep Clustering with a Dynamic Autoencoder: From Reconstruction towards Centroids Construction

Abstract

In unsupervised learning, there is no apparent straightforward cost function that can capture the significant factors of variations and similarities. Since natural systems have smooth dynamics, an opportunity is lost if an unsupervised objective function remains static. The absence of concrete supervision suggests that smooth dynamics should be integrated during the training process. Compared to classical static cost functions, dynamic objective functions allow to better make use of the gradual and uncertain knowledge acquired through pseudo-supervision. In this paper, we propose Dynamic Autoencoder (DynAE), a novel model for deep clustering that overcomes a clustering-reconstruction trade-off, by gradually and smoothly eliminating the reconstruction objective function in favor of a construction one. Experimental evaluations on benchmark datasets show that our approach achieves state-of-the-art results compared to the most relevant deep clustering methods.

Usage

You can run the code of DynAE on google Colaboratory plateform without installing any package.

1. Create a folder entitled "Colab" in your google drive.
2. Download the two folders "DynAE" and "data" and drop them in the Colab folder.
3. Run the notebook "dynAE.ipynb" using Colab.

Citation

@article{mrabah2020deep,
title={Deep clustering with a dynamic autoencoder: From reconstruction towards centroids construction},
author={Mrabah, Nairouz and Khan, Naimul Mefraz and Ksantini, Riadh and Lachiri, Zied},
journal={Neural Networks},
volume={130},
pages={206--228},
year={2020},
publisher={Elsevier}
} 

dynae's Issues

Critic weights are not modified during pre-training

Hi,

it appears to me there is an issue during training in the published version of the code leading to the critic/discriminator weights not being trained. In function aci_ae_constructor in file DynAE.py the weights of the critic are set as non-trainable, because the critic should not be trained when the auto encoder is trained: critic.trainable = False. However, this change also propagates to the critic and discriminator model, such that DynAE.train_on_batch_disc() does not update any weights.

This can be verified in two ways, either checking whether the critic weights are updated by inserting into DynAE.py line 333 following:

        last_critic_weights = None
        
        #Training loop
        for ite in range(int(maxiter)):
            #Validation interval
            if ite % validate_interval == 0:
                # Check weights modified
                critic_weights = self.disc.get_weights()
                if last_critic_weights is not None:
                    if all([np.all(critic_weights[i] == last_critic_weights[i]) for i in range(len(critic_weights))]):
                        print("Critic weights were not modified during training")
                last_critic_weights = [np.copy(c) for c in critic_weights]

or by directly checking whether the discriminator model has any trainable weights, i.e. modifying:

    def train_on_batch_disc(self, x1, x2, y1, y2):
        y = np.zeros((x1.shape[0],))
        if len(self.disc.trainable_weights) == 0:
            print("The discriminator is being trained with no trainable weights")
        return self.disc.train_on_batch([x1, x2, y1, y2], y)

It is suprising to me how well the model performs despite the discriminator not being trained, but it seems to me that your published work does things differently. Is this a mistake?

Kind regards

Embeddings

Can you guide me as to how you have plotted the embeddings, as visualizing 2D embedded subspaces using TSNE is showing errors? Are you using the npy files generated in the embeddings?