Low MIG and AAM metrics

[yanndubs]

Low MIG and AAM metrics

[Justin-Tan]

Yeah, that's what I mean about the shuffling, thanks for confirming, so that is not the cause of it. I am still confused because AFAIK this repository and the author's code appear to be doing the exact same thing when calculating MIG (on dSprites at least) but this repository is giving much lower MIG when loading the same models. I'll look more into it over this weekend.

I also looked into the discrete estimation of MIG used in [1], appendix C. (Essentially discretize samples from z in ~20 bins and use sklearn to estimate the discrete MI b/w latents z and generative factors v.) Unfortunately it does not agree with the MIG computed using this sampling-based estimation (consistently lower, reasonably insensitive to number of bins) irrespective of whether we use the mean of the latent representation or we sample from q(z|x), so the jury is still out on how to best estimate MIG I suppose.

Edit: It seems like the MIG scores reported in [1] are consistently lower anyway around 0.2 for the best models, so perhaps this is expected.
'
[1]: https://arxiv.org/abs/1811.12359

[YannDubs]

The small MIG is definitely (and unfortunately) something we always had in our experiments. Importantly, I got the same results when using the author's implementation. This is one of the reason we introduced AAM, which measures only the disentanglement rather than disentanglement + amount of information of v about z. I am surprised you get small AAM though.

Here are the results we were getting :

We see that when increasing β by a small amount (from 1 to 4), highly increases axis alignment (from20% to 65%) due to the regularisation of the total correlation, while increasing β by a large amount (from 4 to 50) decreases axis alignment due to the penalisation of the dimension wise KL. I.e. it is not monotonic.

[Justin-Tan]

Thanks for the confirmation, this is slightly puzzling as the authors of beta-TCVAE report MIG scores of O(0.1) in their paper. Other third-party implementations also report MIG of O(0.1), e.g. "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations". I might try emailing the authors directly to see how they obtained their results. 

…

On Sun, Nov 24, 2019 at 8:21 PM Yann Dubois ***@***.***> wrote: The small MIG is definitely (and unfortunately) something we always had in our experiments. Importantly, I got the same results when using (the author's implementation)[https://github.com/rtqichen/beta-tcvae]. This is one of the reason we introduced AAM, which measures only the disentanglement rather than disentanglement + amount of information of *v* about *z*. I am surprised you get small AAM though. Here are the results we were getting : [image: Screen Shot 2019-11-24 at 4 12 12 AM] <https://user-images.githubusercontent.com/24327668/69492500-a0cb6900-0e71-11ea-9cea-fd7ad8912912.png> We see that when increasing β by a small amount (from 1 to 4), highly increases axis alignment (from20% to 65%) due to the regularisation of the total correlation, while increasing β by a large amount (from 4 to 50) decreases axis alignment due to the penalisation of the dimension wise KL. I.e. it is not monotonic. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#52?email_source=notifications&email_token=AGRNY6HZANTIBXI4WQFAQJLQVJBQXA5CNFSM4JLL55R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFAHBRI#issuecomment-557871301>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGRNY6FEEHKLWLYGZSITEGDQVJBQXANCNFSM4JLL55RQ> .

[YannDubs]

Yes it is, if you get an answer / insights please post it here. I would be interested + other people might be.

And just to be clear, I have not tried rerunning the authors code. I only tried using their MIG code to compute the MIG for our results :/ . I.e. it does not seem that the issue comes from the computation of MIG, but to be honest I have not spent too much time on MIG as this was a late addition before a deadline.

[Justin-Tan]

After some digging I am getting better results using the author's MIG calculation code - around 0.3-0.8 for most of my trained models. Perhaps the problem lies in shuffling the dataloader? I notice when I shuffle the dataloader I get a very low MIG (on dSprites).

# Load dataloader
all_loader = (..., shuffle=False)

vae = model
N = len(all_loader.dataset)     # number of data samples - don't shuffle
K = vae.latent_dim              # number of latent variables
nparams = 2
vae.eval()

qzCx_params = torch.Tensor(N, K, nparams)

n = 0

with torch.no_grad():
    for x, gen_factors in all_loader:
        batch_size = x.size(0)
        x, gen_factors = x.to(device, dtype=torch.float), gen_factors.to(device)
        qzCx_stats = torch.stack(vae.encoder(x)['continuous'], dim=2)
        qzCx_params[n:n + batch_size] = qzCx_stats.data
        n += batch_size

# Reshape to get known generative factors
qzCx_params = qzCx_params.view(3, 6, 40, 32, 32, K, nparams).to(device)

# Sample from diagonal Gaussian posterior q(z|x) using given parameters (mu, logvar)
qzCx_samples = qzCx_sample(params=qzCx_params)

I think the reshape on the second last line requires the dataset to be in the native order so that the generative factors are in the correct order - it's not obvious that they should be though, this is a quirk of the dSprites dataset.

[YannDubs]

Thanks for digging into it. What exactly do you mean by shuffling ? We do no shuffle the test loader (

disentangling-vae/main.py

Line 235 in a54b794

shuffle=False,

) if that's your point.

BTW : I'm more than happy to accept PRs