Inplace error when running FactorVAE

[yanndubs]

Inplace error when running FactorVAE

[DianeBouchacourt]

With a bit more research, it comes from the call of optimizer.step() here

disentangling-vae/disvae/models/losses.py

Line 283 in 535bbd2

optimizer.step()

commenting it out works fine, but then how to do the optimization step of the model before training the discriminator?

[YannDubs]

Hi @DianeBouchacourt,

This seems to be an issue with pytorch > 1.4 (e.g. pytorch/pytorch#39141 (comment) )

I.e. I'm currently modifying the weights before finishing computing the gradients, which is not allowed from pytorch 1.5.

I see 2 possibilities:

• downgrading to 1.4. Probably the quickest for now.
• moving VAE optimization right before discriminator optimization. I believe this should work, but I have to test it.

[YannDubs]

It should work now with Pytorch 1.7 (I only tested factor_mnist). Let me know if it doesn't and thank you for pointing the issue.

[DianeBouchacourt]

All good it works now, but doesn't it change the optimization dynamics ? Since now model parameters are not updated via optimizer.step() before latent_sample2 = model.sample_latent(data2) is called

[YannDubs]

I believe that it does slightly change the dynamics (in that the discriminator now tries to discriminate sample from "the previous step" but that shouldn't make a big difference). I don't see how to bypass that "issue" without computing the first sample twice (once for each optimizer)... But if you have a better suggestion I'm happy to change that.

What is important is that the gradient computation is correct, and I think it is because we are detaching the second sample z_perm = _permute_dims(latent_sample2).detach() and so no gradients flow in the VAE when computing the gradients for the discriminator.

[DianeBouchacourt]

OK thanks for the quick reply !