jmtomczak/intro_dgm

"Deep Generative Modeling": Introductory Examples

"Deep Generative Modeling": Introductory Examples

This repository contains examples of deep generative models for the book "Deep Generative Modeling":

1. Mixture of Gaussians (MoGs): a mixture of Gaussians
2. Autoregressive Models (ARMs): ARMs parameterized with Causal Convolutionas and Transformers
3. Flow-based models (flows): RealNVP and IDFs (Integer Discrete Flows)
4. Variational Auto-Encoders (VAEs): a plain VAE and various priors, a hierarchical VAE
5. Diffusion-based Deep Generative Models (DDGMs): a Gaussian forward diffusion
6. Score-based Generative Models (SBGMs): score matching, score-based generative models, (conditional) flow matching
7. Hybrid modeling
8. Energy-based Models
9. Generative Adversarial Networks (GANs)
10. Neural Compression with Deep Generative Modeling
11. Large Language Models (LLMs)

The examples might look oversimplistic but that's the point! My idea is that everyone is able to follow every line of the code, and run the experiments within a couple of minutes on almost any laptop or computer. My goal is to encourage people who are new to understand and play with deep generative models. More advanced users, on the other hand, could refresh their knowledge or build on top of that to quickly check their ideas. Either way, I hope the code will help everyone to join a fascinating journey on deep generative modeling!

Requirements

In all examples, we used:

• pytorch 1.7.0
• numpy 1.17.2
• matplotlib 3.1.1
• scikit-learn 0.21.3
• pytorch-model-summary 0.1.1
• jupyter 1.0.0

Examples

All examples of implemented deep generative models are provided as jupyter notebooks. They can be find in the following folders:

1. mog: an example of a mixture of Gaussians with equiprobable components or trainable component probabilities.
2. arms: an example of an autoregressive model with a causal convolutiona layer in 1D and transformers.
3. flows: an example of a flow-based model, namely, RealNVP with coupling layers and permutation layers, and IDFs (Integer Discrete Flows).
4. vaes: (i) an example of a Variational Auto-Encoder using fully-connected layers and a standard Gaussian prior, (ii) an example of various priors for VAEs, (iii) an example of a hierarchical VAE.
5. ddgms: an example of a Diffusion-based Deep Generative Model using a Gaussian forward diffusion with a fixed variace and a reverse diffusion parameterized by an MLP.
6. sbgms: (i) an example of a score model using the score matching method and an MLP-based score model, (ii) an example of an SDE-based diffusion model parameterized by an MLP, (iii) an example of a conditional flow matching model parameterized by an MLP.
7. hybrid_modeling: an example of a hybrid model using fully-connected layers and IDFs.
8. ebms: an example of an energy-based model parameterized by an MLP.
9. gans: an example of a GAN parameterized by MLPs.
10. neural_compression: an example of applying deep generative modeling to image neural compression.
11. llms: an example of a decoder-based transformer (an LLM; here we call it teenyGPT.

Citation

If you use this code in any way, please refer to it by citing my book "Deep Generative Modeling":

• APA style:

Tomczak, J. M. (2022). Deep Generative Modeling. Springer Nature

• Bibtex:

@book{tomczak2022deep,
  title={Deep Generative Modeling},
  author={Tomczak, Jakub M},
  publisher={Springer Nature},
  year={2022}
}

intro_dgm's Issues

Learnable modules in ddgm example

In the ddgm example shouldn't the line:

self.p_dnns = p_dnns

be:

self.p_dnns = nn.ModuleList( p_dnns)

My understanding is that by not placing them in a ModuleList the p_dnns networks are not added in the learned parameters and are left unoptimized.

Thanks for an awesome resource by the way!

Missing something in DDGM example

Hi,

I might be missing a step here but in your explanation for the reparameterization trick you state the following formula

And I can see $i$ as an argument, but I fail to see how you are using it in this function - could you explain what I missed?

def reparameterization_gaussian_diffusion(self, x, i):
    return torch.sqrt(1. - self.beta) * x + torch.sqrt(self.beta) * torch.randn_like(x)

typo in fm_example

Hi, it seems there is a copy-paste typo in fm_example (not sure if eval/train are needed there at all):
self.vnet.eval() # set the vector field net to train again

arm_example forward pass A option related confusion

Hi,
As far as I understand, the option A in arm_example.ipynb refers to causality. Therefore A = True should signal that the model should not be taking the current time step t as an input. In the code implementation we discard the latest outpout of the convolution layer. I am confused by this since I thought we are supposed to discard the latest input. Are we not supposed to discard the latest (current time step) input to the model. In other words, for example, for the first convolution operation for t = 0, are we not supposed to pad left (for 1D) with F-1 many elements, and feed into the first convolution the {(F-1), (F-2), ..., -1} time step elements to produce the output for t = 0 (initial) ?

if self.A:
# Remember , we cannot be dependent on the current
component; therefore , the last element is removed .
return conv1d_out [:, :, : −1]

Sorry if I'm off. Just started to learn about these from the book. I will appreciate any help.

The use of tanh in Euler forward sampling

I'm checking your most recent update on conditional flow matching and I see here you passed a tanh function after the Euler forward step. I wonder if there is any reason behind it?

https://github.com/jmtomczak/intro_dgm/blob/main/sbgms/fm_example.ipynb

    def sample(self,  batch_size=64):
        # Euler method
        # sample x_0 first
        x_t = self.sample_base(torch.empty(batch_size, self.D))
        
        # then go step-by-step to x_1 (data)        
        ts = torch.linspace(0., 1., self.T)
        delta_t = ts[1] - ts[0]
        
        for t in ts[1:]:
            t_embedding = self.time_embedding(torch.Tensor([t]))
            x_t = x_t + self.vnet(x_t + t_embedding) * delta_t
            # Stochastic Euler method
            if self.stochastic_euler:
                x_t = x_t + torch.randn_like(x_t) * delta_t
        
        x_final = torch.tanh(x_t) # **here's my question** 
        return x_final