Question about finetune.py

[jannerm]

Question about finetune.py

[jannerm]

It looks like you might be looking at the RWR train step defined in training/diffusion.py. The DDPO update step is defined in training/policy_gradient.py:L63.

[chaojiewang94]

Thanks, I am reproducing your work with GPU and almost finish it. I am one more question about your work.

What is the difference bettween policy-gradient verision of DDPO and GAN (maxmizing weighted likelihood)

Assume we have a fixed discriminator r(x_0), which is also treated as a reward model.
The obective function of the policy-gradient verision of DDPO and GAN should be the same, formulated as
E_x~p_\theta(x)[r(x_0)], the only difference is how to estimate or optimize this function, is that correct?

[chaojiewang94]

@jannerm

[jannerm]

There is indeed a connection between policy gradient algorithms and GANs, though it requires a little more machinery. I recommend checking out @caseychu's great paper on Probability Functional Descent for the full story.

[chaojiewang94]

@jannerm Thanks!