Minor documentation issue with vanilla policy gradients

[rll]

Minor documentation issue with vanilla policy gradients

[dementrock]

Having a discount factor by itself reduce variance while introducing bias. However, when using the discount factor, when computing the advantage for a time step t you can either start counting the discount as gammat, or start counting the discount as gamma0. The latter will have larger variance, but it's less biased when the true objective does not contain a discount.

The extra 1/T term is purely notational, since it's assumed that each trajectory has up to T time steps, and there are N trajectories in total.

[DanielTakeshi]

OK I think I got confused with what you were referring to. I think the conclusions are now:

No discount factor = no bias, but high variance
Discount factor, starting at gamma^t = some bias, lower variance
Discount factor, starting at gamma^0 = some bias (but less than if we had gamma^t), somewhat higher variance (but less variance than no discount factor).

Though this is probably going to be dependent on the true objective, as you mentioned.