jcwleo/awr-pytorch

Advantage-Weighted Regression

awr-pytorch's Issues

done/dones ?

Hi ! Thanks for the implementation,

In your discount function, you write:

def discount_return(reward, done, value):
    value = value.squeeze()
    num_step = len(value)
    discounted_return = np.zeros([num_step])

    gae = 0
    for t in range(num_step - 1, -1, -1):

        if dones[t]:
            delta = reward[t] - value[t]
        else:
            delta = reward[t] + gamma * value[t + 1] - value[t]
        gae = delta + gamma * lam * (1 - done[t]) * gae

        discounted_return[t] = gae + value[t]

Shouldn't the first condition be if done[t] ?

Thanks !

Status of this project

Hello, I am reimplementing AWR in my own PyTorch codebase, and I haven't been able to get it to work as well as in the paper (on PyBullet Gym environments). I think this is the only other implementation of AWR I have found apart from the initial TF code, but it doesn't seem to work.

Have you been able to get it to train policies well?

buffer 대신 mp 로 한 이유가 궁금하군요

jason peng의 논문보면 아예 awr를 off policy로 분류하고, buffer sampling을 하는데 코드는 ppo 나 a3c 처럼 mp로 구현되어 있길래 여쭤보고 싶었습니다.