stanford-iprl-lab/grac

implementation of our self-guided and self-regularized actor-critic algorithm

grac's Issues

Stabilizing Learning

Hi, thank you for writing this paper and open sourcing this code!
I really like the part where we can "get rid of the target network".

I'm trying to reproduce the paper's result with my own implementation and I have few questions

1. Learning is often stable but in one or two seeds that I am using, the return plummets to 0 (pretty much breaking the Q-function from what I am seeing). My implementation currently does not have any scheduling for alpha and learning rate. From your experience, how important is that?

2. Looking at the hyperparameters table from the paper, The CEM loss weight is somewhat arbitrary. Can you please give me some insight on these values? I also noticed that the actor is slightly smaller in learning rate, does 3e-4 hinder the agent from training well?

Thanks again and this paper is very interesting :)

Given example does not run

Hi,

Thanks for open-souring the code. I am trying to play around with the code. But it seems the GRAC class in grac.py misses some keyword arguments. And I couldn't run the given example.

Traceback (most recent call last):
  File "main.py", line 122, in <module>
    policy = GRAC.GRAC(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'no_critic_cem'

Traceback (most recent call last):
  File "main.py", line 122, in <module>
    policy = GRAC.GRAC(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'policy_noise'