nikhilbarhate99/hierarchical-actor-critic-hac-pytorch

PyTorch implementation of Hierarchical Actor Critic (HAC) for OpenAI gym environments

Hierarchical-Actor-Critic-HAC-PyTorch

This is an implementation of the Hierarchical Actor Critic (HAC) algorithm described in the paper, Learning Multi-Level Hierarchies with Hindsight (ICLR 2019), in PyTorch for OpenAI gym environments. The algorithm learns to reach a goal state by dividing the task into short horizon intermediate goals (subgoals).

Usage

• All the hyperparameters are contained in the train.py file.
• To train a new network run train.py
• To test a preTrained network run test.py
• For a detailed explanation of offsets and bounds, refer to issue #2
• For hyperparameters used for preTraining the pendulum policy refer to issue #3

Implementation Details

• The code is implemented as described in the appendix section of the paper and the Official repository, i.e. without target networks and with bounded Q-values.
• The Actor and Critic networks have 2 hidded layers of size 64.

Citing

Please use this bibtex if you want to cite this repository in your publications :

@misc{pytorch_hac,
    author = {Barhate, Nikhil},
    title = {PyTorch Implementation of Hierarchical Actor-Critic},
    year = {2021},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/nikhilbarhate99/Hierarchical-Actor-Critic-HAC-PyTorch}},
}

Requirements

• Python 3.6
• PyTorch
• OpenAI gym

Results

MountainCarContinuous-v0

(2 levels, H = 20, 200 episodes)	(3 levels, H = 5, 200 episodes)

(2 levels, H = 20, 200 episodes)

References

• Official Paper and Code (TensorFlow)

hierarchical-actor-critic-hac-pytorch's Issues

Hindsight Experience Replay .

that HINDSIGHT GOAL TRANSITIONS described in the paper used the state as the action in the transition but i did not see in the code

or did i miss some details?

DDPG implementation does not use Ornstein-Uhlenbeck process for time-correlated exploration?

Reg. Getting Gradient Loss values while testing

Hi I want to get the gradient loss with respect to input while I am testing. I tried the following in HAC.py after the next_state, rew, done, _ = env.step(action) line in the low level policy:

state = torch.FloatTensor(state).to(device)
action = torch.FloatTensor(action).to(device)
goal = torch.FloatTensor(goal).to(device)
criticval = self.HAC[0].critic(state, action, goal).detach()

with plans of computing a new target_Q in the moment to calculate loss and then gradient. However, I get the error Dimension out of range (expected to be in range of [-1, 0], but got 1). Any reason why this is happening?

Goal for pendulum?

action and state offsets?

I am just curious what do the action/state offset values mean?

https://github.com/nikhilbarhate99/Hierarchical-Actor-Critic-HAC-PyTorch/blob/master/train.py#L36

I can't seem to figure it out. How do you determine them, for example, for a new environment?

Similarly, for the clip low/high values for both action and states? If you could explain those as well I would appreciate it.

Thank you.