jianing-sun/interpolated-policy-gradient-with-ppo-for-robotics-control-

Reinforcement Learning for robotics continuous control, mainly based on Proximal Policy Optimization, extending to Interpolated Policy Gradient and Hindsight Experience Replay (HER)

• Some Results based on FetchReach-v0 environment between IPG and PPO:

A slide for initial result and comparison among PPO, IPG and HER+IPG based on multi-goal RL environment (FetchReach-v0 from Robotic of Gym).

(./Results/Apr-12_23:16-Seed1234.png)

• TODO:

• Combine with Experience Replay!!

• Clean up current neural networks settings and

• Check if it's under correct formula from the paper

• Combine with basic IPG

• More debug, verify if the algorithm is working as it was totally created based on my understanding of the algorithm diagram from the initial paper (it works)

• More experiments on pushing, sliding, pick&place

• Generalize to hindsight experience replay and compare with experience replay

• Change tanh to ReLU

• Change mini batch size to 256 (so far is 500)

• For pushing, sliding tasks, need to train more episodes (in original papar, they train for 50 epochs (one epoch consists of 19 · 2 · 50 = 1 900 full episodes, which amounts to a total of 4.75 · 106 timesteps). And also improve the time steps to 2500 per episode for these tasks. (Should not change the time steps configuration in Gym, but must train with much more episodes, in Multi-goal Reinforcement learning (no.5 reference), they train more than 50000 episodes to see little improvement).

• Try to delete coefficient (1/ET) for on-policy loss

• Add a stochastic target policy for critic of IPG

• Reference:

  • Proximal Policy Gradient
  • Trust Region Policy Gradient
  • Interpolated Policy Gradient
  • Hindsight Experience Replay
  • Generalized Advantage Estimator
  • Multi-goal Reinforcement Learning

 "CODE IS FAR AWAY FROM BUG WITH THE ANIMAL PROTECTING"
 
 *          ##2      ##2
 *       ┏-##1　  ┏-##1
 *    ┏_┛ ┻---━┛_┻━━┓
 *    ┃　　　        ┃　　　　 
 *    ┃　　 ━　      ┃　　　 
 *    ┃ @^　  @^    ┃　　 
 *    ┃　　　　　　  ┃
 *    ┃　　 ┻　　　 ┃
 *    ┃_　　　　　 _┃
 *     ┗━┓　　　┏━┛
 *    　　┃　　　┃神兽保佑
 *    　　┃　　　┃永无BUG！
 *    　　┃　　　┗━━━┓----|
 *    　　┃　　　　　　　  ┣┓}}}
 *    　　┃　　　　　　　  ┏┛
 *    　　┗┓&&&┓-┏&&&┓┏┛-|
 *    　　　┃┫┫　 ┃┫┫
 *    　　　┗┻┛　 ┗┻┛
 *
 *
 "CODE IS FAR AWAY FROM BUG WITH THE ANIMAL PROTECTING"

​

Jianing Sun

Last modified: April 23th, 2018

interpolated-policy-gradient-with-ppo-for-robotics-control-'s Issues

Trouble running IPG

File "main.py", line 258, in main
ceval = CriticEval(qf(), on_policy)
TypeError: 'ContinuousQFunction' object is not callable

How to run the main.py in HER

Hey guy,excuse me,can you tell me how to run the main.py in HER?Thank you