trust-region-policy-optimization-tensorflow's Introduction

parallel-trpo

Acknowledge

Citation

@article{DBLP:journals/corr/SchulmanLMJA15,
  author    = {John Schulman and
               Sergey Levine and
               Philipp Moritz and
               Michael I. Jordan and
               Pieter Abbeel},
  title     = {Trust Region Policy Optimization},
  journal   = {CoRR},
  volume    = {abs/1502.05477},
  year      = {2015},
  url       = {http://arxiv.org/abs/1502.05477},
  timestamp = {Wed, 07 Jun 2017 14:42:34 +0200},
  biburl    = {http://dblp.uni-trier.de/rec/bib/journals/corr/SchulmanLMJA15},
  bibsource = {dblp computer science bibliography, http://dblp.org}
}

Contributions

Update for TensorFlow 1.3
Fix some bugs. In file main.py, Add

history["maxkl"] = []

About

A parallel implementation of Trust Region Policy Optimization (TRPO) on environments from OpenAI Gym.

Now includes hyperparaemter adaptation as well! For more info, check Kevin Frans' post on this project.

Kevin Frans is working towards the ideas at this openAI research request. The code is based off of this implementation.

Kevin Frans is currently working together with Danijar on writing an updated version of this preliminary paper, describing the multiple actors setup.

How to run:

# This just runs a simple training on Reacher-v1.
python main.py

# For the commands used to recreate results, check trials.txt

Parameters:

--task: what gym environment to run on
--timesteps_per_batch: how many timesteps for each policy iteration
--n_iter: number of iterations
--gamma: discount factor for future rewards_1
--max_kl: maximum KL divergence between new and old policy
--cg_damping: damp on the KL constraint (ratio of original gradient to use)
--num_threads: how many async threads to use
--monitor: whether to monitor progress for publishing results to gym or not