gth828r/ppo

An implementation of PPO in Tensorflow

ppo's Issues

Implement CQL

We need to implement the pieces of CQL for offline RL. See https://bair.berkeley.edu/blog/2020/12/07/offline/ for details.

Experiment with CQL

Play around with our PPO implementation both with and without CQL. Evaluate what impact it has.

Implement PPO with Keras API

We need an implementation of PPO. Start with using the Keras API.

Design plan to verify that PPO implementation is working

Once we have our high level PPO implementation, we should verify that it is working. Design a test plan to confirm this, and carry it out.

Make platform-specific encapsulation of constants

To improve portability, we should introduce a mechanism to inspect the underlying hardware (e.g. GPU, TPU, CPU-only) and set values based on that. We can create a factory pattern to return a set of pre-defined constants for hardware we recognize, as well as a default to use for non-CPU hardware which we do not recognize.

Choose an environment to test with

We need a simulation environment to test with. Something like an environment from AI Gym. Choose one to explore the capabilities of PPO combined with CQL.

Convert PPO code into a callable module

Now that we have a working version of PPO in notebook form over in #1, it would be nice to convert it into a python module which can be imported and called.