antymon/ppo_cpp

C++ version of a Deep Reinforcement Learning algorithm PPO

ppo_cpp's Issues

parallel running

Hi Antymon:
Your PPO code is interesting. I am wondering whether it can be used for parallel training, where each episode is a mpi/openmp solver. I would like to perform simulations on a cluster with each episode as a single node openmp based simulation. do you have any suggestion?

Yours Sincerely

Wei

Parallel branch requires fixes of master

Some cherry-picking needs to be done.

Adjustments needed to make for running my task

Hi Antymon,
Thanks very much for the repo so that people can export the trained agent from python to c++.

I would like to confirm the changes I have to make in order to run my own task:

1. Make my environment inherit from Env abstract class under env\env.hpp

2. Modify the main ppo2.cpp which creates instance of an environment and passes it to PPO

*I have two questions:
a. if I would like to run inference only, should I use algorithm.eval(obs) directly? As it seems it uses get_deterministic_action() and I noticed there are also step(const tensorflow::Tensor& obs) and value(const tensorflow::Tensor& obs) which I can't tell exactlly what the differences are between them.

b. It seems a way to resume training using your implementation so that online learning can be achieved?

3. Create own computational graph and potentially make some small modifications to the core algorithm if using more involved policies (currently implementation supports only MLP policies). Graph generation is mentioned below.

• I used 'tf.train.export_meta_graph(graph=model.graph, filename='my-model.meta', clear_devices=True, clear_extraneous_savers=True, strip_default_attrs=True)' as it mentioned here but it has many redundant tensors in the generated model when I visualize it. Some say its because some stuff used for training are preserved. I was wondering have you encountered such situation and how did you solve it?

Thank you again!

Build of simg file fails

Due to main dependency of the Singularity file, it is impossible to build a project simg file since 29th Nov. I am working to resolve this problem. If the previous parent image file was cached, it would be possible to build against this file as a workaround.

Possible g++ compilation?

Dear Antymon,
I am at the stage of trying to compile my_own_ppo2.cpp with my_own_env.hpp inherited from the env.hpp. I am wondering if there is an option of not using the singularity img, eg MAKEFILE style compilation? I compiled the tensorflow from sourse using bazel and downloaded the Eigen/Dense ,but there are still some denpendency issues while I was comiling.
I appreciate any helps on this.

Thank you very much,
Wei