sapanachaudhary/pytorch-cpo

PyTorch implementation of Constrained Policy Optimization

PyTorch implementation of Constrained Policy Optimization (CPO)

This repository has a simple to understand and use implementation of CPO in PyTorch. A dummy constraint function is included and can be adapted based on your needs.

Pre-requisites

• PyTorch (The code is tested on PyTorch 1.2.0.)
• OpenAI Gym.
• MuJoCo (mujoco-py)
• If working with a GPU, set OMP_NUM_THREADS to 1 using:

export OMP_NUM_THREADS=1

Features

1. Tensorboard integration to track learning.
2. Best model is tracked and saved using the value and standard deviation of average reward.

Usage

• python algos/main.py --env-name CartPole-v1 --algo-name=CPO --exp-num=1 --exp-name=CPO/CartPole --save-intermediate-model=10 --gpu-index=0 --max-iter=500

Code Reference

• Khrylx/PyTorch-RL

Technical Details on CPO

pytorch-cpo's Issues

NameError: name 'avg_reward' is not defined

When running the
python algos/main.py --env-name CartPole-v1 --algo-name=CPO --exp-num=1 --exp-name=CPO/CartPole --save-intermediate-model=10 --gpu-index=0 --max-iter=500, I encountered the following error:

Traceback (most recent call last):
  File "algos/main.py", line 229, in <module>
    main_loop()
  File "algos/main.py", line 224, in main_loop
    save_info_obj.dump_lists(avg_reward, num_of_steps, num_of_episodes, total_num_episodes, total_num_steps, rewards_std, env_avg_reward, v_loss_list, p_loss_list, eval_avg_reward, eval_avg_reward_std)
NameError: name 'avg_reward' is not defined