silvicek/distributional-dqn

Implementation of 'A Distributional Perspective on Reinforcement Learning' and 'Distributional Reinforcement Learning with Quantile Regression' based on OpenAi DQN baselines.

Distributional DQN

Implementation of 'A Distributional Perspective on Reinforcement Learning' and 'Distributional Reinforcement Learning with Quantile Regression' based on OpenAi DQN baseline.

C51:

Quantile Regression: (see branch quantile)

Installation

Install the OpenAi fork https://github.com/Silvicek/baselines (parent changes a lot, compatibility isn't guaranteed) Then install requirements

pip3 install -r requirements.txt

Usage:

For simple benchmarking:

python3 train_[{cartpole, pong}].py
python3 enjoy_[{cartpole, pong}].py

For full Atari options see help

python3 train_atari.py --help

after learning, you can visualize the distributions by running

python3 enjoy_atari.py --visual ...

---

This implementation has been successfully tested on: Pong, Qbert, Seaquest

Some baseline features not supported (prioritized replay, double q-learning, dueling)

distributional-dqn's Issues

[Error report] baselines.common.tf_util functions

baselines.common.tf_util seems like it has no attribute 'ensure_tf_input' and 'BatchInput'
Any solutions?

train_cartpole.py

Hi Silvicek,

When I ran your train_cartpole.py, I got this error:

Traceback (most recent call last):
File "train_cartpole.py", line 40, in
main()
File "train_cartpole.py", line 33, in main
dist_params={'Vmin': 0, 'Vmax': 25, 'nb_atoms': 11}
File "C:\distributional-dqn-master\distdeepq\simple.py", line 201, in learn
dist_params=dist_params
File "C:\distributional-dqn-master\distdeepq\build_graph.py", line 215, in build_train
act_f = build_act(make_obs_ph, p_dist_func, num_actions, dist_params, scope=scope, reuse=reuse)
File "C:\distributional-dqn-master\distdeepq\build_graph.py", line 132, in build_act
observations_ph = make_obs_ph("observation")
File "C:\distributional-dqn-master\distdeepq\simple.py", line 186, in make_obs_ph
return U.BatchInput(env.observation_space.shape, name=name)
AttributeError: module 'baselines.common.tf_util' has no attribute 'BatchInput'

 Thanks for your help!
 W

Links in README.md are not working

The links in this README (https://github.com/Silvicek/distributional-dqn/blob/master/distdeepq/README.md) are not working.

The mean 100 episode reward first raise and then drop.

Hi,

When I run the QR-DQN code on cartpole environment, I meet a strange problem:

the mean 100 episode reward first raises rapidly (from 20 to about 170) but then drop to 100 after about 60000+ steps and finally coverage at that score. Do anyone meet similar problems? I am really confused about that and really need helps.

Thank so much!