fabiopardo/tonic

Tonic RL library

tonic's Issues

No support/benchmark results for Atari games

It seems the current version doesn't support games like Breakout. The following code raise an error:

python -m tonic.train --header "import tonic.torch" --agent "tonic.torch.agents.PPO()" --environment "tonic.environments.Gym('Breakout-v0')" --name PPO-X --seed 0

AssertionError: assert isinstance(env.action_space, gym.spaces.Box)

Is that supposed to happen or the code is ready to use for other environments too other than the ones that you've provided the benchmarks. BTW, I think the benchmark idea for fast comparison is really cool. Thanks for sharing!

training custom envs

hello, thanks for very nice library with baseline RL implementations, which doesn't require to load tensorflow :)
I have a question how to load a custom environment ? a wrapper is required ?

How do you resume training from a checkpoint?

Can I ask you the difference between the MPO and the MO-MPO? I see you link to both

In your code here you link the MO-MPO paper.
I don't know the math well enough to know the difference.
I'm happy to find it in Torch, but I don't know that I could convert an MPO to a MO-MPO.

Any plans to write MO-MPO in torch? Any interest in writing it or getting on a Zoom call with me for a few satoshis?

Training using reference data

Hi

Is it possible to use pre-recorded demonstration data for the training? If yes, which algorithms support this feature?

Thanks

Any plans for CUDA support?

In my system, python is alias to old Python 2.x

Maybe you should use python3 in examples?

Minimal plotting example returning error

I ran the minimal working example for BipedalWalker and when I ran the plot script, I got this trace:

(base) samlerman@Marvin vcpkg % python3 -m tonic.plot --path BipedalWalker-v3/ --baselines all Loading data... BipedalWalker-v3 PPO-X Loading TensorFlow baselines... Path: /Users/samlerman/Code/Libraries/tonic/data/logs/tensorflow_logs.pkl BipedalWalker-v3 A2C BipedalWalker-v3 DDPG BipedalWalker-v3 MPO BipedalWalker-v3 PPO BipedalWalker-v3 SAC BipedalWalker-v3 TD3 BipedalWalker-v3 TRPO Plotting... Traceback (most recent call last): File "/Users/samlerman/code/miniforge3/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/Users/samlerman/code/miniforge3/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/samlerman/Code/Libraries/tonic/tonic/plot.py", line 469, in <module> fig = plot(**vars(args), fig=None) File "/Users/samlerman/Code/Libraries/tonic/tonic/plot.py", line 399, in plot target_width = h_packer.get_extent(renderer)[0] File "/Users/samlerman/code/miniforge3/lib/python3.9/site-packages/matplotlib/offsetbox.py", line 341, in get_extent w, h, xd, yd, offsets = self.get_extent_offsets(renderer) File "/Users/samlerman/code/miniforge3/lib/python3.9/site-packages/matplotlib/offsetbox.py", line 540, in get_extent_offsets dpicor = renderer.points_to_pixels(1.) AttributeError: 'NoneType' object has no attribute 'points_to_pixels'

Regarding the D4PG implementation

First of all, thank you for providing an excellent RL library.

If I understand correctly, according to the D4PG paper, we should create multiple actors. But it seems the implementation added from DDPG is only the distributional critic part?

reference: https://arxiv.org/pdf/1804.08617v1.pdf

[Feedback] Please consider using Hydra

Hello @fabiopardo :)

Thank you for open-sourcing tonic. It looks pretty interesting and useful. I have one feedback:

While I can understand the motivation of using snippets of Python code to configure the experiment, specifying these things via cmd makes the user experience much less pleasent. May I suggest you consider using Hydra. I think it will improve the user experience and make the development workflow easier. This blog post is a pretty good intro to Hydra.

Win 10 compatibility issue with gym box (Comment Not an ISSUE)

Unfortunately for some reason, in Windows 10, there will be an assertion error for all the environments, even for the provided ones. It seems to me this is a Win 10 compatibility problem or a version issue with the Numpy or any other packages because simply commenting the following line in the gym\spaces\box.py can resolve the problem.

assert np.isscalar(low) and np.isscalar(high)

For example, without removing that line, the following code is not going to run. However, after commenting, it's working.

python -m tonic.train --header "import tonic.torch" --agent "tonic.torch.agents.PPO()" --environment "tonic.environments.Gym('BipedalWalker-v2')" --name PPO-X --seed 0

Hopefully, this helps others with the same problem.

Plotting bug

Hi, renderer from plot.py kept returning None and as a result I kept getting this error:

AttributeError: 'NoneType' object has no attribute 'points_to_pixels'

To fix it, I just commented out these lines:

        renderer = legend_ax.get_renderer_cache()
        h_packer = legend.get_children()[0].get_children()[1]
        target_width = h_packer.get_extent(renderer)[0]
        current_width = sum(
            [ch.get_extent(renderer)[0] for ch in h_packer.get_children()])
        if target_width > 1.3 * current_width:
            break

Then plotting seemed to work fine, but may I ask, what do those lines do? The figure looks fine without them.