aboudykreidieh/h-baselines

A repository of high-performing hierarchical reinforcement learning models and algorithms.

h-baselines's Issues

Swtiching Policies w/ SNN

Visualize multiagent_traffic_light_grid

Hi,

Sorry I'm reposing this from flow repo, kinda in a desperate rushing condition for tests.
I've tried this by just importing the MultiFeedForwardPolicy in run_eval.py, apparently doesn't work and require more work.

Any plan to deploy this soon?
Thanks.

"AntFall" "AntGather" environemnt import error

Maybe there are some mistakes when using envs from "efficient-hrl", I guess maybe it mainly for compatible problem?
Error information:
AttributeError: 'AntFall' object has no attribute 'wrapped_env'

determine a good value for "c" in hierarchical policies

modifications to training LSTMs

See:

• https://gist.github.com/awjuliani/35d2ab3409fc818011b6519f0f1629df
• https://arxiv.org/abs/1609.05521

enable training with observation normalization

Tasks:

• make sure it works
• modify it if it doesn't
• unit test it
• rerun fully connected networks on HalfCheetah-v2 with it
• rerun fully connected networks on Hopper-v2 with it
• run LSTMs on HalfCheetah-v2 with it
• run LSTMs on Hopper-v2 with it

update logged names to include date

tests for replay buffers

Model Abstractions

DeepMind Locomotive

Original paper: https://arxiv.org/pdf/1610.05182.pdf

Conflicting Dependencies

Installing the repository on Linux (as detailed in the README) gives me the following error when executing pip install -e .:

ERROR: Cannot install h-baselines because these package versions have conflicting dependencies.

The conflict is caused by:
    gym 0.14.0 depends on cloudpickle~=1.2.0
    tensorflow-probability 0.8.0 depends on cloudpickle==1.1.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Is this a known issue, or have I gone wrong at some point?

Best,
Markus

Installation and Conda Envs

• create setup.py
• add a version and version file
• add method to create and environment during setup, along with an environment.yaml file

Option-Critic

Original paper: https://arxiv.org/pdf/1609.05140.pdf

Feudal Networks

Original paper: https://arxiv.org/pdf/1703.01161.pdf

Data-Efficient HRL

Original paper: https://arxiv.org/pdf/1805.08296.pdf

Algortihm and importation abstractions

two problems （HAC and AntPush)

I used h-baselines to reproduce HIRO and HAC. But there are two problems:

1. HAC performance is poor. This is somewhat different from the performance in the HAC paper. Is it the reason for the code or something?
2. When I was doing “AntPush” experiment, the command was like "python experiments/run_hrl.py "AntPush" --use_huber --evaluate --eval_interval 50000 --nb_eval_episodes 50 --total_steps 3000000 --relative_goals --off_policy_corrections" . Are these settings correct? Because I run like this, HIRO's success rate has always been 0.

Unit testing Abstractions

• Create a test file, and add dummy tests
• Check the integration of travis
• Find out how to output test coverage
• add enforcing pep8 and pydoc with numpy style

Where is the benchmarking results?

The link in README.md seems invalid.
And are there any docs to describe how to reproduce benchmark results?

Training Abstractions

Hierarchical Latent Space

Original paper: https://arxiv.org/pdf/1804.02808.pdf

Compatibility with Ray and EC2 Computing

Documenting the README

What's the use of 'evaluate' in AntMaze env?

In AntMaze env, if 'evaluate' is true then there will be 3 AntMaze env with different context goal. Is the use of 'evaluate' testing the performance of agent instead of training?

replace 8 with memory.trace_length

fix hierarchical memory replay

separate actor and critic hidden_layers input

Support for Discrete action spaces

It appears the implementations here currently don't support using Gym environments with Discrete action spaces. For example, the following code produces an error:

$ python -c 'from hbaselines.algorithms import RLAlgorithm; from hbaselines.fcnet.sac import FeedForwardPolicy; alg=RLAlgorithm(policy=FeedForwardPolicy, env="CartPole-v0", total_steps=1000000)'
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

pygame 2.0.1 (SDL 2.0.14, Python 3.7.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
2021-09-16 13:25:32.015754: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-09-16 13:25:32.031343: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599880000 Hz
2021-09-16 13:25:32.031693: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56147bad4370 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-09-16 13:25:32.031734: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jlilly7/Code/h-baselines-d/hbaselines/algorithms/rl_algorithm.py", line 581, in __init__
    self.trainable_vars = self.setup_model()
  File "/home/jlilly7/Code/h-baselines-d/hbaselines/algorithms/rl_algorithm.py", line 689, in setup_model
    **self.policy_kwargs
  File "/home/jlilly7/Code/h-baselines-d/hbaselines/fcnet/sac.py", line 205, in __init__
    self._ac_means = 0.5 * (ac_space.high + ac_space.low)
AttributeError: 'Discrete' object has no attribute 'high'

Given how broadly applicable discrete action spaces are, it would be good for this repo to support them. Alternatively, if I've misunderstood and done something wrong, please let me know.

I'm currently attempting to make adjustments for discrete environments in a fork; possibly a pull request could come out of this if it works well, but no promises yet.

action=nan problem

Hello. When I use run_hrl to run "python run_hrl "HalfCheeth-v2" ", a problem comes with action=nan. It occures in self._policy() of ri_algorithm.py. Of course, run_fcnet is well. How can I deal with it?