henrycharlesworth/multi_action_head_ppo

PPO with multi-head/autoregressive action outputs

Hi there 👋

multi_action_head_ppo's Issues

Should we mask out the entropy

Hey,

Thanks for your work.

There is one point I am not quite sure. Since the action probs for all the heads except for the first catgorical one are masked out, should we also mask out the corresponding entropy？

multi_action_head_PPO/core/multi_action_heads.py

Line 107 in fa24cfd

joint_action_log_prob += (log_prob_mask * head_dist.log_probs(head_action))

multi_action_head_PPO/core/multi_action_heads.py

Line 111 in fa24cfd

entropy += head_dist.entropy().mean()

The abstract method "env_is_wrapped" in VecEnv in "stable_baselines3" should be implemented

When I use stable_baselines3 1.6.2 to run the sample command in README.md, the program does not run and the error message indicates:

Traceback (most recent call last):
File "main.py", line 164, in
main()
File "main.py", line 36, in main
envs = make_vec_envs(args.env_name, args.seed, args.num_processes,
File "/home/cenyu/multi_action_head_PPO/core/envs.py", line 75, in make_vec_envs
envs = SubprocVecEnv(envs)
TypeError: Can't instantiate abstract class SubprocVecEnv with abstract methods env_is_wrapped

Once I implemented the corresponding abstract method env_is_wrapped in the SubprocVecEnv class in subproc_vec_env.py, the sample works fine