gemcollector/rl-vigen

This is the repo of "RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization"

rl-vigen's Issues

About the scale of images

Thanks a lot for your wonderful work! I'm curious about the scale of image inputs in the robosuite. Do we directly input the image within [0, 255]? Or is it necessary to rescale it into [0, 1]? Thanks a lot!

Cannot reproduce results from the original papers

Hello @gemcollector
I have been trying to reproduce SGQN and PIEG results for DMC. For SGQN, for walker walk environment in hard settings, I get the results about 380. For train settings, I get the results about 750. If we look at the paper https://arxiv.org/pdf/2209.09203.pdf in Table 2 (page 9), the results must be much much higher.
I know those papers work on DMC-GB from Hansen (you also mentioned in the paper), but I don't see the major difference in both the environments from the eval videos (especially for DMC case). Can you please try to replicate and see if you get the same results, or can you explain what creates the difference.
If you need log files or any further explanation, please let me know.

programs are killed by Dataloader worker

Thanks a lot for the wonderful work! When I run the code with the task in the robosuite, sometimes the code will be killed with the Error information like

RuntimeError: DataLoader worker (pid 22358) is killed by signal: Killed.

It seems that the problem is because of the workers in Dataloader. Unfortunately, I still have this problem when I set replay_buffer_num_workers=1. Have you met this problem? Thanks a lot again!

Question about carla env

I'm trying to run carla_env, I ran bash scripts/carlatrain.sh directly and encountered the problem:
pygame 2.1.2 (SDL 2.0.16, Python 3.8.18) Hello from the pygame community. https://www.pygame.org/contribute.html workspace: /raid/fangyu_data/RL-ViGen/exp_local/2024.01.29/164236_action_repeat=2,feature_dim=50,lr=1e-4,nstep=1,num_train_frames=1001000,save_snapshot=True,save_video=False,seed=3,task=carla,use_tb=False,use_wandb=False,wandb_group= Error executing job with overrides: ['task=carla', 'seed=3', 'action_repeat=2', 'use_wandb=False', 'use_tb=False', 'num_train_frames=1001000', 'save_snapshot=True', 'save_video=False', 'lr=1e-4', 'feature_dim=50', 'nstep=1', 'wandb_group='] Traceback (most recent call last): File "carlatrain.py", line 262, in main workspace = W(cfg) File "/raid/fangyu_data/RL-ViGen/carlatrain.py", line 51, in __init__ self.setup() File "/raid/fangyu_data/RL-ViGen/carlatrain.py", line 71, in setup self.train_env = carla_make(action_repeat=self.cfg.action_repeat) File "/raid/fangyu_data/RL-ViGen/wrappers/carla_wrapper.py", line 109, in carla_make env = make_env_10(action_repeat) File "/raid/fangyu_data/RL-ViGen/./wrappers/carlaenv/utils.py", line 34, in make_env_10 env = CarlaEnv10(cfg_dict) File "/raid/fangyu_data/RL-ViGen/./wrappers/carlaenv/carla_env_10.py", line 263, in __init__ self.world = self.client.load_world(cfg_dict['map']) # change map here RuntimeError: time-out of 500000ms while waiting for the simulator, make sure the simulator is ready and connected to localhost:2018
I followed carla_README.md and wrote the code:

from carlaenv.utils import make_env_10
action_repeat = 2
env = make_env_10(action_repeat)

I have encountered the same problem. Can you please help me to solve it? Thank you

Question about the Adroit environment?

Hi, thanks for your great work. However, I have trouble finding the Dexterous Hand Manipulation mentioned in Sec. 4.1.3. of the paper. I searched the whole repo for Adroit / color_hard, but I could not find any code related to the experiment results.

I noticed there's a separate branch called ViGen-adroit, do I have to checkout to that branch to use the generalization environment?

The experiment cannot be reproduced.

When training svea or pieg on the walker_walk task of the dmc environment, even if I set the seed, the results are still different each time. why is that? Looking forward to your reply.

Error for no agent in locodmc_eval

RL-ViGen/locodmc_eval.py

Line 52 in 0158cec

self._global_step = agent['_global_step']

This throws error as there is no agent variable defined.
Instead, this should be:

agent = torch.load('%s/snapshot.pt' % (work_dir), map_location='cuda:0')
self.agent = agent['agent']
self._global_step = agent['_global_step']

Btw, great work!

About parallel training

Hello, authors,
Thanks for your excellent work, it is really helpful for the community.
But I am confused about how can we achieve parallel training (it is actually not an issue of this repo). For example, the image size of dm_control suite is about 84*84, and when I train one exp, the used GPU memory is quite small but the GPU-Util is high. If I manually train several seeds at the same time, every training process will be slow.
So my question is that how can I achieve parallel training to accelerate the process (multiprocessing?).

Error in saving video during evaluation

I changed the locodmc_eval.sh and added save_video=True in the script. It shows error while running, specifically:

  File "locodmc_eval.py", line 166, in main
    workspace.eval()
  File "/home/vedant/RL-ViGen/locodmc_eval.py", line 130, in eval
    self.video_recorder.save(f'{self.global_frame}.mp4')
  File "/home/vedant/RL-ViGen/video.py", line 40, in save
    imageio.mimsave(str(path), self.frames, fps=self.fps)
  File "/home/vedant/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/imageio/core/functions.py", line 408, in mimwrite
    raise ValueError("Image is not numeric, but {}.".format(imt.__name__))
ValueError: Image is not numeric, but NoneType.

This arises due to the fact that all the frames during rendering from video i.e returns None.

RL-ViGen/video.py

Line 34 in 0158cec

frame = env.render()

FYI, I have done this additionally in locodmc_eval.py to enable video capture:

RL-ViGen/video.py

Line 34 in 0158cec

frame = env.render()

to

self.video_recorder =VideoRecorder(self.work_dir if self.cfg.save_video else None)

Am I missing here something?

Questions Regarding the Adroit Testing Environment

I successfully trained the vrl3 model, but I have some questions during testing.
I discovered that during testing with mode video-hard, the agent's actual input is the observation obtained from observation = time_step.observation, as shown below:

However, neither of the two video recording functions returns something corresponding to the above image:

1: In the record_gif function, what's actually recorded is the image of size (224, 224, 3) rendered by self.eval_env._env._env.env.sim.render. The image and final video are as follows:

2: In self.video_recorder.record(self.eval_env), what's recorded is the image of size (224, 224, 3) returned by env.render(). The image and final video are as follows (these are also the images provided in the paper and on the website):

1000000.mp4

I would like to know why the images presented in the paper and on the website are not those of the agent's actual testing, or what do these two videos mean separately?
Thank you for adding new environments to the visual reinforcement learning generalization. I look forward to your response.

Missing evaluation scores on Robosuite with cross embodiments settings

Hello, @gemcollector,
I have noticed that you have loaded the evaluation scores of most experiments. However, the detailed evaluation scores on Robosuite with cross embodiments settings are not provded, specifically in Figure 9. Could you upload the missing results, please?

Running CARLA without rendering

Thank you for your nice benchmark.

Is it possible to run CARLA without a display? I want to run CARLA on a GPU server, but I encounter the following error.

Is there something I might have missed?

Errors encountered while reproducing Adroit environment

Thank you for this valuable project. I encountered an issue while trying to reproduce the Adroit environment and would appreciate your help.

I installed the environment using:

conda create -n vigen-adroit python=3.8
bash install_adroit.sh

However, when executing:

cd src/
bash train.sh

I encountered an error:
ModuleNotFoundError: No module named 'mj_envs.hand_manipulation_suite.door_shadowhand_v0'

After executing the commented-out commands in bash install_adroit.sh:

git clone https://github.com/watchernyu/rrl-dependencies.git \
   && cd rrl-dependencies \
   && pip install -e mj_envs/. \
   && pip install -e mjrl/.

I encountered a new error:

Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/_internal/utils.py", line 368, in <lambda>
    lambda: hydra.run(
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 110, in run
    _ = ret.return_value
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "train_adroit_dhm.py", line 535, in main
    workspace = W(cfg)
  File "train_adroit_dhm.py", line 73, in __init__
    self.setup()
  File "train_adroit_dhm.py", line 143, in setup
    self.train_env = dhm.make_env_RRL(env_name, test_image=False, num_repeats=self.cfg.action_repeat,
  File "/datadrive/xxx/code/RL-ViGen/DHM/dhm/make_env_RRL.py", line 894, in make_env_RRL
    env = DHMEnv(cfg_dict, env_name, test_image, cam_list, num_repeats, num_frames, env_feature_type, device,
  File "/datadrive/xxx/code/RL-ViGen/DHM/dhm/make_env_RRL.py", line 703, in __init__
    env = GymEnv(self.env_name)
  File "/datadrive/xxx/code/RL-ViGen/DHM/dhm/utils.py", line 27, in __init__
    env = gym.make(env)
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/gym/envs/registration.py", line 235, in make
    return registry.make(id, **kwargs)
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/gym/envs/registration.py", line 128, in make
    spec = self.spec(path)
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/gym/envs/registration.py", line 203, in spec
    raise error.UnregisteredEnv("No registered env with id: {}".format(id))
gym.error.UnregisteredEnv: No registered env with id: hammer-adroit-v0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_adroit_dhm.py", line 545, in <module>
    main()
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/_internal/utils.py", line 367, in _run_hydra
    run_and_report(
  File "/home/xxx/miniconda3/envs/vigen-adroit/lib/python3.8/site-packages/hydra/_internal/utils.py", line 251, in run_and_report
    assert mdl is not None
AssertionError

I suspect it might be a version issue. Do you know how to resolve this? Looking forward to your reply, and thank you in advance.

Save video option when training robosuite env

Hi!

I got the following error when training the robosuite env:

Traceback (most recent call last):
  File "train.py", line 304, in main
    workspace.train()
  File "/home/xzc/Documents/RL-ViGen/train.py", line 260, in train
    self.eval()
  File "/home/xzc/Documents/RL-ViGen/train.py", line 168, in eval
    self.video_recorder.init(self.eval_env, enabled=(episode == 0))
  File "/home/xzc/Documents/RL-ViGen/video.py", line 25, in init
    self.record(env)
  File "/home/xzc/Documents/RL-ViGen/video.py", line 39, in record
    frame = env.render()
  File "/home/xzc/Documents/RL-ViGen/wrappers/dmc.py", line 183, in __getattr__
    return getattr(self._env, name)
  File "/home/xzc/Documents/RL-ViGen/wrappers/robo_wrapper.py", line 106, in __getattr__
    return getattr(self._env, name)
  File "/home/xzc/mambaforge/envs/pieg/lib/python3.8/site-packages/dm_control/suite/wrappers/action_scale.py", line 103, in __getattr__
    return getattr(self._env, name)
  File "/home/xzc/Documents/RL-ViGen/wrappers/dmc.py", line 67, in __getattr__
    return getattr(self._env, name)
  File "/home/xzc/Documents/RL-ViGen/wrappers/dmc.py", line 149, in __getattr__
    return getattr(self._env, name)
AttributeError: 'Gym2DMC' object has no attribute 'render'

It seems that this error was triggered by missing __getattr_ in Gym2DMC wrapper. Is that right?

Question about Carla environment config

In appendix d.1.2,
"The Easy level includes soft_noisy_low_light and soft_high_light, while the Medium level is defined by the HardRainSunset and SoftRainSunset conditions. The Hard level contains hard_low_light and hard_noisy_low_light."

Q1. For the easy level, the weather is 'soft_noisy_low_light'. However, when setting the weather in carlaenv10_eval_config.yaml to "soft_noisy_low_light', the following error occurs.

I think I might need to change the weather: soft_noisy_low_light in cfgs/setting/setting.yaml to custom_soft_noisy_low_light. Is this correct? That is,

• easy
  • cfgs/setting.setting.yaml
    • soft_noisy_low_light -> custom_soft_noisy_low_light
    • soft_high_light -> custom_soft_high_light
• medium
  • HardRainSunset , SoftRainSunset (currently working well)
• hard
  • cfgs/setting.setting.yaml
    • hard_low_light-> custom_hard_low_light
    • hard_noisy_low_light -> custom_hard_noisy_low_light

Q2. There are two weathers each for Easy, Medium, and Hard levels. So, for something like the Score in Figure 4, do you evaluate each difficulty with num_eval_episode for both environments (e.g., HardRainSunset and SoftRainSunset for Medium) and then display the average score of the two weathers?

glfw.GLFWError when running locoeval.py or ./envs/DMCVGB/test/env_test.py

Hello, @gemcollector,
I'm trying to reproduce the locomotion results illustrated in Figure 18. The model is obtained by running train.py and the task is unitree stand. When I'm running the locoeval.py, the following problem is occurred:

Traceback (most recent call last):
File "locoeval.py", line 70, in evaluate
obs = env.reset()
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/suite/wrappers/action_scale.py", line 94, in reset
return self._env.reset()
File "/home/zhqin/RL_ViGen/envs/DMCVGB/dmcvgb/wrappers.py", line 90, in reset
return self.env.reset()
File "/home/zhqin/RL_ViGen/envs/DMCVGB/dmcvgb/wrappers.py", line 213, in reset
obs = self.env.reset()
File "/home/zhqin/RL_ViGen/envs/DMCVGB/dmcvgb/wrappers.py", line 329, in reset
return self._greenscreen(self.env.reset())
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/gym/wrappers/time_limit.py", line 26, in reset
return self.env.reset(**kwargs)
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/gym/wrappers/order_enforcing.py", line 18, in reset
return self.env.reset(**kwargs)
File "/home/zhqin/RL_ViGen/envs/DMCVGB/dmc2gym/dmc2gym/wrappers.py", line 199, in reset
obs = self._get_obs(time_step)
File "/home/zhqin/RL_ViGen/envs/DMCVGB/dmc2gym/dmc2gym/wrappers.py", line 140, in _get_obs
obs = self.render(
File "/home/zhqin/RL_ViGen/envs/DMCVGB/dmc2gym/dmc2gym/wrappers.py", line 215, in render
return self._env.physics.render(
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 216, in render
camera = Camera(
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 709, in init
if self._physics.contexts.mujoco is not None:
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 540, in contexts
self._make_rendering_contexts()
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 523, in _make_rendering_contexts
render_context = _render.Renderer(
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/_render/glfw_renderer.py", line 40, in init
super().init(max_width, max_height, executor.PassthroughRenderExecutor)
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/_render/base.py", line 63, in init
ctx.call(self._platform_init, max_width, max_height)
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/_render/executor/render_executor.py", line 138, in call
return func(*args, **kwargs)
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/dm_control/_render/glfw_renderer.py", line 51, in _platform_init
self._context = glfw.create_window(width=max_width, height=max_height,
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/glfw/init.py", line 1168, in create_window
return _glfw.glfwCreateWindow(width, height, _to_char_p(title),
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/glfw/init.py", line 628, in errcheck
_reraise(exc[1], exc[2])
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/glfw/init.py", line 52, in _reraise
raise exception.with_traceback(traceback)
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/glfw/init.py", line 607, in callback_wrapper
return func(*args, **kwargs)
File "/home/zhqin/anaconda3/envs/rl-vigen/lib/python3.8/site-packages/glfw/init.py", line 820, in _handle_glfw_errors
raise GLFWError(message)
glfw.GLFWError: (65543) b'GLX: Failed to create context: BadLength (poly request too large or internal Xlib length error)'

Have you ever encountered this problem? How can I solve it?

SVEA implementation

Is SVEA in this repo based on Drq-v2?
So, does that mean, Drq(v1) is the only SAC_based algorithm among the baselines?