lifelong-robot-learning/libero

Benchmarking Knowledge Transfer in Lifelong Robot Learning

libero's Issues

Image rendering issues

Hi,

Sometimes when I'm using LIBERO environments (doesn't matter which suite), the images I get from the environment are look like white noise partially through the rollout, or they flip upside down, or even sometimes the agentview_image returns the in hand image (and I'm positive the key is correctly passed). Most of the times, most images are normal.

Some examples attached:

rollouts_0_d233400ba867523b664b.mp4

rollouts_0_0630b38323cc0c55489a.mp4

This could possibly be a problem with my custom code somewhere, so just curious if you ever encountered something like this?

Flipped demonstration images

Hello, thank you for your excellent work!

I have a concern that might be incorrect, but it appears that the images in the demonstration dataset, both agentview_rgb and eye_in_hand_rgb, seem to be flipped upside down. I noticed this when comparing the first image from env.reset() with a second image from the demonstration data.

For reference, this is the code I used to visualize the demonstration data:

import h5py
import cv2
import numpy as np
import argparse

def main():
    # Open the H5 file
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--dataset",
        type=str,
        help="path to hdf5 dataset",
    )
    args = parser.parse_args()

    with h5py.File(args.dataset, "r") as f:

        agentview_images = f["data/demo_0/obs/agentview_rgb"]
     
        # Display each image in the sequence
        for img in agentview_images:
            # Make video upside down.
            # img = np.flipud(img)
            cv2.imshow("AgentView Image", cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
            cv2.waitKey(100)

        cv2.destroyAllWindows()

if __name__ == '__main__':
    main()

about the picture

Thanks for this wonderful benchmark.
I'm a bit confused, are these two parts written in reverse?

Does the init file provided in the code correspond to the Demonstraion in the dataset?

Thanks for this wonderful benchmark.

I want to replay the demonstration to record more information, however, I found the initial state of objects does not correspond to the demonstration.

The first video is the RGB demonstration provided by the dataset, and the second is my replay trajectory.

demonstration.mp4

replay.mp4

Here is my code to replay the trajectory:

from libero.libero import benchmark
from libero.libero.envs import OffScreenRenderEnv,DemoRenderEnv
import os
# import init_path
from libero.libero import benchmark, get_libero_path
from robosuite.wrappers import DataCollectionWrapper, VisualizationWrapper

import h5py

benchmark_dict = benchmark.get_benchmark_dict()
task_suite_name = "libero_object" # can also choose libero_spatial, libero_object, etc.
task_suite = benchmark_dict[task_suite_name]()

# retrieve a specific task
task_id = 2
task = task_suite.get_task(task_id)
task_name = task.name
print("the task name is:", task_name)
task_description = task.language
task_bddl_file = os.path.join(get_libero_path("bddl_files"), task.problem_folder, task.bddl_file)
print(f"[info] retrieving task {task_id} from suite {task_suite_name}, the " + \
      f"language instruction is {task_description}, and the bddl file is {task_bddl_file}")

# step over the environment
env_args = {
    "bddl_file_name": task_bddl_file,
    "camera_heights": 128,
    "camera_widths": 128,
    "has_renderer": True,
    "has_offscreen_renderer": False,
}
env = DemoRenderEnv(**env_args)

env = VisualizationWrapper(env.env)

env.seed(0)
env.reset()
init_states = task_suite.get_task_init_states(task_id) # for benchmarking purpose, we fix the a set of initial states
init_state_id = 40
print("init states is:", init_states.shape)
# env.set_init_state(init_states[init_state_id])
env.sim.set_state_from_flattened(init_states[init_state_id])
print("the task_bddl_file is", task_bddl_file)


dataset_path = "libero/datasets/libero_object/pick_up_the_salad_dressing_and_place_it_in_the_basket_demo.hdf5"

data = h5py.File(dataset_path)["data"]["demo_40"]["actions"]
print("data is:", data.shape)

for action in data:
    obs, reward, done, info = env.step(action)
    env.render()

env.close()

If it is due to the mismatch of the initial state, how can I get the corresponding initial state of the demonstrations so that I can replay the demonstration to get more information?

Questions regarding the benchmark details and the model checkpoints

Hi @Cranial-XIX @zhuyifengzju , thanks for the great work! Impressive!
I have a few questions:

1. The model checkpoint links on the website are empty. I would like to know when these checkpoints are available.
2. If I understand correctly, each spatial, goal, object dataset contains 10 tasks defined by 10 language instructions. Each task contains 50 fixed initial states, and each initial state has its corresponding demonstrations in the dataset, right?
3. Regarding the pretraining experiment in Figure 3, what does the success rate mean? Are the results averaged over all LIBERO_10 tasks after performing full fine-tuning on LIBERO_10? It is really unclear to me what are the settings for the results and the methods (particularly w/o pretraining and multitask) in Figure 3. Also, I am curious whether there are any intuition or insights why pretraining on LIBERO_90 do not work well.
4. Where could we find the results (succ. rate) of pretraining on LIBERO_90 and then testing on LIBERO_90?

Again, I am very interested in this work and feedback would be highly appreciated. Thanks in advance.

Question regarding SubprocVectorEnv failure

Hi, when I try to use the evaluation script on a headless machine (cloud server) with A10G GPU, I occasionally come across the following error:

Process Process-1:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/venv.py", line 222, in _worker
    env = env_fn_wrapper.data()
  File "peft/evaluate.py", line 35, in <lambda>
    [lambda: OffScreenRenderEnv(**env_args) for _ in range(env_num)])
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/env_wrapper.py", line 161, in __init__
    super().__init__(**kwargs)
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/env_wrapper.py", line 56, in __init__
    self.env = TASK_MAPPING[self.problem_name](
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/problems/libero_tabletop_manipulation.py", line 40, in __init__
    super().__init__(bddl_file_name, *args, **kwargs)
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/bddl_base_domain.py", line 135, in __init__
    super().__init__(
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/environments/manipulation/manipulation_env.py", line 162, in __init__
    super().__init__(
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/environments/robot_env.py", line 214, in __init__
    super().__init__(
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/environments/base.py", line 143, in __init__
    self._reset_internal()
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/bddl_base_domain.py", line 735, in _reset_internal
    super()._reset_internal()
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/environments/robot_env.py", line 510, in _reset_internal
    super()._reset_internal()
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/environments/base.py", line 296, in _reset_internal
    render_context = MjRenderContextOffscreen(self.sim, device_id=self.render_gpu_device_id)
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/utils/binding_utils.py", line 210, in __init__
    super().__init__(sim, offscreen=True, device_id=device_id, max_width=max_width, max_height=max_height)
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/utils/binding_utils.py", line 78, in __init__
    self.gl_ctx = GLContext(max_width=max_width, max_height=max_height, device_id=self.device_id)
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/renderers/context/egl_context.py", line 136, in __init__
    self._context = EGL.eglCreateContext(EGL_DISPLAY, config, EGL.EGL_NO_CONTEXT, None)
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 415, in __call__
    return self( *args, **named )
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/OpenGL/error.py", line 230, in glCheckError
    raise self._errorClass(
OpenGL.raw.EGL._errors.EGLError: EGLError(
        err = EGL_BAD_ALLOC,
        baseOperation = eglCreateContext,
        cArguments = (
                <OpenGL._opaque.EGLDisplay_pointer object at 0x7eff68f41640>,
                <OpenGL._opaque.EGLConfig_pointer object at 0x7eff68f41540>,
                <OpenGL._opaque.EGLContext_pointer object at 0x7eff8a264b40>,
                None,
        ),
        result = <OpenGL._opaque.EGLContext_pointer object at 0x7eff68f41a40>
)
Exception ignored in: <function EGLGLContext.__del__ at 0x7eff8a1461f0>
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/renderers/context/egl_context.py", line 155, in __del__
    self.free()
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/renderers/context/egl_context.py", line 146, in free
    if self._context:
AttributeError: 'EGLGLContext' object has no attribute '_context'
Exception ignored in: <function MjRenderContext.__del__ at 0x7eff8a1463a0>
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/site-packages/robosuite/utils/binding_utils.py", line 198, in __del__
    self.con.free()
AttributeError: 'MjRenderContextOffscreen' object has no attribute 'con'
Traceback (most recent call last):
  File "lifelong/evaluate.py", line 239, in main
    env.reset()
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/venv.py", line 702, in reset
    ret_list = [self.workers[i].recv() for i in id]
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/venv.py", line 702, in <listcomp>
    ret_list = [self.workers[i].recv() for i in id]
  File "/home/ubuntu/1_repo/LIBERO/libero/libero/envs/venv.py", line 428, in recv
    result = self.parent_remote.recv()
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/home/ubuntu/anaconda3/envs/libero/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

Sometimes I came across this issue due to insufficient CUDA memory; however, now even with enough memory, I still encounter this problem and have no idea how to solve it.
I can use the evaluation script with DummyVectorEnv, but it seems to be too slow.
So I am wondering whether you have encountered similar issues? Any hints would be appreciated. Thanks in advance.

Issues with Playback

I'm having a hard time reproducing (by playback) trajectories in the dataset (I'm interested in generating / logging some more details) -- they mostly look correct, but in some cases, the replayed trajectory "fails" while the original logged trajectory succeeds

I've been trying to follow scripts/create_dataset.py to replicate exactly how the files should be replayed, but I've been finding that almost always, the new states are at least 0.01 apart from those logged, and for a nontrivial number of states, it goes up >=1

I'm wondering

1. if there could be some dependency on a specific Mujoco version that was used to collect the data or
2. there could be the need for an off-by-one or something like that when replaying?
3. if there's some burn-in no-actions that need to be taken at the beginning of an episode (to warm up the controller)?

    states = f["data/{}/states".format(ep)][()]
    actions = np.array(f["data/{}/actions".format(ep)][()])
    init_idx = 0

    env.reset_from_xml_string(model_xml)
    env.sim.reset()
    env.sim.set_state_from_flattened(states[init_idx])
    env.sim.forward()
    all_obs = []
    all_actions = []
    n_errors = 0
    for j, action in enumerate(data["data"][ep]["actions"]):
        obs, reward, done, info = env.step(action)
        if j < len(actions) - 1:
            # ensure that the actions deterministically lead to the same recorded states
            state_playback = env.sim.get_state().flatten()
            # assert(np.all(np.equal(states[j + 1], state_playback)))
            err = np.linalg.norm(states[j + 1] - state_playback)

[warning] playback diverged by 0.24 for ep demo_4 at step 0
[warning] playback diverged by 1.29 for ep demo_4 at step 45
[warning] playback diverged by 0.22 for ep demo_4 at step 46
[warning] playback diverged by 1.02 for ep demo_4 at step 47
[warning] playback diverged by 2.04 for ep demo_4 at step 48