Error when using isaacgym

[dexsuite]

Error when using isaacgym

[yzqin]

Thank you for reporting this issue. I've been able to reproduce the bugs on my machine following @y8han. The problem appears to stem from isaacgym overriding the built-in pybind11 registrar of C++ std::vector<std::string>. This affects all pybind11-based Python libraries that explicitly return a std::vectorstd::string, .e.g pinocchio.

A temporary workaround involves adjusting the import order to ensure Pinocchio loads before isaacgym. Additionally, since isaacgym needs to be loaded before torch, the recommended import order is:

pinocchio -> isaacgym -> torch

Here's an example of how to structure your imports:

# Other unrelated libraries not depending on pybind11
import other_unrelated_libs

import pinocchio
import isaacgym
import torch
import dex_retargeting

# More unrelated libraries
import more_unrelated_libs

To fix the error in detect_from_video.py mentioned by @y8han, you can modify the import section as follows:

import pickle
from pathlib import Path

import cv2
import tqdm
import tyro

import pinocchio
import isaacgym
import torch
from dex_retargeting.constants import RobotName, RetargetingType, HandType, get_default_config_path
from dex_retargeting.retargeting_config import RetargetingConfig
from dex_retargeting.seq_retarget import SeqRetargeting
from single_hand_detector import SingleHandDetector

This solution should resolve the immediate issue. However, if anyone discovers a more elegant solution, please feel free to discuss it in this issue thread.

[yzqin]

Could you please provide a detailed script to reproduce this error? I am using isaacgym myself and I do not encounter this error. Here is the screenshot of my terminal command.

Python 3.8.19 (default, Mar 20 2024, 19:58:24) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dex_retargeting
>>> import isaacgym
Importing module 'gym_38' (/home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json

Or with a different import order:

Python 3.8.19 (default, Mar 20 2024, 19:58:24) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import isaacgym
Importing module 'gym_38' (/home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/yuzhe/miniconda3/envs/debug/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
>>> import dex_retargeting

Here is my version of pin and isaacgym

pip3 list | grep isaacgym
isaacgym                 1.0rc4

pip3 list | grep pin
pin                      2.7.0

[y8han]

Hi, I encountered the same issue here, and I feel this could be some conflicts between isaacgym simulator and dex-retargeting codebase. For verification, I just added one line import isaacgym to the python script detect_from_video.py and run the demo command

python3 detect_from_video.py \
  --robot-name allegro \
  --video-path data/human_hand_video.mp4 \
  --retargeting-type dexpilot \
  --hand-type right \
  --output-path data/allegro_joints.pkl 

Then, I got the same error message
TypeError: No Python class registered for C++ class std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >
Without importing the isaacgym, this script works fine as expected.

[y8han]

Thank you for reporting this issue. I've been able to reproduce the bugs on my machine following @y8han. The problem appears to stem from isaacgym overriding the built-in pybind11 registrar of C++ std::vector<std::string>. This affects all pybind11-based Python libraries that explicitly return a std::vectorstd::string, .e.g pinocchio.

A temporary workaround involves adjusting the import order to ensure Pinocchio loads before isaacgym. Additionally, since isaacgym needs to be loaded before torch, the recommended import order is:

pinocchio -> isaacgym -> torch

Here's an example of how to structure your imports:

# Other unrelated libraries not depending on pybind11
import other_unrelated_libs

import pinocchio
import isaacgym
import torch
import dex_retargeting

# More unrelated libraries
import more_unrelated_libs

To fix the error in detect_from_video.py mentioned by @y8han, you can modify the import section as follows:

import pickle
from pathlib import Path

import cv2
import tqdm
import tyro

import pinocchio
import isaacgym
import torch
from dex_retargeting.constants import RobotName, RetargetingType, HandType, get_default_config_path
from dex_retargeting.retargeting_config import RetargetingConfig
from dex_retargeting.seq_retarget import SeqRetargeting
from single_hand_detector import SingleHandDetector

This solution should resolve the immediate issue. However, if anyone discovers a more elegant solution, please feel free to discuss it in this issue thread.

This works perfectly on my local machine! Thank you for your quick reply!