dexsuite/dex-retargeting

DexSuite

DexSuite is a comprehensive package that includes related resources for dexterous robotic hands. It encompasses models of dexterous hands, compatible object models for these hands, and resources for retargeting and teleoperation specifically designed for multi-fingered dexterous hands.

dex-retargeting's Issues

Expand the retargeting algorithm to 2 arm-hand system

Hi, thank you for your astonishing work! Right now I have a self-designed humanoid robot with two arms and I want apply your retargeting algorithm to the whole upper body of my robot. So I want to teleop my robot, just the two arms on its upper body, in a sim environment just using the dex-retargeting algorithm. But I am not sure how to make the retargeting algorithm work on the humanoid robot arms such that it could mimic the human operator's action.

error during visualizing robot hand-object Interaction

I have followed the instructions and successfully visualized human hand interaction. However, when using this command to visualize robot hand interaction, the following error occurred. Is there any way to fix this problem? Thank you!

command:
python3 visualize_hand_object.py --dexycb-dir=./dexycb --robots allegro shadow svh
error:

Failed loading a new robot hand model into RetargetingConfig.

Hi @yzqin

Thank you so much for your amazing work.
While loading a new robot hand model(similar to Allegro Hand) using this code:

retargeting = RetargetingConfig.load_from_file('/home/ryanjiao/RoboticManipulator/dex-retargeting/dex_retargeting/configs/teleop/leap_hand_right.yml').build()

I encountered an error as follow:

/home/ryanjiao/anaconda3/envs/retgt/lib/python3.8/site-packages/numpy/lib/twodim_base.py:295: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  v = asanyarray(v)
Traceback (most recent call last):
  File "/home/ryanjiao/RoboticManipulator/dex-retargeting/example/vector_retargeting/real_time_mujoco.py", line 106, in <module>
    main()
  File "/home/ryanjiao/RoboticManipulator/dex-retargeting/example/vector_retargeting/real_time_mujoco.py", line 37, in main
    retargeting = RetargetingConfig.load_from_file('/home/ryanjiao/RoboticManipulator/dex-retargeting/dex_retargeting/configs/teleop/leap_hand_right.yml').build()
  File "/home/ryanjiao/RoboticManipulator/dex-retargeting/dex_retargeting/retargeting_config.py", line 137, in build
    sapien_model = SAPIENKinematicsModelStandalone(
  File "/home/ryanjiao/RoboticManipulator/dex-retargeting/dex_retargeting/optimizer_utils.py", line 109, in __init__
    builder = loader.load_file_as_articulation_builder(urdf_path)
IndexError: stof

I created an yaml file for this robot hand model and the content of the yaml file is almost the same as the yaml file for Allegro Hand.
And my URDF work fine in Mujoco environment. But the meshes in my URDF is using .stl instead of .mtl+.obj. I wonder whether this would cause the error?

Plan to add dummy joints under Pin Model?

Hi Yuzhe @yzqin

Thanks for your great work! I noticed that you tried to remove requirements on sapien in your last few commits and replaced them with pinocchio models. It's really thoughtful. 👍

However, the support of dummy joints may have also been removed, which may be necessary for position-based retargeting from my perspective. Do I understand this correctly? If so, will dummy joints be added under the current framework in the future?

Thanks again. 🙏

Error when using isaacgym

Hi, thank you for your awesome work!
When we tried to use the project for retargeting in the issacgym environment, we encountered an error:
TypeError: No Python class registered for C++ class std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >
The error seemed to be caused by a conflict between isaacgym and pinocchio, where if we read the model from urdf and run print(model.names)
after
import isaacgym
the above error occurs.

Would you please give us some ideas about how to deal with it? Our isaacgym version is 1.0rc4.

methods clarification

Hi @yzqin

Exciting work! Can you please clarify which methods are used for hand pose estimation and retargeting? I think they are from https://github.com/yzqin/dex-hand-teleop/tree/main, just to double-check.

How To do position retarget like AnyTeleop

Hi
it's a greate work. I sam hand detection retarget logic, how about wrist pose retarget using RBG-D camera ?
as it mentions in AnyTeleop paper, can you give some code examples?

how to convert wrist frame to camera frame ?

Thanks

teleop on physic simulator to collect data

Thank you for your amazing work. I am recently use your code to collect some demonstration data for my project. I can see from your prior work , that you can control the hand on an predefined environment and I also expect that you can do this with this project. Do you have any plan to provide another example file for this task or can you give me a brief instruction of how to do this task?
Thank you

I'm having the following issue running your code.

Thanks for your useful work!
I'm having the following issue running your code:
python detect_from_video.py --robot-name allegro --video-path data/human_hand_video.mp4 --retargeting-type vector --hand-type right --output-path data/allegro_joints.pkl
Traceback (most recent call last):
File "detect_from_video.py", line 9, in
from dex_retargeting.retargeting_config import RetargetingConfig
File "/public/home/wushj12023/dex-retargeting-main/dex_retargeting/retargeting_config.py", line 1, in
import sapien.core as sapien
File "/public/home/wushj12023/.conda/envs/retarget/lib/python3.8/site-packages/sapien/init.py", line 9, in
from sapien import core, sensor, asset, example, utils
File "/public/home/wushj12023/.conda/envs/retarget/lib/python3.8/site-packages/sapien/core/init.py", line 3, in
from .pysapien import *
ImportError: libvulkan.so.1: cannot open shared object file: No such file or directory
I run your code on the GPU server, so I can't execute the sudo command. How do I fix this?

hand root position

Hi @yzqin

For the current code, does it produce the 3D position estimation of the hand? I am not sure if my understanding is correct, the media-pipe produces the 3D joint positions in hand's joint 0 frame? What about the wrist rotation matric, is it in camera frame?

How To do position retarget in real world?

Hi:
it's a awesome work. I have a leap hand in my Lab. I want to use my origin laptop's camera to capture my hand's finger pose, then use dex-retargeting to retarget my finger pose to my leap hand. However my os is Win11. Can I use it? How can I do it? Maybe my question is so easy, but I really need your help. Thank you so much : )
(I am now able to achieve joint control of dexterous hands, but I dont know how to deploy your codes) :)

hand detection retarget not accurate

hello

I have tested the hand detection retarget using allegro hand. the test code is pr

I found some position is not accurate, below are some test snapshots

tested env:

• camera: built-in camera on the computer is used.
• pc board name rog7
• system: windows 11

Using Isaac sim to reproduce your code.

Hi,yuzhe!
I am using Isaac Sim to reproduce your code.But I encountered a problem during the reproduction process.I really hope you can give me some help to solve this problem. The problem mainly arises in the calculation of Jacobian matrices for specific links. The code is as follows:

            jacobians = []

            jacobians_all = self.robot.articulation_view.get_jacobians()[0]     
     
            for i , index in enumerate(self.target_link_indices):
                link_jacobian = jacobians_all[index][:3, self.target_joint_indices]
                jacobians.append(link_jacobian)
            jacobians = np.stack(jacobians, axis=0)

            huber_distance.backward()
            grad_pos = torch_body_pos.grad.cpu().numpy()[:, None, :]
            grad_qpos = np.matmul(grad_pos, np.array(jacobians))
            grad_qpos = grad_qpos.mean(1).sum(0)
            grad_qpos += 2 * self.norm_delta * (x - last_qpos)
            grad[:] = grad_qpos[:]

I use the API 'articulation_view.get_jacobians()' and the shape of returned result is (33,6,30). I tried slicing it to obtain the Jacobian matrix for a specific link. But I found that the result calculated by your code is inconsistent with the result returned. Given the position of the joint, how do I calculate the Jacobian matrix for a specific body in Isaac Sim? I really hope you can give me some help to solve this problem.

The vector retargeting readme is incomplete

Thank you for your excellent and useful work.
I have noticed that vector retargeting readme is incomplete and I attempted to use vector retargeting by following the method of position retargeting, but observed a large position error in the dexterous hand.
I look forward to your plan to supplement it.

AttributeError

--robot-name allegro \

--video-path data/human_hand_video.mp4
--retargeting-type dexpilot
--hand-type right
--output-path data/allegro_joints.pkl
Traceback (most recent call last):
File "/home/badboy/code/dex-retargeting/example/vector_retargeting/detect_from_video.py", line 87, in
tyro.cli(main)
File "/home/badboy/code/dex-retargeting/example/vector_retargeting/myenv/lib/python3.10/site-packages/tyro/_cli.py", line 217, in cli
return run_with_args_from_cli()
File "/home/badboy/code/dex-retargeting/example/vector_retargeting/detect_from_video.py", line 82, in main
retargeting = RetargetingConfig.load_from_file(config_path).build()
File "/home/badboy/code/dex-retargeting/dex_retargeting/retargeting_config.py", line 161, in build
robot = RobotWrapper(temp_path)
File "/home/badboy/code/dex-retargeting/dex_retargeting/robot_wrapper.py", line 15, in init
self.model: pin.Model = pin.buildModelFromUrdf(urdf_path)
AttributeError: module 'pinocchio' has no attribute 'buildModelFromUrdf'

Import the new manipulator's URDF

Hi @yzqin ，
Your work has left a deep impression on me, but I am currently encountering issues while using the code. I am attempting to import a humanoid three-finger dexterous hand, consisting of a thumb, index finger, and middle finger. I have imported the URDF, but it seems that the redirection process has encountered some problems. I would like to inquire if there are any specific considerations to keep in mind when importing URDF files. I look forward to your response. Thank you!

When my humanoid hand is fully extended, the robotic hand does not synchronize with mine; instead, it bends to 90 degrees.

How to relax restriction on collision between links?

Hi, I was trying the leap hand urdf in the dex-urdf repository. And I manually add an fingertip link like this：

  <joint name="RF_fixtip" type="fixed">
      <parent link="fingertip_3" />
      <child link="RF_touchtip" />
      <origin rpy="0.0 -1.57079633 0.0" xyz="-0.01 -0.04 0.01 " />
  </joint>

And this is the yaml I used:

retargeting:
  type: vector
  urdf_path: yzqin_leap_hand/leap_hand_right.urdf
  wrist_link_name: "base"

  # Target refers to the retargeting target, which is the robot hand
  target_joint_names: null
  target_origin_link_names: [ "base", "base", "base", "base" ]
  target_task_link_names: [ "TH_touchtip", "FF_touchtip", "MF_touchtip", "RF_touchtip" ]
  scaling_factor: 1.4

  # Source refers to the retargeting input, which usually corresponds to the human hand
  # The joint indices of human hand joint which corresponds to each link in the target_link_names
  target_link_human_indices: [ [ 0, 0, 0, 0 ], [ 4, 8, 12, 16 ] ]

  # A smaller alpha means stronger filtering, i.e. more smooth but also larger latency
  low_pass_alpha: 0.2

The pose seems to be alright, but when I try to grasp something, the gap between fingertips looks too huge. I add a video to show the problem. And I' like to ask how to relax restriction on collision between links?
https://github.com/dexsuite/dex-retargeting/assets/56286149/7441fd18-96b5-4d9e-a037-2e25b9d4e0cc

question about mano conver

Hi

Thanks for you greate work. I have some question about the vector retarget code.
in single handle detector L 87, the detect results are mapped to MANO convention, what is the purpose of the convert ? can you give me more details ?

Thanks

Faild to load Leap Hand model

I am trying to use the leap hand model, the model I use is from https://github.com/dexsuite/dex-urdf. However, it stills failed to load.

Is there anything I should notice for using new robot model?

Issue running "pip3 install -e ."

Hello! I am trying to use this repo for a project, but I am running into build issues.
I have attached a file with the terminal error output below. I am on a Mac running Sonoma 14.1.2, using a pyenv virtualenv environment running Python 3.8.19. Is there a specific Python version I should be running? It's saying there is an issue with nlopt.

error.txt

Retargeting of example allegro inconsistent with video in project website?

Hi, I was trying to run the exmple of retarget allegro in the example folder. However, it seems the retarget result is different for some hand pose, the following is from website and from local runs, do I need to modify any other parts?

retargeted_allegro.mp4