Comments (6)
Hi @xbkaishui
I'm not quite sure I understand your question. Could you clarify what you mean by "wrist pose retarget" in the context of AnyTeleop?
Also, I'm not sure why I would need to convert the wrist frame to the camera frame for teleoperation. Could you provide some more details or context about your question?
from dex-retargeting.
Hi @yzqin
Thanks for your quick response, I want to use Wrist Pose Detection result to teleoperation the robot arms
Below is reference from original paper Wrist Pose Detection from RGB-D.
We use the pixel positions of the detected keypoints to retrieve the corresponding
depth values from the depth image. Then, utilizing known intrinsic camera parameters, we compute the 3D positions
of the keypoints in the camera frame. The alignment of the RGB and depth images is handled by the camera
driver. With the 3D keypoint positions in both the local wrist frame and global camera frame, we can estimate the wrist
pose using the Perspective-n-Point (PnP) algorithm.
Here, the 3D camera captures depth information. The wrist position can be obtained from keypoints within the wrist coordinate system, and the wrist position can also be observed in the camera coordinate system. Should we create a transformation matrix between the wrist coordinate system and the camera coordinate system here? The wrist pose from the camera's perspective should be this transformation matrix, right?
from dex-retargeting.
Hi @xbkaishui
Apologies for the delayed response; I was traveling.
To clarify, we use the FrankMocap model for wrist pose estimation in this project. However, due to licensing restrictions, we cannot directly release that part of the code within AnyTeleop but you can download FrankMocap yourself for free.
If you're interested in exploring wrist pose detection from RGB-D data, you can find our previous implementation here:
Let me know if you have any other questions!
from dex-retargeting.
Hi Yuzhe
I still don't fully understand the entire data collection process. I'm not clear on how to control the robotic arm using my hand. How are the hand coordinates captured by the 3D camera mapped to the robotic arm? can you explain more ?
Thanks
from dex-retargeting.
Hi @xbkaishui
The arm motion control is quiet more complicated than hand retargeting. I am working on some paper submission deadlines for now and I will try to wrap up better documentation about arm later. It requires much more dependencies on the software side and more effort on the tutorial.
from dex-retargeting.
ok, got it
from dex-retargeting.
Related Issues (20)
- hand detection retarget not accurate HOT 6
- Issue running "pip3 install -e ." HOT 1
- Import the new manipulator's URDF HOT 4
- How To do position retarget in real world? HOT 19
- Expand the retargeting algorithm to 2 arm-hand system HOT 1
- teleop on physic simulator to collect data HOT 1
- Error when using isaacgym HOT 4
- AttributeError
- Curobo implementation
- Difference bettwenn VectorOptimizer and DexPilotOptimizer HOT 7
- retargetting problem with laptop's built-in camera HOT 7
- some mistake when I install dex-retargeting HOT 1
- Basic Mathmatics about calculating the joint_pos variable in SingleHandDetector.py HOT 2
- install error HOT 3
- How can I retargeting with my hand keypoints as input? HOT 1
- Deployment on physical inspire dexterous hand HOT 1
- Recommended Python Version HOT 2
- position retargeting demo fails when only `*_glb.urdf` is available HOT 3
- using `warm_start` with position optimizer HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dex-retargeting.