About the comments about coordinate systems in tv_wrapper.py

[silencht]

This issue is created to address the branching problem raised in issue #19.
The problem is described as follows:

Hi I am reading the comments about coordinate systems in tv_wrapper. I am a little bit confused by the wrist coordinates as stated in the comments:

 (Left Wrist) XR/AppleVisionPro Convention:
    - the x-axis pointing from wrist toward middle.
    - the y-axis pointing from index toward pinky.
    - the z-axis pointing from palm toward back of the hand.

 (Right Wrist) XR/AppleVisionPro Convention:
    - the x-axis pointing from wrist toward middle.
    - the y-axis pointing from pinky toward index.
    - the z-axis pointing from palm toward back of the hand.
    
 p.s. From website: https://registry.khronos.org/OpenXR/specs/1.1/man/html/openxr.html.
     You can find **(Left/Right Wrist) XR/AppleVisionPro Convention** related information like this below:
       "The wrist joint is located at the pivot point of the wrist, which is location invariant when twisting the hand without moving the forearm. 
        The backward (+Z) direction is parallel to the line from wrist joint to middle finger metacarpal joint, and points away from the finger tips. 
        The up (+Y) direction points out towards back of the hand and perpendicular to the skin at wrist. 
        The X direction is perpendicular to the Y and Z directions and follows the right hand rule."
     Note: The above context is of course under **(basis) OpenXR Convention**.

The coordinates of both hands do not align with the website statement. Can you explain what is the difference?

[silencht]

@BBBBBBob

For this issue, a reference image is provided for your convenience. If you have any further questions, please feel free to add them under this issue. Thank you!

Taking the left hand as an example:

• Red

under (basis) Robot Convention, wrist / hand's initial pose convention:
(Left Wrist) XR/AppleVisionPro Convention:
- the x-axis pointing from wrist toward middle.
- the y-axis pointing from index toward pinky.
- the z-axis pointing from palm toward back of the hand.

• Blue

under (basis) OpenXR Convention
The backward (+Z) direction is parallel to the line from wrist joint to middle finger metacarpal joint, and points away from the finger tips.
The up (+Y) direction points out towards back of the hand and perpendicular to the skin at wrist.
The X direction is perpendicular to the Y and Z directions and follows the right hand rule.

[BBBBBBob]

@silencht Thank you for the answer! I am still confused about the Unitree convention of hand. Assuming that the coordinate of robot convention is fixed, is the initial position of Unitree hand different from the VR position or is the coordinate system different? Because the coordinate of our hands varies from the example, I want to be clear how T_to_unitree_hand is calculated.

[silencht]

@BBBBBBob
In short, the role of tv_wrapper is to transform the OpenXR convention into the Robot convention commonly used in robotics.

As for some of the other content, using the left hand as an example:
The purpose of T_to_unitree_left_wrist is simply to rotate the initial coordinate system by 90°. This is because, whether in the OpenXR or Robot convention, the initial orientation (represented by the identity matrix) assumes that the palm faces downward, whereas the palm of the Unitree left hand faces to the right.

As for the T_to_unitree_hand you mentioned, it is related to the usage of the dex_retargeting library.
I recommend that you refer to dexsuite/dex-retargeting#13 for further details.

[BBBBBBob]

@silencht Thank you for pointing out! T_to_unitree_hand should be a transformation matrix mapping the OpenXR convention to the hand convention. Is it correct?

[BBBBBBob]

Under the robot convention, does the unitree convention look like the following pose:

so that

(Left Hand URDF) Unitree Convention:
- The x-axis pointing from palm toward back of the hand.
- The y-axis pointing from middle toward wrist.
- The z-axis pointing from pinky toward index.

is satisfied.

[silencht]

Yes. What you drew is correct.
However, essentially, I made some modifications to the URDF of the UFACTORY dexterous hand to make its coordinate system consistent with the previous Inspire dexterous hand URDF.
You can see that I applied a rotation at this position in the URDF file(The same applies to the right-hand URDF):

avp_teleoperate/assets/unitree_hand/unitree_dex3_left.urdf

Lines 257 to 261 in 0b598c1

	<joint name="base" type="fixed">
	<parent link="base_link" />
	<child link="left_hand_palm_link" />
	<origin rpy="0 0 -1.57" xyz="0 0 0" />
	</joint>

Perhaps I can use a diagram to explain this:

To be honest, explaining these coordinate transformations is indeed quite troublesome. I hope that in the future, I can explain all these changes in the form of documentation. However, this will take time, and the timeline is uncertain.

If you have any further questions, feel free to ask here.

[BBBBBBob]

Indeed, I have already solved the transformation based on our hand's urdf. Take your image for example:

The green coordinate is the unitree hand convention as described above. I first adjusted the hand configuration as shown in the image and then calculated the rotation matrix between the red and green coordinates, which also works for our urdf coordinate. I hope this way is easier to understand.