question on adding a new robot

[luoye2333]

Hi author Luo! Thanks for your great works.
I am working on training phc on a new robot structure. I have added urdf, xml, meshes in phc/data/assets and i am working on adding config files in phc/cfg/robot. Is there a full workflow to add a new robot? Much appreciated if you could write some in the readme file.

Btw, is A100 necessary for training? How much gpu memory does it takes? Is it possible to train on a single 4090?

[luoye2333]

Finished creating a new .yaml file, but i dont know the meaning of "extend config". Does it mean config the fixed links (e.g. the head and hand links) in "extend config"?

I am using the dance_sample_g1.pkl and getting error:

phc/env/tasks/humanoid_amp.py", line 625, in _set_env_state
    self._rigid_body_pos[env_ids] = rigid_body_pos
RuntimeError: shape mismatch: value tensor of shape [3072, 26, 3] cannot be broadcast to indexing result of shape [3072, 29, 3]

seems i need to create a new reference data to make body numbers and indexes correct, or the problem is in the .yaml file?

[luoye2333]

Found the method to retarget the dataset (Sorry didnt see other issues first): #77 (comment)

[IlanElfen]

Hi I'm also confused on how you retarget a different robot. I'm using human2humanoid, which uses PCH to retarget to a different robot. I'll admit I'm a little lost. Could you please walk me through the files you added, changed, etc. Thank you very much!

[luoye2333]

Hi I'm also confused on how you retarget a different robot. I'm using human2humanoid, which uses PCH to retarget to a different robot. I'll admit I'm a little lost. Could you please walk me through the files you added, changed, etc. Thank you very much!

1. add .urdf, .xml and /meshes to phc/data/assets/robot
2. add a yaml file like unitree_g1.yaml in phc/data/cfg/robot
3. add a env config in phc/data/cfg/env
4. add the robot names in whatever u find h1 and g1 (can be done by searching through all files (vscode ide can do this) in the workspace), such as humanoid.py and humanoid_amp.py. Pid params, torque limits, default pos is added in humanoid.py.
5. retarget the dataset using the method in Questions Regarding H1 and G1 Support #77 (comment). Btw we should fit the shape first, and then fit the motion.
6. I dont know (currently working on 5)

[IlanElfen]

Thanks for your response! We'll see who can get to step 6 first :)

[ZhengyiLuo]

The command for fitting AMASS motion to a new robot would be:

python scripts/data_process/fit_smpl_motion.py robot=[unitree_g1_fitting, unitree_h1_fitting, etc] +fit_all=True
python scripts/data_process/fit_smpl_shape.py robot=[unitree_g1_fitting, unitree_h1_fitting, etc] +fit_all=True
python scripts/vis/vis_q_mj.py robot=[unitree_g1_fitting, unitree_h1_fitting, etc] +motion_name=[name_1, name_2, etc]

which should be run in that order.

Yes, each robot needs its own config yaml file and specific motion file. I will try to write more detailed documentation on this.

[ZhengyiLuo]

Extend_config is the configuration used for extending the robot arms/legs in order to add an end-effector to it. If your robot has hands/toes you do not need it. Untree H1 does not have a hand/head joint so I added them to match SMPL's rough kinematic structure.

[dbdxnuliba]

@luoye2333 @ZhengyiLuo ，could you please show the full command to evalute g1/h1 after trained,many thanks

[dbdxnuliba]

@ZhengyiLuo @luoye2333 I meet err when run python scripts/data_process/fit_smpl_motion.py robot=unitree_g1_fitting +fit_all=True
could you please have a look
(isaac) rob@rob:~/rl/PHC$ python scripts/data_process/fit_smpl_motion.py robot=unitree_g1_fitting +fit_all=True
2024-11-10 13:27:46,325 - INFO - logger - logger initialized
2024-11-10 13:27:47,186 - DEBUG - Setting JobRuntime:name=UNKNOWN_NAME
2024-11-10 13:27:47,186 - DEBUG - Setting JobRuntime:name=fit_smpl_motion
Error executing job with overrides: ['robot=unitree_g1_fitting', '+fit_all=True']
Traceback (most recent call last):
File "scripts/data_process/fit_smpl_motion.py", line 200, in main
jobs= [jobs[i:i + chunk] for i in range(0, len(jobs), chunk)]
ValueError: range() arg 3 must not be zero

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

The command for fitting AMASS motion to a new robot would be:

python scripts/data_process/fit_smpl_motion.py robot=[unitree_g1_fitting, unitree_h1_fitting, etc] +fit_all=True
python scripts/data_process/fit_smpl_shape.py robot=[unitree_g1_fitting, unitree_h1_fitting, etc] +fit_all=True
python scripts/vis/vis_q_mj.py robot=[unitree_g1_fitting, unitree_h1_fitting, etc] +motion_name=[name_1, name_2, etc]

which should be run in that order.

Yes, each robot needs its own config yaml file and specific motion file. I will try to write more detailed documentation on this.

[luoye2333]

@dbdxnuliba
for g1 and h1, create new dataset is not needed. use g1_dance.pkl downloaded by download_data.sh is ok.
If you want to fit a new, you should first fit smpl betas and scale parameter

python scripts/data_process/fit_smpl_shape.py robot=unitree_g1_fitting +fit_all=True +vis=True

Then fit the motion on the fitted smpl parameter. This will take several hours. So i recommend use a small dataset to test first.

python scripts/data_process/fit_smpl_motion.py robot=unitree_g1_fitting +fit_all=True

To visualize the fitted motion file

python scripts/vis/vis_q_mj.py robot=unitree_g1_fitting +motion_name=amass_all

[IlanElfen]

@dbdxnuliba Look at this issue

[dbdxnuliba]

If you want to fit a new, you should first fit smpl betas and scale parameter

thanks，I'll have a try

[dbdxnuliba]

@dbdxnuliba Look at this issue

ok,thank you very much

[dbdxnuliba]

@luoye2333，Hello，sorry to trouble you
I executed the following on November 11, and by November 14, it still wasn't completed. I only put a small part of the dataset in. Why is this phenomenon occurring?

(isaac) rob@rob:~/rl/PHC$ python scripts/data_process/fit_smpl_motion.py robot=unitree_g1_fitting +fit_all=True
2024-11-11 23:06:33,853 - INFO - logger - logger initialized
2024-11-11 23:06:34,753 - DEBUG - Setting JobRuntime:name=UNKNOWN_NAME
2024-11-11 23:06:34,753 - DEBUG - Setting JobRuntime:name=fit_smpl_motion
key_names: ['0-ACCAD_Female1General_c3d_A2 - Sway_poses', '0-ACCAD_Female1General_c3d_A2 - Sway t2_poses', '0-ACCAD_Female1General_c3d_A1 - Stand_poses']
enter process_motion
enter process_motion
enter process_motion
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 43/43 [00:00<00:00, 137.04it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 84/84 [00:00<00:00, 112347.43it/s]
load forward kinematics model
load smpl model
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 43/43 [00:00<00:00, 136.06it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 84/84 [00:00<00:00, 115628.99it/s]
load forward kinematics model
load smpl model
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 43/43 [00:00<00:00, 114.48it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 84/84 [00:00<00:00, 117558.07it/s]
load forward kinematics model
load smpl model

From the added print information, it seems that it is known to be stuck at this statement

here is the dataset I put in only ACCAD folder， and only 3 npz files in it， 

rob@rob:~/rl/PHC/amass_data/Amass_part$ tree 
.
└── ACCAD
    └── Female1General_c3d
        ├── A1 - Stand_poses.npz
        ├── A2 - Sway_poses.npz
        └── A2 - Sway t2_poses.npz

2 directories, 3 files

here is the amas_root path
amass_root = "/home/rob/rl/PHC/amass_data/Amass_part"