smpl_params_global and camera extrinsic parameters

[tucker666]

hi, thank you for your great job.
i have two questions
(1) smpl body can be projected into the camera view correctly by using "pred_smpl_params_incam"；but i can not project smpl body into the camera view correctly by using "smpl_params_global" and camera extrinsic parameters. is there a way to do that ?
(2) i can get two people's "smpl_params_global" by change "track_id" using same video, but how can i make the two person having the correct relative position to each other in Blender.
looking forward for your reply.

[tucker666]

您好，想问下：
（1）使用smpl_params_global和相机外参，在相机视角下，能否把人投影回视频的正确位置呢？（2）通过更改track_id可以得到同一个静态相机视频里两个人的"smpl_params_global"参数，两个人的相对关系可以得到正确得到么，在blender里两个人初始帧是叠在一块的

[zehongs]

Hi @tucker666, sorry for the late reply! Thanks for these two important questions.

1. An extra optimization is required to meet your needs. This is because we don't explicitly model the transformation between these two predictions.
2. The relative relationship is also unconstrained, so an extra optimization is needed.

[JingyunLiang]

@tucker666 You can use the following code to find the right RT

import torch

def find_camera_extrinsics(verts_world, verts_camera):
    """
    Find camera extrinsics (R, T) given vertices in world and camera coordinate systems.
    
    Args:
        verts_world: (N, 3) Vertices in world coordinates.
        verts_camera: (N, 3) Vertices in camera coordinates.

    Returns:
        R: (3, 3) Rotation matrix.
        T: (3,) Translation vector.
    """
    # Compute centroids
    centroid_world = verts_world.mean(dim=0)
    centroid_camera = verts_camera.mean(dim=0)
    
    # Center the points
    centered_verts_world = verts_world - centroid_world
    centered_verts_camera = verts_camera - centroid_camera
    
    # Compute the cross-covariance matrix
    H = centered_verts_camera.T @ centered_verts_world
    
    # Perform SVD
    U, _, Vt = torch.linalg.svd(H)
    
    # Compute rotation matrix
    R = U @ Vt
    # Ensure R is a proper rotation matrix
    if torch.det(R) < 0:
        U[:, -1] *= -1  # Adjust U if determinant is negative
        R = U @ Vt
    
    # Compute translation vector
    T = centroid_camera - R @ centroid_world
    
    return R, T

R, T = find_camera_extrinsics((verts_world, verts_camera)
verts_camera_pred = (verts_world @ R.T) + T
print(torch.allclose(verts_camera_pred, verts_camera, atol=1e-6))  # Should return True