Ambiguous 'keypoints' field in data_dictionary for training

[askisc]

Hi @neeek2303, question about the keypoints field in the items loaded from the pickle files during training, specifically the ones from the datasets/extrime_faces_pairs.py, datasets/voxceleb2hq_pairs.py, and datasets/mead_faces_pairs.py dataset scripts. When generating these 'keypoints' during preprocessing, are these the pitch, yaw, and roll values calculated from the https://github.com/hhj1897/face_detection github repo, or is the field representative of another set of points and calculated by some other means?

[kebijuelun]

@neeek2303
Thank you for the excellent open-source work.

• It seems that the keypoints are 3D facial keypoints. Could you please let me know which code library this is based on?
• And is it feasible to train without using keypoints and masks in dataloader? My understanding is that if we remove the keypoint-related logic and set all the masks to 1, it should work

[askisc]

@neeek2303 Thank you for the excellent open-source work.

• It seems that the keypoints are 3D facial keypoints. Could you please let me know which code library this is based on?
• And is it feasible to train without using keypoints and masks in dataloader? My understanding is that if we remove the keypoint-related logic and set all the masks to 1, it should work

Reading the EMOPortraits paper, the data preparation implementation details mentions it follows the same protocols established for MegaPortraits. This MegaPortraits paper mentions the use of Adrian Bulat's work as a keypoint detector, which seems to detect 3D facial keypoints.

[kebijuelun]

@neeek2303 Thank you for the excellent open-source work.

• It seems that the keypoints are 3D facial keypoints. Could you please let me know which code library this is based on?
• And is it feasible to train without using keypoints and masks in dataloader? My understanding is that if we remove the keypoint-related logic and set all the masks to 1, it should work

Reading the EMOPortraits paper, the data preparation implementation details mentions it follows the same protocols established for MegaPortraits. This MegaPortraits paper mentions the use of Adrian Bulat's work as a keypoint detector, which seems to detect 3D facial keypoints.

Thank you for your response! I had also guessed that the face-alignment library (https://github.com/1adrianb/face-alignment) could be used to extract keypoints. After trying it out, I found that the data format matches, and the model training converges properly.