Bug report: Build hl2ss_ulm.so and main_ulm.cpp

[Secret-Lin]

Thanks for your great work. I am trying to do some image process on what holoLens2 sees. So, I wanted to import hl2ss as shared library(hl2ss_ulm.so). Here are bugs I met and how I solved:
For building hl2ss_ulm.so:

1. ‘std’ does not name a type, like mutex; How to fixed: using c++17.
2. ‘wstring_convert’ is not a member of ‘std’; How to fixed: using c++17 and add '#include < l o c a l e >' (delete spaces)in hl2ss_ulm.cpp.
3. Not find defination of hl2ss_mt::source::xxxxx; How to fixed: run 'g++ -fPIC -shared -std=c++17 -o libhl2ss_ulm.so hl2ss_ulm.cpp hl2ss_mt.cpp' to build hl2ss_ulm.so.
   For building main_ulm:
4. No such file or directory of xxxxx, like opencv or can not find lhl2ss_ulm. Here is the tasks.json of I used:
   '{
   "version": "2.0.0",
   "tasks": [
   {
   "type": "cppbuild",
   "label": "C/C++: g++ build active file",
   "command": "/usr/bin/g++",
   "args": [
   "-fdiagnostics-color=always",
   "-g",
   "-std=c++17",
   "${file}",
   "hl2ss.cpp",
   "hl2ss_lnm.cpp",
   "../../3rdparty/Zdepth/src/.cpp",
   "../../3rdparty/Zdepth/zstd/src/.c",
   "-o",
   "${fileDirname}/${fileBasenameNoExtension}",
   "-DHL2SS_ENABLE_ZDEPTH",
   "-I../../3rdparty/Zdepth/include",
   "-I../../3rdparty/Zdepth/zstd/include",
   "-I../../3rdparty/Zdepth/zstd/src",
   "-lavcodec",
   "-lavutil",
   "-I/usr/include/opencv4", // OpenCV 4 include path
   "-L/usr/local/lib",
   "-L/home/rvc/colcon_ws/utils/hl2ss/extensions/client_cpp",
   "-lopencv_highgui",
   "-lopencv_imgcodecs",
   "-lopencv_imgproc",
   "-lopencv_core",
   "-O3",
   "-lhl2ss_ulm",
   "-Wl,-rpath,/home/rvc/colcon_ws/utils/hl2ss/extensions/client_cpp",
   "-lpthread"
   ],
   "options": {
   "cwd": "${fileDirname}"
   },
   "problemMatcher": [
   "$gcc"
   ],
   "group": {
   "kind": "build",
   "isDefault": true
   },
   "detail": "Task generated by Debugger."
   }
   ]

}'
2. error: a reinterpret_cast is not a constant expression. In hl2ss_ulm.h, the author uses constexpr to void multidefination, but reinterpret_cast can not be used in the return. How to fixed: here is the code I changed for constexpr functions in hl2ss_ulm.h:
'constexpr
map_rm_vlc unpack_rm_vlc(uint8_t* payload)
{
rm_vlc_metadata* metadata = reinterpret_cast<rm_vlc_metadata*>(payload + parameters_rm_vlc::PIXELS);
return { payload, metadata };
// return { payload, (rm_vlc_metadata*)(payload + parameters_rm_vlc::PIXELS) };
}
constexpr
map_rm_depth_ahat unpack_rm_depth_ahat(uint8_t* payload)
{
uint16_t* payload_temp = (uint16_t*)(payload);
uint16_t* return_value2 = (uint16_t*)(payload + (parameters_rm_depth_ahat::PIXELS * sizeof(uint16_t)));
rm_depth_ahat_metadata* metadata = (rm_depth_ahat_metadata*)(payload + (2 * parameters_rm_depth_ahat::PIXELS * sizeof(uint16_t)));
return { payload_temp, return_value2, metadata };
}

constexpr
map_rm_depth_longthrow unpack_rm_depth_longthrow(uint8_t* payload)
{
uint16_t* payload_temp = (uint16_t*)(payload);
uint16_t* return_value2 = (uint16_t*)(payload + (parameters_rm_depth_longthrow::PIXELS * sizeof(uint16_t)));
rm_depth_longthrow_metadata* metadata = (rm_depth_longthrow_metadata*)(payload + (2 * parameters_rm_depth_longthrow::PIXELS * sizeof(uint16_t)));
return { payload_temp, return_value2, metadata };
}

constexpr
map_rm_imu unpack_rm_imu(uint8_t* payload)
{
rm_imu_sample* payload_temp = (rm_imu_sample*)(payload);
return { payload_temp };
}

constexpr
map_pv unpack_pv(uint8_t* payload, uint32_t size)
{
pv_metadata* metadata = (pv_metadata*)(payload + size - sizeof(pv_metadata));
return { payload, metadata };
}

constexpr
map_microphone_raw unpack_microphone_raw(uint8_t* payload)
{
int16_t* payload_temp = (int16_t*)(payload);
return { payload_temp };
}

constexpr
map_microphone_aac unpack_microphone_aac(uint8_t* payload)
{
float* payload_temp = (float*)(payload);
return { payload_temp };
}

constexpr
map_microphone_array unpack_microphone_array(uint8_t* payload)
{
float* payload_temp = (float*)(payload);
return { payload_temp };
}

constexpr
map_si unpack_si(uint8_t* payload)
{
si_frame* payload_temp = (si_frame*)(payload);
return { payload_temp };
}

constexpr
map_eet unpack_eet(uint8_t* payload)
{
eet_frame* payload_temp = (eet_frame*)(payload);
return { payload_temp };
}

constexpr
map_extended_audio_raw unpack_extended_audio_raw(uint8_t* payload)
{
int16_t* payload_temp = (int16_t*)(payload);
return { payload_temp };
}

constexpr
map_extended_audio_aac unpack_extended_audio_aac(uint8_t* payload)
{
float* payload_temp = (float*)(payload);
return { payload_temp };
}'
Finally, I successfully built main_ulm and saw the pv. Thanks again for the author's great work. And these bugs may due to my ubuntu20.04 foxy env. Hope this issue can help someone.

[jdibenes]

Very cool. Thank you.
Indeed, I only tested the client shared library on Windows and MSVC.

[Secret-Lin]

@jdibenes I figure that you do not write a corresponding hl2ss_3dcv for cpp, I want to transfer the value of longthrow to depth refer to its camera in cpp. By reading your sample_pv_depth_lt.py, I have following questions:

1. Are the world coordinates of pv and lt the same one?
2. Why you do transformation on xy1_o and xy1_d seperately and finally hstack them togather?

(By the way, I cannot run sample_pv_depth_lt.py, it just stuck and cannot be killed. And I can run client_steam_*.py to see the sensors' information)

[jdibenes]

1. It is the same coordinate frame but have different transforms wrt rig node.
2. To extract depth from xy1_o and for readability. Some of the transforms could be combined for efficiency.
3. Try import hl2ss_imshow before import hl2ss.

[Secret-Lin]

Thanks for replying, this work well:

3. Try import hl2ss_imshow before import hl2ss.

When I test result of sample_pv_depth_lt.py, I found that you use hl2ss_3dcv.pv_fix_calibration(pv_intrinsics, pv_extrinsics) to generate color_extrinsics where pv_extrinsics is np.eyes. And color_extrinsics is
[[ 1. 0. 0. 0.]
[ 0. -1. 0. 0.]
[ 0. 0. -1. 0.]
[ 0. 0. 0. 1.]]
My question is why not use extrinsics from calibration_pv (calibration_pv = hl2ss_3dcv.get_calibration_pv(host, hl2ss.StreamPort.PERSONAL_VIDEO, calibration_path, 1000, pv_width, pv_height, pv_fps) from #134 )? Here is the calibration_pv.extrinsics:
[[ 0.0281502 -0.9993722 0.02151053 0. ]
[-0.99930006 -0.0276047 0.02524317 0. ]
[-0.02463353 -0.02220608 -0.99944973 0. ]
[-0.0494109 0.00193774 0.01321972 1. ]]
I test the result using calibration_pv.extrinsics to replace color_extrinsics, the max diff of pv_depth and pv_depth_origin is 0.018292844 for a fixed holoLens facing a black flat wall with around 0.3m.Maybe the diff is acceptable.

I test the final pv_z and pv_z_origin, the max diff is 0.44158256 which is not acceptable. And need change v0:v1 to v1:v0 as v1<v0.