This repo contains ROS2 drivers for various sensors to collect raw, synchronized, high-speed RGB video, stereoscopic depth video, mmWave radar, thermal, event, LiDAR and audio measurements.
This repo also contains all mechanical designs used, as well as extrinsic calibration data and procedures used in the design of this system.
In addition, we present the first M2S2 dataset containing wild cheetah behaviour.
- High-speed RGB Camera: Ximea -- MQ022CG-CM (Driver: ximea_ROS2_driver)
- RGBD Camera: Intel RealSense D435i Stereo Camera (Driver: realsense_ROS2_driver
- Thermal Camera: FLIR Boson 640 (Driver: flir_boson_ROS2_driver)
- LiDAR: Livox Avia (Driver: livox_lidar_ROS2_driver)
- Radar: Texas Instruments DCA1000 Raw ADC capture board (Driver: coming soon)
- Event Camera: Dynamic Vision iniVation DVXplorer (Driver: dvxplorer_ROS2_driver)
- Microphone: Wildtronics Pro Mono Parabolic Microphone (Driver: audio_ROS2_driver)
- Mini PC: Intel NUC 13 Pro
-
For dependencies of specific drivers, please refer to the corresponding README file.
For ROS2 installation, please refer to the official ROS2 installation guide: ROS2-foxy installation guide
Using eCAL RWM as an alternative to ROS2 DDS middleware implementations showed significant perfomance improvements.
Install eCAL
$ sudo add-apt-repository ppa:ecal/ecal-latest
$ sudo apt-get update
$ sudo apt-get install ecalClone the latest release of the ecal-rmw repository into your ROS2 workspace and build it.
$ cd ~/ros2_ws
$ git clone https://github.com/eclipse-ecal/rmw_ecal.git
$ cd ~/ros2_ws/rmw_ecal
$ colcon build --packages-skip rmw_ecal_proto_cpp
$ vim ~/.bashrc # add "source ~/ros2_ws/rmw_ecal/install/setup.bash"
$ source ~/.bashrcAdd the following line into your .bashrc to run all nodes using eCAL middleware
export RMW_IMPLEMENTATION=rmw_ecal_dynamic_cppThe launch files in this repo can be used to launch the ros2 nodes of all the sensors used in M2S2 simultaneously and visualise (and hear) the output streams.
clone this repo to your ros2 workspace source directory:
$ cd ~/ros2_ws
$ git clone https://github.com/African-Robotics-Unit/M2S2.git
$ cd ~/ros2_ws/M2S2
$ colcon build --packages-select m2s2_bringup
$ ros2 launch m2s2_bringup m2s2.launch.py
# To view the stream
$ ros2 launch m2s2_bringup m2s2_display.launch.py Note: For usage instructions of specific drivers, please refer to the corresponding README file.
Using the eCAL RMW, one can record data using their powerful recording tool. All data is stored as an ecal_measurement in .hdf5 files.
Our m2s2_ecal_deserializers provides deserializers to convert raw M2S2 measurements into readable data.
The Intel Realsense RGB camera was used as the reference (global) frame for all extrinsic calibration between M2S2 sensors.
To perform extrinsic calibration between the Intel RealSense and FLIR Boson 640, we propose a calibration target made of a laser-cut checkerboard pattern overlayed onto an aluminum covered surface. The checkerboard is then heated using two halogen lamps. This creates distinct features in the calibration target and the board is visible in both the RGB and IR camera.
Given the different resolutions of the thermal images (640 x 512) and the reference images from the Intel RealSense (1080 x 720), we used the MATLAB community Calibration-Under Different-Resolution toolbox. This toolboox modifies Matlab’s calibration app to accept cameras of different resolutions.
Matlab's Lidar Camera Calibrator app was used to perform geometric calibration between the Livox Avia and the Intel RealSense.
We used the open-source toolbox E2Calib to perform event-camera calibration. The toolbox reconstructs visible images from event data using E2VID. The reconstructed images were then used in Matlab's Stereo Camera calibrator app to perform extrinsic calibration between the event camera and the reference Intel RealSense.
A moving corner reflector target was tracked in the realsense and radar FOV. Using the camera intrinsics, the azimuth and depth of the corner reflector was determined and compared to the output of the radar's range azimuth process. The Range-Azimuth CFAR matrix was rolled (shifted and wrapped) in range and azimuth to minimize the error between the two outputs over time.
Matlab's Stereo Camera calibrator app was used to perform extrinsic calibration between the Ximea and Intel RealSense.
All calibration data used can be found here: M2S2 Calibration Data
Download our CAD design for our casing and 3D designs for our various sensor mounts here: M2S2 CAD
Download our dataset in .hdf5 format here: M2S2 Dataset