Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent Topic Visibility and std::bad_alloc Crashes in Isaac ROS with iRobot Create 3 on Nvidia Jetson ORIN (rmw_cyclone_cpp) #59

Open
kmelti opened this issue Oct 18, 2024 · 1 comment
Open

Inconsistent Topic Visibility and std::bad_alloc Crashes in Isaac ROS with iRobot Create 3 on Nvidia Jetson ORIN (rmw_cyclone_cpp) #59

kmelti opened this issue Oct 18, 2024 · 1 comment

Comments

@kmelti
Copy link

kmelti commented Oct 18, 2024

Hello there,

I am using an iRobot Create 3 connected via Ethernet to an Nvidia Jetson ORIN, running the Isaac-ROS NVIDIA Docker with the ROS 2 Humble distribution. I have installed the necessary icreate-robot packages, and the connection to the robot works correctly—I am able to send commands successfully.

However, when I launch the Isaac ROS Visual Slam package with the same domain ID as the robot (ROS_DOMAIN_ID=15), the following happens:

ROS_DOMAIN_ID 15
Launching irobot...
[INFO] [launch]: All log files can be found below /root/.ros/log/2024-10-18-14-23-09-413178-irobot-nav-46
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [60]
[component_container_mt-1] [INFO] [1729261390.376779482] [shared_nvblox_container]: Load Library: /isaac_ros/install/realsense_splitter/lib/librealsense_splitter_component.so
[component_container_mt-1] [INFO] [1729261390.399077701] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvblox::RealsenseSplitterNode>
[component_container_mt-1] [INFO] [1729261390.399214277] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvblox::RealsenseSplitterNode>
[component_container_mt-1] [INFO] [1729261390.409318376] [camera.realsense_splitter_node]: Creating a RealsenseSplitterNode().
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/camera/realsense_splitter_node' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261390.439180045] [shared_nvblox_container]: Load Library: /isaac_ros/install/realsense2_camera/lib/librealsense2_camera.so
[component_container_mt-1] [INFO] [1729261390.688660088] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1729261390.688784441] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1729261390.723771695] [camera.camera]: RealSense ROS v4.51.1
[component_container_mt-1] [INFO] [1729261390.723897391] [camera.camera]: Built with LibRealSense v2.56.1
[component_container_mt-1] [INFO] [1729261390.723924047] [camera.camera]: Running with LibRealSense v2.56.1
[component_container_mt-1] [INFO] [1729261390.725845622] [shared_nvblox_container]: Load Library: /isaac_ros/install/isaac_ros_visual_slam/lib/libvisual_slam_node.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/camera/camera' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261390.760082986] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::visual_slam::VisualSlamNode>
[component_container_mt-1] [INFO] [1729261390.760212810] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::visual_slam::VisualSlamNode>
[component_container_mt-1] [INFO] [1729261390.817062570] [visual_slam_node]: cuVSLAM version: 11.4
[component_container_mt-1] [INFO] [1729261391.286244764] [camera.camera]: Device with serial number 242322075051 was found.
[component_container_mt-1]
[component_container_mt-1] [INFO] [1729261391.286382812] [camera.camera]: Device with physical ID 2-1-2 was found.
[component_container_mt-1] [INFO] [1729261391.286404540] [camera.camera]: Device with name Intel RealSense D435I was found.
[component_container_mt-1] [INFO] [1729261391.287092383] [camera.camera]: Device with port number 2-1 was found.
[component_container_mt-1] [INFO] [1729261391.287135231] [camera.camera]: Device USB type: 3.2
[component_container_mt-1] [INFO] [1729261391.288714724] [camera.camera]: getParameters...
[component_container_mt-1] [INFO] [1729261391.289301126] [camera.camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1729261391.289344742] [camera.camera]: Device Name: Intel RealSense D435I
[component_container_mt-1] [INFO] [1729261391.289364838] [camera.camera]: Device Serial No: 242322075051
[component_container_mt-1] [INFO] [1729261391.289380423] [camera.camera]: Device physical port: 2-1-2
[component_container_mt-1] [INFO] [1729261391.289393671] [camera.camera]: Device FW version: 5.13.0.50
[component_container_mt-1] [INFO] [1729261391.289406119] [camera.camera]: Device Product ID: 0x0B3A
[component_container_mt-1] [INFO] [1729261391.289417703] [camera.camera]: Sync Mode: Off
[component_container_mt-1] [WARN] [1729261395.221548045] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261395.222460432] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261395.224452663] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261396.026933569] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261396.027252386] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261396.028845991] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [INFO] [1729261396.051922677] [camera.camera]: Stopping Sensor: Stereo Module
[component_container_mt-1] [INFO] [1729261396.337416724] [camera.camera]: Starting Sensor: Stereo Module
[component_container_mt-1] [INFO] [1729261396.454462013] [camera.camera]: Open profile: stream_type: Infra(1), Format: Y8, Width: 640, Height: 480, FPS: 60
[component_container_mt-1] [INFO] [1729261396.454606877] [camera.camera]: Open profile: stream_type: Infra(2), Format: Y8, Width: 640, Height: 480, FPS: 60
[component_container_mt-1] [INFO] [1729261396.454640701] [camera.camera]: Open profile: stream_type: Depth(0), Format: Z16, Width: 640, Height: 480, FPS: 60
[component_container_mt-1] [INFO] [1729261396.455086271] [camera.camera]: Stopping Sensor: RGB Camera
[component_container_mt-1]  18/10 14:23:16,458 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1] [INFO] [1729261396.462526264] [camera.camera]: Starting Sensor: RGB Camera
[component_container_mt-1] [INFO] [1729261396.492937054] [camera.camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 640, Height: 480, FPS: 15
[component_container_mt-1] [INFO] [1729261396.495037957] [camera.camera]: RealSense Node Is Up!
[component_container_mt-1] [WARN] [1729261396.530686461] [camera.camera]:
[component_container_mt-1]  18/10 14:23:16,574 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,789 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,799 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,810 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,821 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,945 ERROR [281469993863472] (uvc-streamer.cpp:105) uvc streamer watchdog triggered on endpoint: 131
[component_container_mt-1]  18/10 14:23:18,493 ERROR [281469968685360] (uvc-streamer.cpp:105) uvc streamer watchdog triggered on endpoint: 132
[component_container_mt-1] [INFO] [1729261405.740218472] [visual_slam_node]: Time taken by CUVSLAM_WarmUpGPU(): 14.923056
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/visual_slam_node' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261405.763154004] [visual_slam_node]: left_pose_right reading from CameraInfo
[component_container_mt-1] [INFO] [1729261405.763272181] [visual_slam_node]: Baseline is : 0.050081
[component_container_mt-1] [INFO] [1729261405.763345589] [visual_slam_node]: Use use_gpu: true
[component_container_mt-1] [INFO] [1729261405.763369461] [visual_slam_node]: Enable IMU Fusion: false
[component_container_mt-1] [INFO] [1729261405.764972346] [shared_nvblox_container]: Load Library: /isaac_ros/install/odometry_flattener/lib/libodometry_flattener_component.so
[component_container_mt-1] [INFO] [1729261405.777143043] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvblox::OdometryFlattenerNode>
[component_container_mt-1] [INFO] [1729261405.777276611] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvblox::OdometryFlattenerNode>
[component_container_mt-1] [INFO] [1729261405.791782580] [odometry_flattener_node]: Creating a OdometryFlattenerNode().
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/odometry_flattener_node' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261405.833976512] [visual_slam_node]: Time taken by CUVSLAM_CreateTracker(): 0.070590
[component_container_mt-1] [WARN] [1729261409.023345713] [visual_slam_node]: Delta between current and previous frame [100.175104] is above threshold [33.330000]

On the host system, after exporting ROS_DOMAIN_ID=15, I can only see the topics from the iRobot Create 3:
ros2 topic list

/irobot/battery_state
/irobot/cliff_intensity
/irobot/cmd_audio
/irobot/cmd_lightring
/irobot/cmd_vel
/irobot/cmd_vel_stamped
/irobot/dock_status
/irobot/hazard_detection
/irobot/imu
/irobot/interface_buttons
/irobot/ir_intensity
/irobot/ir_opcode
/irobot/kidnap_status
/irobot/mobility_monitor/transition_event
/irobot/mouse
/irobot/odom
/irobot/robot_state/transition_event
/irobot/slip_status
/irobot/static_transform/transition_event
/irobot/stop_status
/irobot/tf
/irobot/tf_static
/irobot/wheel_status
/irobot/wheel_ticks
/irobot/wheel_vels
/parameter_events
/rosout
/tf

However, when I launch the Isaac ROS Visual Slam package with the same domain ID as the robot (ROS_DOMAIN_ID=15), I encounter various issues:

In some cases, the topics from the isaac_ros package are visible, but no data is published.

In other cases, the topics from the robot disappear entirely.

Additionally, I sometimes have the topics from the robot, but other ROS 2 packages crash with the following error:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 4:    46 Aborted

Additionally, running this command on the Jetson:

ros2 launch create3_republisher create3_republisher_launch.py republisher_ns:=/irobot_republish robot_ns:=/irobot
also crashes with the same error as above. I am using rmw_cyclone_cpp as the DDS.

I am using rmw_cyclone_cpp as the DDS.

Question:

  1. Did I miss something in my configuration that could be causing the inconsistent topic visibility or the std::bad_alloc errors?
  2. Are there specific settings or adjustments I need to make for DDS (rmw_cyclone_cpp) to ensure reliable communication across nodes with the same ROS_DOMAIN_ID?
@martincerven
Copy link

Hey @kmelti, by ethernet you mean USB cable?

I also have Jetson & Create3 and I wanted to test these Isaac ROS algorithms...

  • Do you have Isaac in containers?
  • Do you have cyclone set up everywhere?
  • Do you have synced time? - with chrony for example

I will try them later, but It might be one of two:

  • time issue: the navigating packages need fresh data, even if you are off by few seconds your queues could fill and crash?
  • permissions/flags/RMW profile setup inside containers?

Are you using custom RMW profile? How are you syncing time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@martincerven @kmelti