Duration is out of dual 32-bit range

[xav12358]

Hello,

I get a strange error when I use libfranka. After few move I get that error:

[ INFO] [1650017555.624464339]: Controller 'position_joint_trajectory_controller' successfully finished
[ INFO] [1650017555.707375009]: Completed trajectory execution with status SUCCEEDED ...
[ INFO] [1650017555.714236078]: Received event 'stop'
[ INFO] [1650017555.812988489]: Received request to compute Cartesian path
[ INFO] [1650017555.813370991]: Attempting to follow 2 waypoints for link 'panda_link8' using a step of 0.010000 m and jump threshold 0.000000 (in global reference frame)
[ INFO] [1650017555.813802012]: Computed Cartesian path with 3 points (followed 100.000000% of requested trajectory)
[ INFO] [1650017555.814520703]: Execution request received
[ INFO] [1650017556.041427698]: Controller 'position_joint_trajectory_controller' successfully finished
[ INFO] [1650017556.140868267]: Completed trajectory execution with status SUCCEEDED ...
[ INFO] [1650017556.140992920]: Execution completed: SUCCEEDED
[ INFO] [1650017557.674071654]: Combined planning and execution request received for MoveGroup action. Forwarding to planning and execution pipeline.
[ INFO] [1650017557.674226121]: Planning attempt 1 of at most 1
[ INFO] [1650017557.675771675]: Planner configuration 'panda_arm' will use planner 'geometric::RRTConnect'. Additional configuration parameters will be set when the planner is constructed.
[ INFO] [1650017557.676177264]: panda_arm/panda_arm: Starting planning with 1 states already in datastructure
[ INFO] [1650017557.699567260]: panda_arm/panda_arm: Created 5 states (2 start + 3 goal)
[ INFO] [1650017557.699626543]: Solution found in 0.023645 seconds
[ INFO] [1650017557.716953937]: SimpleSetup: Path simplification took 0.017239 seconds and changed from 4 to 2 states
[ WARN] [1650017557.917534940]: FrankaHW: 
	panda_joint4: 48.052401 degrees to joint limits (limits: [-3.071800, -0.069800] q: -2.233127) 
[ INFO] [1650017560.191609366]: Controller 'position_joint_trajectory_controller' successfully finished
[ INFO] [1650017560.274275395]: Completed trajectory execution with status SUCCEEDED ...
[ INFO] [1650017560.281329970]: Received event 'stop'
[ INFO] [1650017560.340796921]: Combined planning and execution request received for MoveGroup action. Forwarding to planning and execution pipeline.
[ INFO] [1650017560.340938019]: Planning attempt 1 of at most 1
[ INFO] [1650017560.341766800]: Planner configuration 'panda_arm' will use planner 'geometric::RRTConnect'. Additional configuration parameters will be set when the planner is constructed.
[ INFO] [1650017560.341938683]: panda_arm/panda_arm: Starting planning with 1 states already in datastructure
[ INFO] [1650017560.357619875]: panda_arm/panda_arm: Created 5 states (2 start + 3 goal)
[ INFO] [1650017560.357657397]: Solution found in 0.015800 seconds
[ INFO] [1650017560.390585205]: SimpleSetup: Path simplification took 0.032877 seconds and changed from 4 to 2 states
[ WARN] [1650017562.918474636]: FrankaHW: 
	panda_joint4: 47.972937 degrees to joint limits (limits: [-3.071800, -0.069800] q: -2.234514) 
[ INFO] [1650017563.157946778]: Controller 'position_joint_trajectory_controller' successfully finished
[ INFO] [1650017563.240689409]: Completed trajectory execution with status SUCCEEDED ...
[ INFO] [1650017563.247838920]: Received event 'stop'
[ INFO] [1650017563.342252969]: Received request to compute Cartesian path
[ INFO] [1650017563.342370025]: Attempting to follow 2 waypoints for link 'panda_link8' using a step of 0.010000 m and jump threshold 0.000000 (in global reference frame)
[ INFO] [1650017563.342754963]: Computed Cartesian path with 3 points (followed 100.000000% of requested trajectory)
[ INFO] [1650017563.343340847]: Execution request received
terminate called after throwing an instance of 'std::runtime_error'
  what():  Duration is out of dual 32-bit range
================================================================================REQUIRED process [franka_control-3] has died!
process has died [pid 139304, exit code -6, cmd /home/fff/dev/catkin_ws/devel/lib/franka_control/franka_control_node __name:=franka_control __log:=/home/fff/.ros/log/8a163c10-bca4-11ec-b7e8-f91b2a3bdfbe/franka_control-3.log].
log file: /home/fff/.ros/log/8a163c10-bca4-11ec-b7e8-f91b2a3bdfbe/franka_control-3*.log
Initiating shutdown!
================================================================================
[rviz_fff_139214_966731689332244255-10] killing on exit
[move_group-9] killing on exit
[virtual_joint_broadcaster_0-8] killing on exit
[controller_spawner-7] killing on exit
[joint_state_publisher-6] killing on exit
[robot_state_publisher-5] killing on exit
[state_controller_spawner-4] killing on exit
[franka_control-3] killing on exit
[franka_gripper-2] killing on exit
[INFO] [1650017563.645560]: Shutting down spawner. Stopping and unloading controllers...
[INFO] [1650017563.645870]: Shutting down spawner. Stopping and unloading controllers...
[INFO] [1650017563.647024]: Stopping all controllers...
[INFO] [1650017563.647739]: Stopping all controllers...
[WARN] [1650017563.649646]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[WARN] [1650017563.650208]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete

I use gdb to debug it and here is the result:

here
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff74bd859 in __GI_abort () at abort.c:79
#2  0x00007ffff7746911 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff775238c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff77523f7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff77526fd in __cxa_rethrow () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7f89054 in franka::ControlLoop<franka::JointPositions>::operator()() () from /home/fff/dev/libfranka/build/libfranka.so.0.9
#7  0x00007ffff7fa5e6d in franka::Robot::control(std::function<franka::JointPositions (franka::RobotState const&, franka::Duration)>, franka::ControllerMode, bool, double) ()
   from /home/fff/dev/libfranka/build/libfranka.so.0.9
#8  0x00007ffff7df4ed9 in franka_hw::FrankaHW::<lambda(franka::Robot&, Callback)>::operator()(franka::Robot &, franka_hw::FrankaHW::<lambda(franka::Robot&, Callback)>::Callback) const (
    __closure=0x7fffdc002ce0, robot=..., ros_callback=...) at /home/fff/dev/catkin_ws/src/franka_ros/franka_hw/src/franka_hw.cpp:455
#9  0x00007ffff7df85bc in std::_Function_handler<void(franka::Robot&, std::function<bool(const franka::RobotState&, franka::Duration)>), franka_hw::FrankaHW::setRunFunction(const franka_hw::ControlMode&, bool, double, franka::ControllerMode)::<lambda(franka::Robot&, Callback)> >::_M_invoke(const std::_Any_data &, franka::Robot &, std::function<bool(const franka::RobotState&, franka::Duration)> &&) (
    __functor=..., __args#0=..., __args#1=...) at /usr/include/c++/9/bits/std_function.h:300
#10 0x00007ffff7e05af5 in std::function<void (franka::Robot&, std::function<bool (franka::RobotState const&, franka::Duration)>)>::operator()(franka::Robot&, std::function<bool (franka::RobotState const&, franka::Duration)>) const (this=0x7fffffffd930, __args#0=..., __args#1=...) at /usr/include/c++/9/bits/std_function.h:688
#11 0x00007ffff7df23b4 in franka_hw::FrankaHW::control(std::function<bool (ros::Time const&, ros::Duration const&)> const&) (this=0x7fffffffb360, ros_callback=...)
    at /home/fff/dev/catkin_ws/src/franka_ros/franka_hw/src/franka_hw.cpp:234
#12 0x00005555555fa3d4 in main (argc=1, argv=0x7fffffffda78) at /home/fff/dev/catkin_ws/src/franka_ros/franka_control/src/franka_control_node.cpp:141
(gdb) 

My example run on Ubuntu 20.04 on Noetic.
Libfranka is compile in debug mode
Franka ros is compile also in debug mode using libfranka build.

Edit:
I build libfranka in debug:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff72d8859 in __GI_abort () at abort.c:79
#2  0x00007ffff7561911 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff756d38c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff756d3f7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff756d6fd in __cxa_rethrow () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7eed857 in franka::ControlLoop<franka::JointPositions>::operator() (this=0x7fffffff9ed0) at /home/mdt/dev/libfranka/src/control_loop.cpp:157
#7  0x00007ffff7f4d89d in franka::Robot::control(std::function<franka::JointPositions (franka::RobotState const&, franka::Duration)>, franka::ControllerMode, bool, double) (this=0x5555556b59a0, 
    motion_generator_callback=..., controller_mode=franka::ControllerMode::kJointImpedance, limit_rate=true, cutoff_frequency=100) at /home/mdt/dev/libfranka/src/robot.cpp:145
#8  0x00007ffff7c0fed9 in franka_hw::FrankaHW::<lambda(franka::Robot&, Callback)>::operator()(franka::Robot &, franka_hw::FrankaHW::<lambda(franka::Robot&, Callback)>::Callback) const (
    __closure=0x7fffd400d770, robot=..., ros_callback=...) at /home/mdt/dev/catkin_ws/src/franka_ros/franka_hw/src/franka_hw.cpp:455
#9  0x00007ffff7c135bc in std::_Function_handler<void(franka::Robot&, std::function<bool(const franka::RobotState&, franka::Duration)>), franka_hw::FrankaHW::setRunFunction(const franka_hw::ControlMode&, bool, double, franka::ControllerMode)::<lambda(franka::Robot&, Callback)> >::_M_invoke(const std::_Any_data &, franka::Robot &, std::function<bool(const franka::RobotState&, franka::Duration)> &&) (
    __functor=..., __args#0=..., __args#1=...) at /usr/include/c++/9/bits/std_function.h:300
#10 0x00007ffff7c20af5 in std::function<void (franka::Robot&, std::function<bool (franka::RobotState const&, franka::Duration)>)>::operator()(franka::Robot&, std::function<bool (franka::RobotState const&, franka::Duration)>) const (this=0x7fffffffd930, __args#0=..., __args#1=...) at /usr/include/c++/9/bits/std_function.h:688
#11 0x00007ffff7c0d3b4 in franka_hw::FrankaHW::control(std::function<bool (ros::Time const&, ros::Duration const&)> const&) (this=0x7fffffffb360, ros_callback=...)
    at /home/mdt/dev/catkin_ws/src/franka_ros/franka_hw/src/franka_hw.cpp:234
#12 0x00005555555fa3d4 in main (argc=1, argv=0x7fffffffda78) at /home/mdt/dev/catkin_ws/src/franka_ros/franka_control/src/franka_control_node.cpp:141

[marcbone]

Indeed very strange. Do the libfranka examples work for you?
You could also try to print out the robot state and check if it contains some error right before this line

[xav12358]

The example seems to work properly but I did try several time to notice the trouble. In fact it comes after few trajectories.
How could I print the robot state ?

Edit :

I finally manage to display robotstate at the line

robot_state 
{
   "O_T_EE":[
      0.996945,
      -0.0619811,
      0.0473318,
      0,
      -0.0607056,
      -0.997755,
      -0.0279273,
      0,
      0.0489575,
      0.0249692,
      -0.998489,
      0,
      0.498686,
      0.0542688,
      0.346505,
      1
   ],
   "O_T_EE_d":[
      0.996919,
      -0.0621948,
      0.0475884,
      0,
      -0.0609586,
      -0.997765,
      -0.0270017,
      0,
      0.0491623,
      0.0240181,
      -0.998502,
      0,
      0.498704,
      0.054228,
      0.346497,
      1
   ],
   "F_T_NE":[
      0.7071,
      -0.7071,
      0,
      0,
      0.7071,
      0.7071,
      0,
      0,
      0,
      0,
      1,
      0,
      0,
      0,
      0.1034,
      1
   ],
   "NE_T_EE":[
      1,
      0,
      0,
      0,
      0,
      1,
      0,
      0,
      0,
      0,
      1,
      0,
      0,
      0,
      0,
      1
   ],
   "F_T_EE":[
      0.7071,
      -0.7071,
      0,
      0,
      0.7071,
      0.7071,
      0,
      0,
      0,
      0,
      1,
      0,
      0,
      0,
      0.1034,
      1
   ],
   "EE_T_K":[
      1,
      0,
      0,
      0,
      0,
      1,
      0,
      0,
      0,
      0,
      1,
      0,
      0,
      0,
      0,
      1
   ],
   "m_ee":0.73,
   "F_x_Cee":[
      -0.01,
      0,
      0.03
   ],
   "I_ee":[
      0.001,
      0,
      0,
      0,
      0.0025,
      0,
      0,
      0,
      0.0017
   ],
   "m_load":0,
   "F_x_Cload":[
      0,
      0,
      0
   ],
   "I_load":[
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "m_total":0.73,
   "F_x_Ctotal":[
      -0.01,
      0,
      0.03
   ],
   "I_total":[
      0.001,
      0,
      0,
      0,
      0.0025,
      0,
      0,
      0,
      0.0017
   ],
   "elbow":[
      -1.22372,
      -1
   ],
   "elbow_d":[
      -1.22365,
      -1
   ],
   "elbow_c":[
      -1.22365,
      -1
   ],
   "delbow_c":[
      0,
      0
   ],
   "ddelbow_c":[
      0,
      0
   ],
   "tau_J":[
      0.139658,
      16.4523,
      23.1207,
      12.8739,
      0.399223,
      1.9573,
      0.27099
   ],
   "tau_J_d":[
      0,
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "dtau_J":[
      -27.2432,
      82.5512,
      -8.17079,
      67.8198,
      -27.0724,
      1.49786,
      -12.3204
   ],
   "q":[
      1.70736,
      -1.04182,
      -1.22372,
      -2.15479,
      -0.966112,
      1.65601,
      1.65263
   ],
   "dq":[
      -0.000177993,
      -0.000765296,
      0.000301725,
      0.000132541,
      0.000141749,
      -0.000402616,
      -0.000159667
   ],
   "q_d":[
      1.70761,
      -1.04168,
      -1.22365,
      -2.15476,
      -0.966783,
      1.65548,
      1.65316
   ],
   "dq_d":[
      0,
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "ddq_d":[
      0,
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "joint_contact":[
      0,
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "cartesian_contact":[
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "joint_collision":[
      0,
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "cartesian_collision":[
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "tau_ext_hat_filtered":[
      0.140164,
      0.343611,
      0.338843,
      -0.0955836,
      -0.337628,
      -0.396731,
      0.281813
   ],
   "O_F_ext_hat_K":[
      -0.58635,
      0.529576,
      0.677138,
      -0.641246,
      -0.589601,
      0.0643429
   ],
   "K_F_ext_hat_K":[
      -0.585344,
      -0.511713,
      -0.691598,
      -0.50093,
      0.0851334,
      0.205794
   ],
   "O_dP_EE_d":[
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "O_ddP_O":[
      0,
      0,
      -9.81
   ],
   "O_T_EE_c":[
      0.996919,
      -0.0621948,
      0.0475884,
      0,
      -0.0609586,
      -0.997765,
      -0.0270017,
      0,
      0.0491623,
      0.0240181,
      -0.998502,
      0,
      0.498704,
      0.054228,
      0.346497,
      1
   ],
   "O_dP_EE_c":[
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "O_ddP_EE_c":[
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "theta":[
      1.70738,
      -1.04067,
      -1.2221,
      -2.15388,
      -0.966068,
      1.65622,
      1.65266
   ],
   "dtheta":[
      0,
      0,
      0,
      0,
      0,
      0,
      0
   ],
   "current_errors":[
      
   ],
   "last_motion_errors":[
      
   ],
   "control_command_success_rate":0,
   "robot_mode":"Idle",
   "time":655796
}
ControlLoop<T>::operator() 16 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Duration is out of dual 32-bit range

The problem seems to come from the line 176

[xav12358]

Now I get a other error:

robot_state {"O_T_EE": [0.998503,0.00655939,0.0541181,0,0.0107181,-0.99697,-0.0769153,0,0.0534506,0.0773817,-0.995568,0,0.624205,-0.0461188,0.313763,1], "O_T_EE_d": [0.998506,0.00663385,0.0540552,0,0.0107797,-0.996981,-0.07677,0,0.0533837,0.0772395,-0.995582,0,0.62411,-0.0460058,0.31394,1], "F_T_NE": [0.7071,-0.7071,0,0,0.7071,0.7071,0,0,0,0,1,0,0,0,0.1034,1], "NE_T_EE": [1,0,0,0,0,1,0,0,0,0,1,0,0,0,0,1], "F_T_EE": [0.7071,-0.7071,0,0,0.7071,0.7071,0,0,0,0,1,0,0,0,0.1034,1], "EE_T_K": [1,0,0,0,0,1,0,0,0,0,1,0,0,0,0,1], "m_ee": 0.73, "F_x_Cee": [-0.01,0,0.03], "I_ee": [0.001,0,0,0,0.0025,0,0,0,0.0017], "m_load": 0, "F_x_Cload": [0,0,0], "I_load": [0,0,0,0,0,0,0,0,0], "m_total": 0.73, "F_x_Ctotal": [-0.01,0,0.03], "I_total": [0.001,0,0,0,0.0025,0,0,0,0.0017], "elbow": [-0.539811,-1], "elbow_d": [-0.539849,-1], "elbow_c": [-0.539849,-1], "delbow_c": [0,0], "ddelbow_c": [0,0], "tau_J": [0.338022,-34.2416,-3.54739,22.082,0.405631,2.13309,0.171197], "tau_J_d": [0,0,0,0,0,0,0], "dtau_J": [-23.1824,-70.7726,15.5215,-5.69258,-13.1749,-9.29907,-0.338779], "q": [0.400719,0.253954,-0.539811,-1.82521,0.243849,2.07589,0.539669], "dq": [5.47495e-05,0.000430492,-0.00013211,-0.000507437,6.92507e-06,7.0402e-05,0.000216986], "q_d": [0.401021,0.253459,-0.539849,-1.82536,0.243353,2.07562,0.540098], "dq_d": [0,0,0,0,0,0,0], "ddq_d": [0,0,0,0,0,0,0], "joint_contact": [0,0,0,0,0,0,0], "cartesian_contact": [0,0,0,0,0,0], "joint_collision": [0,0,0,0,0,0,0], "cartesian_collision": [0,0,0,0,0,0], "tau_ext_hat_filtered": [0.34337,-0.506352,0.132142,-0.0942173,-0.292322,-0.160086,0.164135], "O_F_ext_hat_K": [-2.74616,1.10059,1.02853,-0.956291,-1.71322,0.350538], "K_F_ext_hat_K": [-2.67922,-1.20583,-1.08559,-0.575427,0.21904,0.162541], "O_dP_EE_d": [0,0,0,0,0,0], "O_ddP_O": [0,0,-9.81], "O_T_EE_c": [0.998506,0.00663385,0.0540552,0,0.0107797,-0.996981,-0.07677,0,0.0533837,0.0772395,-0.995582,0,0.62411,-0.0460058,0.31394,1], "O_dP_EE_c": [0,0,0,0,0,0], "O_ddP_EE_c": [0,0,0,0,0,0], "theta": [0.400743,0.25155,-0.54006,-1.82366,0.243893,2.07612,0.539688], "dtheta": [0,0,0,0,0,0,0], "current_errors": [], "last_motion_errors": ["joint_motion_generator_position_limits_violation"], "control_command_success_rate": 0, "robot_mode": "Idle", "time": 2410974}
terminate called after throwing an instance of 'std::runtime_error'
  what():  Duration is out of dual 32-bit range

Error:
joint_motion_generator_position_limits_violation

[marcbone]

" joint_motion_generator_position_limits_violation" means that you run into the joint limits of the robot. So you should try a different trajectory

[xav12358]

Indeed it comes from a robot joint limit.

For the duration exception, I have decide to remove the throw 👍

It seems to work.

Does it seem good ?

[marcbone]

Interesting. I mean if this works for you 🤷 . However, I am leaving this issue open since there seems to be a different issue somewhere

[xav12358]

The trouble seems to comes again. What could be the procedure to check where it comes from ?

[xav12358]

I have no answer and the trouble is not solved. Is there a way to give you me information? Trace, log, debug info ?

[marcbone]

What I basically need would be the value of every variable in the entire stack. Somewhere something is not properly set. If i would know which variable is the problem I could maybe find the reason. But without being able to reproduce it myself it is very hard to debug this.

[xav12358]

How could I generate the value of every variable in the entire stack? Could I made a direct call to you via Team or Meet?
I could also let the control of my computer temporally

[xav12358]

@marcbone It seems it comes from the version of my moveit. I update the all packages and it seems there is no more pb since 1 week. I hope it will solve my problem definitively

[Maverobot]

The issue seems to be solved. Closed due to inactivity.