rs_int_example failing

[f1ren]

Describe the bug
rs_int_example.py failing on An error occurred during each attempt of this action.

To Reproduce
Steps to reproduce the behavior:

1. ./run_docker.sh
2. python -m omnigibson.examples.action_primitives.rs_int_example

Expected behavior
Program ends after printing Finished executing place.

Screenshots

                  Plan has 2 steps
                  Executing arm movement plan step 1/2
                  Executing arm movement plan step 2/2
Traceback (most recent call last):
  File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 70, in <module>
    main()
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 60, in main
    execute_controller(controller.apply_ref(StarterSemanticActionPrimitiveSet.GRASP, apple), env)
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 18, in execute_controller
    for action in ctrl_gen:
  File "/root/omnigibson/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 545, in apply_ref
    raise ActionPrimitiveErrorGroup(errors)
omnigibson.action_primitives.action_primitive_set_base.ActionPrimitiveErrorGroup: An error occurred during each attempt of this action.

Attempt 0: EXECUTION_ERROR: Could not navigate to the target position. Additional info: {'target pose': (tensor([ 0.0310, -0.6776,  0.0100]), tensor([ 0.0000,  0.0000, -0.8991,  0.4377]))}

Attempt 1: POST_CONDITION_ERROR: Grasp completed, but no object detected in hand after executing grasp. Additional info: {'target object': 'apple'}

Attempt 2: POST_CONDITION_ERROR: Grasp completed, but no object detected in hand after executing grasp. Additional info: {'target object': 'apple'}

Desktop (please complete the following information):

• OS: Ubuntu 24.04.1 LTS
• Isaac Sim Version (see docker)
• OmniGibson Version 1.1.1 (docker)

Additional context
The behavior is non-deterministic and different attempts are printed depending on state at failure.

[hang-yin]

Hi @f1ren , thanks for reaching out!

Since the action primitives use sampling-based motion planning, it is expected that the robot behavior can be non-deterministic. This primitives feature is also somewhat experimental and needs a lot more tuning. We are actively working on making this more robust and experimenting with alternative motion planners. But for now, you can try the following to make the primitives more fault-tolerant:

In omnigibson/examples/action_primitives/rs_int_example.py, you can look for controller.apply_ref(StarterSemanticActionPrimitiveSet.GRASP, apple) and controller.apply_ref(StarterSemanticActionPrimitiveSet.PLACE_ON_TOP, cabinet). This apply_ref function has an attempts argument, which defaults to 3. You can try to make this higher either 1) directly in the example you are running or 2) make the default higher in omnigibson/action_primitives/stater_semantic_action_primitives.py::StarterSemanticActionPrimitives::apply_ref.

We do have some really exciting progress, stay tuned for our next few releases!

[f1ren]

Thanks @hang-yin , setting attempts to 10 indeed seems to improve the results :)

Do you have a quick tip for resetting the planner state? It fails if I reset the env.
I'm trying to run it in a for loop, but the planner fails after a reset. I guess it holds the state before the reset.

[hang-yin]

Hi @f1ren , that's a good point! We don't currently have a reset method implemented (@cgokmen can confirm), but one hacky, and potentially slow way of doing this is you can manually create a new StarterSemanticActionPrimitives object after each environment reset. Improving the usability and robustness of these primitives is on our recent roadmap though, will probably have a better iteration of it in the upcoming months.

[f1ren]

Thanks for the tip @hang-yin , however, re-instantiation of StarterSemanticActionPrimitives
after env.reset() yields the same results.
Do you happen to know if there's any state that is stored somewhere else? Perhaps in the OMPL side?

Thanks again!

[hang-yin]

@cgokmen have you seen this type of behavior before?