Multi PC AWSIM + Autoware Tests

[xmfcx]

I have made 7 tests using AWSIM + Autoware.

System Configuration

Specs

Autoware PC

CPU: 5900X (12c24t)
GPU: RTX 3090 (24GB VRAM)
Memory: 64GB

Simulation PC

CPU: 5900X (12c24t)
GPU: RTX 4080 (16GB VRAM)
Memory: 64GB

Connectivity info

• Both PCs had https://www.tp-link.com/us/home-networking/pci-adapter/tx401/ 10 Gigabit PCI Express Network Adapter and connected with a 5m CAT7 cable.
• ROS2 DDS settings were tuned to only use this adapter.
• Both PCs had DDS settings tuned to support high bandwidth traffic.
  /etc/sysctl.d/10-cyclone-max.conf file:

  net.core.rmem_max=2147483647
  net.ipv4.ipfrag_time=3
  net.ipv4.ipfrag_high_thresh=134217728 # (128 MB)

  cyclonedds.xml file:

  <?xml version="1.0" encoding="UTF-8" ?>
  <CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
      <Domain Id="any">
          <General>
              <Interfaces>
                  <NetworkInterface autodetermine="false" name="enp36s0" priority="default" multicast="default" />
              </Interfaces>
              <AllowMulticast>default</AllowMulticast>
              <MaxMessageSize>65500B</MaxMessageSize>
          </General>
          <Internal>
              <SocketReceiveBufferSize min="10MB"/>
              <Watermarks>
                  <WhcHigh>500kB</WhcHigh>
              </Watermarks>
          </Internal>
      </Domain>
  </CycloneDDS>

• iperf tests were made and saw 9.21 Gbits/sec transfer rates.

Time synchronization

• Used chrony for time sync.
• Autoware PC /etc/chrony/chrony.conf file (Main PC)
• AWSIM PC /etc/chrony/chrony.conf file (Client PC)
• Achieved ~50us time synchronization, shown in the video recordings.

Additional info

• AWSIM Branch: tier4/AWSIM@b5f5910
• Used following exact commit hashes: https://gist.github.com/xmfcx/ab20652891b2ac2cb00cb7095f71cadd (awsim-stable branch)
• Added https://assetstore.unity.com/packages/tools/gui/graphy-ultimate-fps-counter-stats-monitor-debugger-105778 to show the performance graphs
• Made the top lidar into VLS128 and reduced the horizontal res to 0.4 degrees to have better performance.

Tests

For each test, see the video descriptions for the time tags for easy navigation.

[xmfcx]

Test 1

https://www.youtube.com/watch?v=qHces72BmaI

00:00 - test
00:42 - ndt faulty init
00:49 - ndt manual reinit
1:07 - test 1 goal pose
1:23 - auto doesnt work
1:56 - test 1 start
3:32 - planning/perception reacts late
4:38 - extremely slow lane change
5:18 - stops in the middle of the road
5:32 - blocks the road because of the empty walkway
6:52 - test 1 end
7:27 - test 2 goal pose
7:47 - test 2 start
8:37 - crashes into other vehicle from side, stops after the crash
10:15 - mission planner doesn't work for lane change, failure ending

[mehmetdogru]

cc: @xmfcx @TakaHoribe @mitsudome-r

I have created some scenarios for now to kind of initiate the discussions:

Repository commit hashes for the videos:

autoware: 0e5ec31
autoware.universe: 33f62b43431ba18c353f2df09e7243ca71040969
autoware_launch: 405e5fbe3a21446d57959faf01b76432a79a3743

---

3:32 - planning/perception reacts late

• Scenario
• Map (nishishinjuku)
• Video
• Explanation:
  • The issue seems here is that NPC is not stopping for the traffic light. Or traffic light is green for NPC while at the same time it is green for us. So it creates a problem. To simulate the issue I didn't consider the traffic light for the scenario. Needs to be analysed.

---

5:18 - stops in the middle of the road
5:32 - blocks the road because of the empty walkway

• Scenario
• Map (nishishinjuku)
• Video
• Explanation:
  • I couldn't reproduce the problem "stops in the middle of the road". Probably about the obstacles around us. Needs to be evaluated further.
  • To walkways by default we are stopping at the moment. Needs to be discussed if necessary.

---

10:15 - mission planner doesn't work for lane change, failure ending

• Scenario
• Map (nishishinjuku)
• Video
• Explanation:
  • In my case, vehicle is getting stuck with the information lane_change_right. Probably behavior_path_planner is not able to perform a lane change to the right neighbor lane so it gets stuck before the intersection. Needs further investigation.

[xmfcx]

Test 2

https://www.youtube.com/watch?v=nS32hayC2i8

0:00 - test
0:39 - ndt faulty init
0:44 - ndt reinit
1:34 - test 1 goal pose
1:42 - test 1 start
4:08 - test 1 end
4:33 - test 2 goal pose
4:52 - test 2 start
6:30 - perception/planning reacts late to the car in path
6:55 - traffic jam failure ending

[xmfcx]

Test 3

https://www.youtube.com/watch?v=BPVMzYocuUA

0:00 - test
0:36 - ndt faulty init
0:41 - ndt reinit
1:07 - test 1 goal pose
1:31 - test 1 start
3:22 - test 1 end
4:58 - test 2 goal pose
5:27 - test 2 start
6:53 - surround obstacle blocks the way (neighbor vehicle)
7:25 - mission planner can't plan lane change, failure ending

[xmfcx]

Test 4

https://www.youtube.com/watch?v=5IZJ4yZiBEw

0:00 - test
0:37 - ndt init
1:01 - test 1 goal pose
1:15 - test 1 start
4:56 - perception or planning reacts late
7:31 - indecisive stopping on intersection
8:20 - test 1 end
10:25 - test 2 goal pose
10:35 - test 2 start
11:55 - npc van clips through ego car
12:33 - traffic jam failure ending

[xmfcx]

Test 5

https://www.youtube.com/watch?v=ed181rcJR5k

0:00 - test
0:14 - test 1 goal pose
0:22 - test 1 start
1:31 - perception/planning reacts late to the vehicle
2:18 - test 1 end
2:57 - test 2 goal pose
3:11 - test 2 start
3:44 - ndt fails maybe
4:09 - vehicle falls off the map, failure ending

[xmfcx]

Test 6

https://www.youtube.com/watch?v=WID6m6z18Gc

0:00 - test
0:35 - ndt faulty init
0:40 - ndt reinit
1:12 - test 1 goal pose
1:23 - test 1 start
2:59 - extremely slow lane change
4:13 - waits in the middle of the road due to empty walkway
5:17 - mission planner failure ending

[xmfcx]

Test 7

https://www.youtube.com/watch?v=jiEt1uyHXJE

0:00 - test
0:47 - ndt failure init
0:55 - ndt reinit
1:14 - test 1 goal pose
1:28 - test 1 start
4:01 - test 1 end
4:13 - test 2 goal pose
4:19 - test 2 start
5:08 - reacts late to the vehicle
5:19 - after waiting, still waits extra for the intersection
6:13 - test 2 end
6:29 - test 3 goal pose
6:38 - test 3 start
6:50 - refuses to exit after the stopline, failure ending

[mehmetdogru]

@Autostone-c I am not very well informed about perception related topics but maybe cuda, cudnn or tensorrt versions are mismatching? Could you confirm the versions are as autoware supports?

And did you get any warning/error during build process of autoware on awsim-stable branch regarding version mismatching?

cc: @kaancolak @xmfcx

[xmfcx]

I see that you had:

1694400101.9227071 [component_container_mt-51] [TRT_COMMON][WARNING]: CUDA initialization failure with error: 35 at L168.

And then,

1694400102.1120832 [ERROR] [component_container_mt-51]: process has died [pid 7027, exit code -11, cmd '/data/cjf/crayon-masp/install/rclcpp_components/lib/rclcpp_components/component_container_mt --ros-args -r __node:=traffic_light_node_container -r __ns:=/perception/traffic_light_recognition -p use_sim_time:=True -p wheel_radius:=0.383 -p wheel_width:=0.235 -p wheel_base:=2.79 -p wheel_tread:=1.64 -p front_overhang:=1.0 -p rear_overhang:=1.1 -p left_overhang:=0.128 -p right_overhang:=0.128 -p vehicle_height:=2.5 -p max_steer_angle:=0.7'].

I'm not sure why that happened but you might want to isolate and debug that node. I don't have experience running Autoware in Jetson Orin. @miursh san do you have any idea?

[miursh]

Sorry I don't have any experience with Orin either.
@manato @dan-dnn Do you have any idea about this?

[miursh]

CUDA error35 looks like some kind of driver error.

Could you confirm that other ML packages or CUDA samples will work?

[Autostone-c]

i will test and verify it。

[mitsudome-r]

@xmfcx Just as mentioned in today's Planning / Control WG today, there could be some issues that might have been solved in the latest branch. I have created the tag for 2023.09.14 from the latest main branch, and confirmed that it works on AWSIM v1.1.0 release.
It might be better if you can use the tag if you plan to do future tests.

[mitsudome-r]

https://github.com/autowarefoundation/autoware/tree/2023.09.14