Run multiple Autoware in Carla with Zenoh

[evshary]

Hi community,

We are working on autoware_carla_launch, a project to run multiple Autoware in the Carla simulator with the help of Zenoh. Sometimes, we want to experiment with multiple vehicles, especially fleet management and the interaction between vehicles, so it's important to run several vehicles in the same simulator. We found that Zenoh can help in this scenario.

While speaking to multiple Autoware, the most important thing is avoiding ROS 2 topic conflict. In traditional ways, we might need to rename the topic name or add the namespace, which is tedious sometimes. In our solution, we developed a zenoh_carla_bridge. The bridge will use Carla API to get/set some information and then connect to Zenoh Network. On the other side, zenoh-bridge-dds receives the Zenoh data and transforms it into the Autoware topic directly. The key point here is that we can use one of Zenoh's feature "scope" to separate identical ROS topics in different Autoware. That's why we need Zenoh here.

Let's show the demo. In our demo, we run Carla + zenoh_carla_bridge in one host, and two Autoware + zenoh-bridge-dds in another host. The ROS traffic packets in Autoware should not be outside the network, because we want to avoid ROS topic conflict and also reduce unnecessary topics exposed to the network. To simplify, we use different ROS_DOMAIN_ID for each Autoware + zenoh-bridge-dds. We'll use container + ROS_LOCALHOST_ONLY to limit the traffic in the future, but the concept is the same. You can see that two hosts are connected with the help of Zenoh.

The video above shows how we run two Autoware in Carla at the same time. You can see two vehicles detecting each other at 3'40''.

In the future, we want to connect vehicles with Zenoh. As we've already mentioned in the discussion "How to use Zenoh in Autoware", we think Zenoh can be used on external communication, no matter for FMS or V2X. Therefore, we want to use Zenoh to act as the communication layer between Autoware.

Special thanks to some community efforts:

• hatem-darweesh: Excellent works on CARLA Simulator + Autoware Universe Tutorials. I used the map he generated and also refer to some of his vehicle parameters.
• jerry73204: The zenoh_carla_bridge here is based on the Carla Rust API he provided. He also did some contributions to the zenoh_carla_bridge.

[VRichardJP]

The key point here is that we can use one of Zenoh's feature "scope" to separate identical ROS topics in different Autoware. That's why we need Zenoh here.

Wouldn't it be possible to reproduce the same behavior with ROS2 domain id?

[evshary]

Yes, it's possible. However, since ROS 2 can't run different ROS 2 domain ID in the same program, there are two ways to do that:

1. If your "carla-autoware bridge" is using ROS 2, you might need to run several "carla-autoware bridge" with different ROS 2 domain ID corresponding to different Autoware. I think it's annoying sometimes, and also I'm not sure about the performance of this architecture.
2. You can use DDS to implement "carla-autoware bridge" directly, the DDS role here is similar to Zenoh in my implementation. DDS can run several domain ID at the same time, so you don't need to run several "carla-autoware bridge".

The reasons I use Zenoh instead of DDS to implement the architecture are:

1. zenoh-bridge-dds can filter ROS 2 topic. That means I don't need to expose all the Autoware topics to the network. I can only expose topics related to the Carla simulator (Sensors & Control). This can decrease some unnecessary bandwidth in the network.
2. Zenoh network overhead is less than DDS, and it also can improve network performance.
3. If your simulator and Autoware are in different LANs, then DDS/ROS can't help you in that case. Zenoh is the better choice here. Maybe this case is not too common, but you might still have some similar network topology in your application, though.

Thank you for the questions. We hope to hear more feedback from the community!

[zeidk]

Would it be possible to have both vehicles in the same RViz window?

[evshary]

It's feasible, but you need to make some effort. First of all, you need to modify the topic name in different Autoware (Adding namespace or something else) to avoid topic conflict. Then you need to modify RViz plugin to be able to send goals to different vehicles. In my opinion, this scenario is a little like the fleet management system for the vehicles. However, while considering vehicles, I don't think we would use ROS 2 / DDS to manage vehicles, since ROS 2 / DDS can only work in LAN. So, I don't think it's reasonable to use RViz here. Maybe I misunderstand your scenario. Please correct me if I'm wrong.

[zeidk]

Thank you for your answer. I don't quite understand why DDS will be an issue here but I understand the topics issues. I am not using the 2D pose estimate or the 2D goal pose from RViz but I am publishing directly to the relevant topics for my work. I only need to see both vehicles in the same map in autoware. Thank you again for your feedback.

[evshary]

In your scenario, DDS won't be an issue since you don't use it as the fleet management system over the Internet. After changing the topic name, RViz should be able to show the different vehicle info.