Skip to main content

Running ROS 2 on Multiple Machines

ROS enables the operation of nodes on an individual robot or across multiple robots. In this guide, you'll discover how to set up ROS to function on several computers and robots. We'll explore two scenarios: when robots are within a local network and when they're connected via the Internet.

ros2 networking tutorial

Prerequisites​

To engage with the guides in this tutorial, you simply need two devices running ROS 2. One can be your personal laptop, while the other could be either a ROSbot XL, ROSbot 2R or ROSbot 2 PRO, all of which come with ROS 2 Humble pre-installed. It is also possible to solve the tutorial using the Gazebo simulation.

For the tutorial on Internet connections, we utilize the Husarnet VPN. While it streamlines and automates a significant portion of the setup, the functionalities of Husarnet highlighted in this context primarily center around facilitating ROS 2 communication over the Internet. However, this doesn't encompass all that Husarnet offers. Dive deeper into its capabilities by exploring the official Husarnet documentation and blog.

info

If you don't already own a ROSbot, you can get one from the Husarion store.

The tutorials are designed to flow sequentially and are best followed in order. If you're looking to quickly assess the content of a specific tutorial, refer to the ROS 2 Tutorials repository, which contains the completed outcomes.

Connecting ROS 2 in LAN​

The default settings in ROS 2 ensure that all devices within the same subnet are connected, provided that multicasting is activated. ROS 2 operates on a layer known as the "ROS Middleware" (RMW). Multiple RMW implementations exist, all based on the DDS (Data Distribution Service) connectivity pub-sub framework. This framework ensures seamless communication between various components of the ROS 2 nodes running in the same subnet.

1. Connect devices to the LAN

The first thing you need to do is connect them to the same LAN. You can check the address of your device with the command.

ROSbot
hostname -I

Then you can verify the connections. On second device type:

Laptop
ping <ROSBOT_IP>

2. Run example

Now, using the ROS 2 CLI, post a custom message of type String to the /msg topic every 1 second.

ROSbot
ros2 topic pub -r 1 /msg std_msgs/msg/String data:\ 'Hello, ROSbot here'
Laptop
ros2 topic echo /msg

You should see a greeting. It's amazing how quickly the result was achieved. 😃

While the default information exchange among all connected devices is convenient, it does come with a significant drawback: What if there's another robot on the network with which we would prefer not to share data? Fortunately, the ROS 2 developers anticipated this issue and have provided a straightforward solution.

Using ROS_DOMAIN_ID​

To ensure that only specific robot components communicate with each other on a shared network, DDS utilizes a unique code known as a "domain identifier." This mechanism manages communication and prevents potential miscommunications. To modify the domain ID, you need to set the ROS_DOMAIN_ID environment variable.

  1. First, set the ROS_DOMAIN_ID environment variable on the first device and start transmitting messages.
ROSbot
export ROS_DOMAIN_ID=1
ros2 topic pub -r 1 /msg std_msgs/msg/String data:\ 'Hello, ROSbot here'
  1. Next, check whether the laptop receives the message. Because of the domain code adjustment, the other device shouldn't intercept any messages now.
Laptop
ros2 topic list
  1. Finally, set the ROS_DOMAIN_ID on the second device to the same value, and then retrieve the messages.
Laptop
export ROS_DOMAIN_ID=1
ros2 topic echo /msg
caution
  1. You can safely set the ROS_DOMAIN_ID value in the range of 0 to 101, inclusive. For more details, click here.
  2. Keep in mind that configuring ROS_DOMAIN_ID is session-specific. If you launch a new terminal, you'll need to configure this variable once again.

Using ROS_DISCOVERY_SERVER​

Another DDS mechanism can segregate 'robot networks' within the same LAN. Instead of depending on the standard multicast-based LAN discovery, which is the default for DDS, you have the option to transition to the Discovery Server approach for DDS discovery. It's important to note that this feature is currently exclusive to FastDDS. FastDDS serves as the default RMW for ROS 2 versions like Foxy (LTS, 2020), Humble (LTS, 2022), and Iron (2023).

  1. First, run the discovery server on one of the devices.
ROSbot
fastdds discovery --server-id 0 --port 11888
  1. Link both the ROSbot and the laptop to the same discovery server. Since the server is initiated on the ROSbot, its LAN IP address is essential. Given that the ROSbot's IP address is 10.5.10.130, it is necessary to configure the ROS_DISCOVERY_SERVER environment variable on each device. This ensures that ROS 2 nodes utilize the Discovery Server instead of the multicast-based discovery method.
ROSbot
export ROS_DISCOVERY_SERVER="10.5.10.130:11888"
ros2 daemon stop # reload ROS 2 daemon

ros2 run demo_nodes_cpp talker
Laptop
export ROS_DISCOVERY_SERVER="10.5.10.130:11888"
ros2 daemon stop # reload ROS 2 daemon

ros2 run demo_nodes_cpp listener
caution

After you set the ROS_DISCOVERY_SERVER, you will notice that only two default topics are available.

husarion@rosbotxl:~$ ros2 topic list
/parameter_events
/rosout

From a user experience standpoint, it might seem like a flaw. However, when viewed from a network traffic perspective, it's an advantage: discovery traffic is only exchanged among participants who have common topics, either as publishers or subscribers. To view all ROS 2 topics across hosts connected to the same Discovery Server, you should opt for the SUPER_CLIENT configuration instead of using ROS_DISCOVERY_SERVER, which sets up the CLIENT configuration. To set up the SUPER_CLIENT config:

  1. Create the fastdds_supeclient.xml file with the following content:
Show me fastdds_supeclient.xml
<?xml version="1.0" encoding="UTF-8"?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <participant profile_name="super_client_profile" is_default_profile="true">
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SUPER_CLIENT</discoveryProtocol>
                        <discoveryServersList>
                            <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                                <metatrafficUnicastLocatorList>
                                    <locator>
                                        <udpv4>
                                            <address>10.5.10.130</address>
                                            <port>11888</port>
                                        </udpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                        </discoveryServersList>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>
  1. Setup the FastDDS default profile file environment variable to the path where the file is located:
export FASTRTPS_DEFAULT_PROFILES_FILE=$(pwd)/fastdds_supeclient.xml

ros2 topic list
# you should see all the topics

ros2 run demo_nodes_cpp listener
# ...and see messages published by the talker

Connecting ROS 2 via Internet​

As highlighted earlier, you can operate ROS nodes across numerous devices as long as they share the same LAN network. This enables communication between the ROS on your laptop and the robot when both are linked to the same Wi-Fi router. However, the aforementioned setup won't function if the robot is connected to the Internet through a different medium, such as an LTE module.

To address the connectivity issue, you could consider leveraging a VPN service. However, it's worth noting that conventional VPNs are not primarily designed for mobile robotics and can introduce certain challenges to your setup, including:

  • The need to manage a VPN server.
  • Encountering higher latency, as all traffic must route through the VPN server.
  • Extended reconfiguration times, especially when your robot switches between Wi-Fi hotspots.
  • The manual configuration requirement for DDS.

Husarnet is a specialized VPN service tailored for ROS 2. As an open-source peer-to-peer VPN, Husarnet facilitates direct robot connections over the Internet without the need for a centralized VPN server to relay data. The traffic is safeguarded with ephemeral encryption keys, and it's seamlessly compatible with ROS. Just set up the Husarnet Client on both your laptop and robot, link them via the same Husarnet network through the user-friendly web dashboard, and enjoy the benefits of swift, direct device-to-device communication.

To set up a free Husarnet account and connect your ROS 2-powered devices, simply follow these 5 easy steps:

Using Husarnet is easy

Create a free account at app.husarnet.com.

Husarnet Register

Manual Configuration (optional)
Manual configuration

If you require a configuration different from the default one supplied by the Husarnet-DDS utility, you will need to generate a custom XML configuration file. The following steps outline how to create this file for both Fast DDS and Cyclone DDS.

Installation

If the default DDS implementation used in you ROS 2 distribution is different than FastDDS, you need to install it:

sudo apt-get update
sudo apt-get install ros-$ROS_DISTRO-rmw-fastrtps-cpp
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp

Preparing a Configuration

Create a Fast DDS configuration file in your system: ~/fastdds_husarnet.xml. To make the communication work, you have to set some parameters as shown in the following template.

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <transport_descriptors>
        <transport_descriptor>
            <transport_id>HusarnetTransport</transport_id>
            <type>UDPv6</type>
        </transport_descriptor>
    </transport_descriptors>

    <participant profile_name="CustomHusarnetParticipant" is_default_profile="true">
        <rtps>
            <userTransports>
                <transport_id>HusarnetTransport</transport_id>
            </userTransports>
            <useBuiltinTransports>true</useBuiltinTransports>
            <builtin>
                <initialPeersList>
                <!-- Repeat this part for each husernet peer -->
                    <locator>
                        <udpv6>
                            <address>husarnet-IPv6-address</address> <!-- or <address>husarnet-hostname</address> -->
                        </udpv6>
                    </locator>
                <!-- End repeat -->
                </initialPeersList>
            </builtin>
        </rtps>
    </participant>
</profiles>

IMPORTANT: Provide all peers under initialPeersList tag.

Launching the Configuration

Set this file as default profile in your .bashrc file, so as to use this configuration every time you boot your system. Open a new terminal and execute:

echo "export RMW_IMPLEMENTATION=rmw_fastrtps_cpp" >> ~/.bashrc
echo "export FASTRTPS_DEFAULT_PROFILES_FILE=/home/$USER/fastdds_husarnet.xml" >> ~/.bashrc
. ~/.bashrc

Test connection

Post a message from the terminal using the familiar ros2 topic command. For example, on one computer you can type:

ROSbot
ros2 topic pub -r 1 /msg std_msgs/msg/String data:\ 'Husarnet is awesome! :D'

and use ros2 topic echo /msg to listen for incoming topic.

More about Husarnet VPN​

Husarnet is open source and free to use up to 5 devices. Here you can find many articles articles among which you will find information describing the principle of operation of Husarnet and many examples of using Husarnet in many applications (e.g. inside Docker).


Nice! At this point you should have your laptop and robot connected via LAN or Internet. Let's run some additional demo nodes to see how the connection works in practice.

Real-time video streaming​

In the first demo, the camera on one of the platforms will be launched. Thanks to the network connection, we can access any topic between devices. In this example, we will show you how to display the image from the camera on the second device.

ROSbot

  1. Configure ROS_DOMAIN_ID for LAN network, or use Husarnet VPN.

  2. Launch ROSbot:

docker compose up -d microros <rosbot_service> <camera_service>

Laptop

  1. Configure ROS_DOMAIN_ID for LAN network, or use Husarnet VPN.

  2. Check if connection works with:

ros2 topic list
  1. Run RViz2 to see a real-time video from the robot. Select topic and QoS policy to get image visualization.
rviz2

Reference projects​

Below we present a project that allows for the rapid exchange of information in the network by using image data compression, which reduces delays in image transmission by reducing the size of the data.

ProjectDescription
rosbot-telepresence
rosbot-xl-telepresence		Control the robot over the Internet with real-time camera feed

Summary​

And there you have it! At this point, you should have a solid understanding of how to manage robots connected via LAN or the Internet. For LAN-based setups, the primary focus is on determining the appropriate domain using ROS_DOMAIN_ID. When dealing with Internet connections, Husarnet, paired with the Husarnet-DDS utility (which automates DDS configuration), is a viable option, or you can opt to manually integrate a VPN with DDS. It's noteworthy that DDS is versatile in its configuration options. For further insights, delve into the Quality of Service, Husarnet documentation, and Husarnet blog. Having gone through this tutorial, you should be proficient in orchestrating multi-device interactions and skilled in transmitting and retrieving data from afar, empowering you to effectively command your robot.

by Rafał Górecki, Husarion

Do you need any support with completing this tutorial or have any difficulties with software or hardware? Feel free to describe your thoughts on our community forum: https://community.husarion.com or to contact our support: support@husarion.com