This repository contains executables and tools that allow to easily simulate arbitrary ROS 2 systems and then measures their performance. The system topology can be provided at runtime using JSON files or with command line options.
The framework tracks the following metrics:
- Latency
- Reliability
- CPU usage
- Memory usage
The core of the framework is entirely developed in C++ and it has no external dependencies beside the ROS 2 core libraries. This makes it very easy to be compiled and used on embedded platforms. The iRobot cross-compilation framework can be found at: https://github.com/irobot-ros/ros2-cross-compilation.
Note that this framework is mostly meant for evaluating single process applications. Although it is also able to measure the performance of multi-process applications, not all metrics will be available in this case.
The nodes under test currently don't perform any sort of computation while they are tested. This means that most of the measured resource usage is due to the overhead of ROS2 communication.
The only runtime requirement is ROS 2 rolling. The build machine requires Python 3, CMake and colcon.
mkdir -p ~/performance_ws/src
cd ~/performance_ws/src
git clone https://github.com/irobot-ros/ros2-performance
cd ros2-performance
git submodule update --init --recursive
cd ../..
colcon build
The irobot_benchmark package contains the main application and example of graph topologies for evaluation.
source ~/performance_ws/install/setup.bash
cd ~/performance_ws/install/irobot_benchmark/lib/irobot_benchmark
./irobot_benchmark topology/sierra_nevada.json
The results will be printed to screen and also saved in the directory ./sierra_nevada_log.
The irobot_benchmark/topology directory contains some examples of json files that can be used to define a system.
If you want to create your own JSON topology, follow the instructions on how to create a new topology. If you want to use your custom ROS 2 message interfaces in the topology, you should look at the performance_test_plugin_cmake.
- performance_test: this package provides the
performance_test::PerformanceNodeclass, which provides API for easily adding publishers, subscriptions, clients and services and monitor the performance of the communication. Moreover theperformance_test::Systemclass allows to start multiple nodes at the same time, while ensuring that they discover each other, and to monitor the performance of the whole system. Moreover, this pacakge contains scripts for visualizing the performance of applications. - performance_metrics: provides tools to measure and log various performance-related metrics in ROS 2 systems.
- performance_test_msgs: this package contains basic interface definitions that are directly used by the
performance_testpackage to measure performance. - performance_test_factory: this package provides the
performance_test_factory::TemplateFactoryclass that can be used to createperformance_test::PerformanceNodeobjects with specific publishers and subscriptions according to some arguments provided at runtime: this can be done through json files or command line options. The interfaces (msg and srv) that can be used in these nodes have to be defined in the so calledperformance_test_factory_plugins. - performance_test_plugin_cmake: this package provides the CMake function used to generate a factory plugin from interfaces definitions.
- irobot_interfaces_plugin: this package is a
performance_test_factory_pluginthat provides all the interfaces used in the iRobot system topologies. - irobot_benchmark: this package provides our main benchmark application. This executable can load one or multiple json topologies and it creates a ROS2 system running in a specific process from each of them. It also contains sample json topologies.
- composition_benchmark: this package contains applications and tools that can be used to profile ROS 2 composition concepts.
ApexAI provides an alternative valid performance evaluation framework, which allows to test different type of messages. Our implementation is inspired by their work.
- RealTime Working Group ReferenceSystem
- ROS2 communication performance evaluation
- ROS2 benchmarking framework
- ROS2 QOS throughput tests
- Evaluation of ROS2 Communication Layer
- ROS2 latency and throughput
- Evaluating QoS performances using ROS demos
- Exploring the performance of ROS2
- Time-Sensitive Networking for robotics
- Real-time Linux communications: an evaluation of the Linux communication stack for real-time robotic applications
- Towards a distributed and real-time framework for robots: Evaluation of ROS 2.0 communications for real-time robotic applications
- Time Synchronization in modular collaborative robots
- Reduce delay and jitter in wireless ROS1 networks
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent