rayvburn/srpb

A source code and the toolkit accompanying the paper "Quantitative Metrics for Benchmarking Human-Aware Robot Navigation"

srpb

The repository contains the source code of the SRPB - Social Robot Planners Benchmark - a benchmark that allows for quantitative evaluation of robot navigation performance and social aspects.

The corresponding package that allows logging data while the robot is navigating (using ROS1 navigation stack) is available at srpb_move_base.

A detailed description of metrics computation formulas is presented in the article Quantitative metrics for benchmarking human-aware robot navigation.

If you are using this benchmark in your research, please cite it as:

@article{karwowski2023quantitative,
  author={Karwowski, Jarosław and Szynkiewicz, Wojciech},
  journal={IEEE Access},
  title={Quantitative Metrics for Benchmarking Human-Aware Robot Navigation},
  year={2023},
  volume={11},
  number={},
  pages={79941-79953},
  doi={10.1109/ACCESS.2023.3299178}
}

Installation

Follow the steps below to clone SRPB-related packages:

cd <WS_DIR>/src
git clone --recurse-submodules https://github.com/rayvburn/srpb.git -b melodic-devel srpb/srpb
rosinstall -n . srpb/srpb/srpb.rosinstall

Usage

A log file is saved once the goal is reached by the srpb_move_base node. Renewing the goal pose before reaching the previous one does not cause the files to be divided into parts.

TBD...

Acknowledgments

The foundation of this package is MRPB: Mobile Robot Local Planning Benchmark. There might be some shared sections of code, but overall, the original package has undergone a major overhaul.

Contributing

Feel free to share your ideas, suggestions in Issues. Contributing to the code development is also appreciated.

srpb's Issues

update READMEs

print out all available parameters in metric evaluation executable

prepare some fancy table with results

e.g., https://stackoverflow.com/questions/9206669/making-a-table-with-printf-in-c

Computational efficiency of a global path planner

Currently, only the local trajectory planner can be evaluated with $m_{cef}$.

reimplement `Oscillations`, `Backward Movements`, `In-Place Rotations` metrics and make them orthogonal

`logger` - modernize file operations

use modern C++

add the package version note in the file with metric results

To indicate with which software version the results were obtained

fix `OpenCV` linking to `srpb_tooling`

Currently, srpb_move_base explicitly depends on OpenCV_LIBRARIES, see srpb_move_base/CMakeLists.txt

investigate validity of perception-related metrics

Check whether trajectories of some planners produces higher rate of human tracking abilities, i.e., humans were detected in more timestamps than with other planners.

Add a metric evaluating the robot speed and distance maintained around the humans

Ref: https://link.springer.com/content/pdf/10.1007/s12369-021-00805-6.pdf

Approximation implemented in social_nav_utils::PassingSpeedComfort

add ROS1 Noetic compatibility

Create a separate branch with integration adjustments.

Requires updating the srpb_move_base too.

add calculation of a smallest distance to an obstacle in `m_obs`

`logger` - transform poses to a common frame

With the current testing setup it is not necessary as poses are expressed in odom frame but with other perception frameworks it may not hold.

Covariances should also be transformed.

add goal reached metric

check if at t_N distance to goal was below the threshold

add `Computational Time Repeatability` metric

investigate whether some allocation issues exist

First time similar error occurred during usage of EBand evaluation on a real robot. Then, with TEB in a simulation (only once):

[ WARN] [1678061991.479280799, 23.167000000]: TebLocalPlannerROS: trajectory is not feasible. Resetting planner...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
[move_base-23] process has died [pid 19117, exit code -6, cmd (...)
[move_base-23] restarting process
process[move_base-23]: started with pid [20314]

consider implementing a custom `PersonData` class in the `logger` namespace

The aim is to create an abstraction layer from the people_msgs_utils::Person type (but maintaining all its features).
The PersonData class must be later propagated to all human-aware metrics.

make metrics included in `create_latex_table_from_results` script configurable

Currently, hard-coded in create_latex_table by metric_latex_map dict.

Export metric parameters file in `evaluation`

Similarly to results file:

srpb/srpb_evaluation/src/utils.cpp

Line 121 in 1d099fe

void createResultsFile(

Creating a LaTeX table with the parameters could also be handy.

rename subproject from `srpb_postprocessing` to `srpb_evaluation`

`evaluation` scripts - create a shared file with the table placement in the Excel sheet

Load the row and a column from a file into scripts that rely on the hard-coded table placement (create_excel_from_results.py, create_latex_table_from_results.py)

add ROS2 compatibility

consider loading params for `evaluation` from YAML file

single cmd line argument to the executable instead of multiple ones

prepare CI

Ref:

• https://github.com/GuillaumeFalourd/clone-github-repo-action/blob/main/.github/workflows/private-repo.yml

activity diagram for `Rewinder` class