This repository contains the framework using in our paper called: "Nonlinear Model Predictive Control for Deep Neural Network-Based Collision Avoidance exploiting Depth Images", submitted to ICRA 2024. The associated (temporary) video can be seen at https://youtu.be/aELjlfwAjfk.
First, install:
Other Python requirements are listed in requirements.txt
and can be installed with pip
:
pip install -r requirements.txt
Clone the repo and navigate to main folder. Then, the package can be install using pip
:
pip install -e .
NOTE: For convenience, the training data base folder is defined in colpred_nmpc/__init__.py
, as the variable COLPREDMPC_DATA_DIR
.
For retraining on your own dataset, please update this global variable or set the correct path in the training/testing scripts.
The repo folder is organized as follow:
- colpred_nmpc: Main module folder, including Neural Nets, Casadi+Acados based controller, and ROS wrapper.
- config: Yaml files for missions configurations and robot parameters.
- gazebo: A handful of Gazebo world files used in simulations. NOTE: spawns robot module which uses mrsim-gazebo for simulations using GenoM. Should be replaced/removed for using with another simulation pipeline.
- logs: Contains execution logs and Neural Nets weight files. This is where the scripts are looking for weights by default.
- scripts: Scripts folder, including NN training/testing and main ROS controller (
nmpc_ros.py
).
The training dataset and neural network weights will be uploaded soon.
Use depth_state_check_train.py
for training the Neural Network.
The class Colpred
encompasses the network definition as well as convenience functions.
The script depth_state_check_train.py
allows quantitative and qualitative evaluation of the trained network.
In particular, it allows to display classfication results (2D slices, or full 3D frustrum volume).
The Neural NMPC controller is defined in controller.py
, and the corresponging ROS interface is defined in ros_wrapper.py
.
The main script nmpc_ros.py
encapsulates the controller initialization, and the actual control loop.
Following ROS services are defined:
- startstop: starts the control loop
- goto: sets the desired waypoint relative to current pose, according the
p_des
in the yaml files. - toggle_colpred: enable collision avoidance objective and constraint?
NOTE: When using ml-casadi, PyTorch builds the graph on first execution. Thus, the first call(s) to the CasADi function will be slow. The main script thus processes a fake image once to prevent the first control iteration to be slow.
We would like to acknowledge Mihir Kulkarni for the code snippets, in particular for inflate_images.py
and collision_checker.py
.
You can contact us for any question: