Giter Site home page Giter Site logo

dytanvo's Introduction

DytanVO: Joint Refinement of Visual Odometry and Motion Segmentation in Dynamic Environments

DytanVO: Joint Refinement of Visual Odometry and Motion Segmentation in Dynamic Environments (ICRA 2023)
By Shihao Shen, Yilin Cai, Wenshan Wang, and Sebastian Scherer.

What's new.

  • 01-17-2023: Our paper has been accepted to ICRA 2023!

  • 01-05-2023: Clean up and upload the codebase for DytanVO. Pretrained weights and datasets are also ready.

  • 09-20-2022: Remove Dynamic Dense RGB-D SLAM with Learning-Based Visual Odometry. The repo will be used to release codebase for the most recent ICRA 2023 submission.

Introduction

DytanVO is a learning-based visual odometry (VO) based on its precursor, TartanVO. It is the first supervised learning-based VO method that deals with dynamic environments. It takes two consecutive monocular frames in real-time and predicts camera ego-motion in an iterative fashion. It achieves an average improvement of 27.7% over state-of-the-art VO solutions in real-world dynamic environments, and even performs competitively among dynamic visual SLAM systems which optimize the trajectory on the backend. Experiments on plentiful unseen environments also demonstrate its generalizability.

Installation

We provide an environment file using anaconda. The code has been tested on an RTX 2080Ti with CUDA 11.4.

conda env create -f environment.yml
conda activate dytanvo

Compile DCNv2.

cd Network/rigidmask/networks/DCNv2/; python setup.py install; cd -

Models and Data

Pretrained weights

Download here and unzip it to the models folder.

KITTI dynamic sequences

Original sequences in KITTI Odometry are trimmed into sub-sequences which contain moving pedestrians, vehicles and cyclists so that VO's robustness to dynamic objects can be explicitly evaluated. Download DynaKITTI and unzip it to the data folder. Please cite this paper if you find it useful in your work.

AirDOS-Shibuya

Follow tartanair-shibuya and download it to the data folder.

(Optional) Scene Flow

One can also test the model on Scene Flow datasets, which was used to train both the VO and the segmentation networks. Scene Flow datasets have very challenging sequences with large areas of dynamic objects in image frames.

You can create symbolic links to wherever the datasets were downloaded in the data folder.

├── data
    ├── AirDOS_shibuya
        ├── RoadCrossing03
            ├── image_0
            ├── ...
            ├── gt_pose.txt
        ├── RoadCrossing04
        ├── ...
    ├── DynaKITTI
        ├── 00_1
            ├── image_2
            ├── ...
            ├── pose_left.txt
            ├── calib.txt
        ├── 01_0
        ├── ...
    ├── SceneFlow
        ├── FlyThings3D
            ├── frames_cleanpass
            ├── frames_finalpass
            ├── optical_flow
            ├── camera_data
        ├── Driving
        ├── Monkaa
    ├── ...

Evaluation

Create a folder to save output flow, segmentation, or poses.

mkdir results

Dynamic sequences in KITTI (loading the finetuned VO model at once)

traj=00_1
python -W ignore::UserWarning vo_trajectory_from_folder.py --vo-model-name vonet_ft.pkl  \
							   --seg-model-name segnet-kitti.pth  \
							   --kitti --kitti-intrinsics-file data/DynaKITTI/$traj/calib.txt  \
							   --test-dir data/DynaKITTI/$traj/image_2  \
							   --pose-file data/DynaKITTI/$traj/pose_left.txt 

AirDOS-Shibuya (loading FlowNet and PoseNet separately)

traj=RoadCrossing03
python -W ignore::UserWarning vo_trajectory_from_folder.py --flow-model-name flownet.pkl  \
							   --pose-model-name posenet.pkl  \
							   --seg-model segnet-sf.pth  \
							   --airdos  \
							   --test-dir data/AirDOS_shibuya/$traj/image_0  \
							   --pose-file data/AirDOS_shibuya/$traj/gt_pose.txt 

Scene Flow

img=Driving/frames_finalpass/15mm_focallength/scene_forwards/fast/left
pose=Driving/camera_data/15mm_focallength/scene_forwards/fast/camera_data.txt
python -W ignore::UserWarning vo_trajectory_from_folder.py --flow-model-name flownet.pkl  \
							   --pose-model-name posenet.pkl  \
							   --seg-model segnet-sf.pth  \
							   --sceneflow  \
							   --test-dir data/SceneFlow/$img  \
							   --pose-file data/SceneFlow/$pose

Add --save-flow tag to save intermediate optical flow outputs into the results folder.

Adjust the batch size and the worker number by --batch-size 10, --worker-num 5.

(Optional) Segmentation Mask Ground Truth

If your dataset has ground truth for camera motion, optical flow and disparity change across consecutive frames, we provide an example script to automatically generate ground truth of segmentation mask given these two modalities based on the pure geometry for the Scene Flow datasets.

python Datasets/segmask_gt.py --database data/SceneFlow --frames_pass clean --dataset FlyingThings3D

Add --debug flag to save visualizations of the generated masks.

Citation

If you find our code, paper or dataset useful, please cite

@inproceedings{shen2023dytanvo,
  title={Dytanvo: Joint refinement of visual odometry and motion segmentation in dynamic environments},
  author={Shen, Shihao and Cai, Yilin and Wang, Wenshan and Scherer, Sebastian},
  booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={4048--4055},
  year={2023},
  organization={IEEE}
}

Acknowledgement

We built DytanVO on top of TartanVO. We implemented the segmentation network by adapting rigidmask. We thank Gengshan Yang for his code and suggestions.

License

This software is BSD licensed.

Copyright (c) 2020, Carnegie Mellon University All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

dytanvo's People

Contributors

geniussh avatar kamilhabrych avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

dytanvo's Issues

output scale factor

When testing the network, you scaled the pose back (posenp = pose * self.pose_norm) but I don't know how to make it work while training. I tried to train using the normalized loss, but ATE diverges. I would really appreciate it if you could share the code snippet.

code

I'm very interested in your work. When will the code be open source?

Train Code

Thank you for your amazing work. When will train code be released?

Get inconsistent results with paper

I follow the instructions and successfully run DytanVO. But the ATE seems to be inconsistent with your paper (Table II). For sequence 07_1, I got ATE=1.6556. However, Table II in the paper gives ATE=0.6367.
I thought maybe the results should convert from NED coordinate to camera coordinate, so I slightly modified the code, but again got inconsistent results with ATE=1.5879 for 07_1.
Inconsistent results also occur in 01_1.
Then I found an helpful issue here: #18 , where the author said they used 136 frames of 07_1. However, I download DynaKITTI followed your instruction, find the 07_1 has 475 frames. Could you share which 136 frames you use ? The frame number of 04_0 and 08_0 also mismatch (04_0: 271 vs 36, 08_0: 46 vs 40).
In addition, the pretrained weight provided by the instruction does not have vonet_ft.pkl (only have vonet.pkl). Maybe I think is reason of the inconsistent problem.

Question about the ATE (m) of Droid-slam on KITTI

Thank you for your outstanding work!
While attempting to reproduce the results of Droid-slam on the KITTI through the provided pre-trained model, I have been unable to achieve results of comparable quality as yours. Could you share some evaluation details of droid-slam? Did you train it all over again on KITTI or just with the pre-trained model? Thanks a bunch!

downloading the DynaKITTI dataset

Thank you for your outstanding work, I have a problem that I would like to ask: I am downloading the DynaKITTI dataset and it always says that I can't get it from the website or I don't have permission to download it, I would like to get your help. Thank you very much.

track files

Could you please provide the track files obtained by testing AirDOS-Shibuya or KITTI data sets with VO methods, such as DeepVO, TrianFlow, CC?

Header file missing while compiling the DCN network

Hi,

It looks like one of the header files required for compiling the DCN network is missing. When I ran the compilation

cd Network/rigidmask/networks/DCNv2/; python setup.py install; cd -

I got the following error:

/mnt/c/Users/Randy/projects/DytanVO/Network/rigidmask/networks/DCNv2/DCN/src/cpu/dcn_v2_cpu.cpp:8:10: fatal error: TH/TH.h: No such file or directory
    8 | #include <TH/TH.h>
      |          ^~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
/mnt/c/Users/Randy/projects/DytanVO

Training code

Very insightful work! Can I get some train code please??

Questions about table 2 in the paper

Hello, I'd like to inquire about Table 2 in the paper. Could you please provide an explanation for the results of dynamic KITII sequences 00, 01, and 02? I've successfully generated the results for sequence 00 using the file labeled 00_1. However, for the dynamic KITII sequence 02, I've noticed three folders named 02_0, 02_1, and 02_2. Could you kindly guide me on how to obtain the results to populate Table 2?

Question about training code

DytanVO proposed a novel and remarkable solution to mitigate the negative infulence of dynamic elements. Do you have the plan to release the training code?

About the AirDOS-Shibuya dataset

I opened the connection and pulled to tartanair-shibuya, but I can't download the file safely. Even if the download is kept, it is still not possible.

Consulting for dataset Tartanair

Dear author,

I see that in the paper you mentioned that using Tartanair with only static scenes, I am not sure about it. Could you please share the dataset content or structure with us? Sincerely thank you!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.