zju3dv/pvo

code for "PVO: Panoptic Visual Odometry", CVPR 2023

PVO: Panoptic Visual Odometry

Project Page | Paper

PVO: Panoptic Visual Odometry

[Weicai Ye, Xinyue Lan]^Co-Authors, Shuo Chen, Yuhang Ming, Xinyuan Yu, Hujun Bao, Zhaopeng Cui, Guofeng Zhang

CVPR 2023

Test on vkitti 15-deg-left datasets.

0. prepare. follow prepare.md

conda activate droidenv

1. generate inital panoptic segmentation.

sh tools/initial_segmentation.sh  

2. vps->vo，vo Module generate pose, flow and depth.

sh tools/test_vo_scene.sh  

3. vo->vps, vps Module use flow and depth from vo Module and generate final video panoptic segmentation results and vpq.

sh tools/test_vps.sh  

Metrics on Virtual_KITTI2

Scene	RMSE	vpq_all/vpq_thing/vpq_stuff
Scene01	0.371	40.39/26.43/44.57
Scene02	0.058	68.84/88.83/62.18
Scene06	0.113	66.38/79.99/62.97
Scene18	0.951	68.35/83.86/63.92
Scene20	3.503	35.11/16.83/40.59

You can get the results in the paper by iterating multiple times.

Train on vkitti 15-deg-left datasets.

1. To train VPS_Module, you can refer to Detectron2 for more training details. Here for example, you can train vkitti 15-deg-left on 4 GPUs, and training results are saved on output/vps_training/. You can modify the config-file according to the hardware conditions.

python3 -W ignore VPS_Module/tools/train_net.py \
--config-file VPS_Module/configs/COCO-PanopticSegmentation/panoptic_fpn_R_50_3x_vkitti_511.yaml --num-gpu 4 \
MODEL.WEIGHTS checkpoints/panFPN.pth \
OUTPUT_DIR output/vps_training/

2. To train VO_Module, you can refer to DROID-SLAM for more training details. Here for example, you can train vkitti on 4 GPUs.

python VO_Module/train.py --gpus=4 --lr=0.00025

Visualization

You can refer to DROID-SLAM for visualization. All demos can be run on a GPU with 11G of memory. While running, press the "s" key to increase the filtering threshold (= more points) and "a" to decrease the filtering threshold (= fewer points).

python VO_Module/evaluation_scripts/test_vo.py --datapath=datasets/Virtual_KITTI2/Scene01 --segm_filter True 

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{Ye2023PVO,
  title={{PVO: Panoptic visual odometry}},
  author={Ye, Weicai and Lan, Xinyue and Chen, Shuo and Ming, Yuhang and Yu, Xingyuan and Bao, Hujun and Cui, Zhaopeng and Zhang, Guofeng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9579--9589},
  year={2023}
}

Acknowledgement

Some code snippets are borrowed from DROID-SLAM and Detectron2. Thanks for these great projects.

pvo's Issues

Got stuck at 'solving environment'

Dear @ywcmaike,

Thanks for your work!

I encountered a problem when deploying the conda environment. After running conda env create -f environment.yaml, the process got stuck at solving the environment for several hours. I wonder if you have any proposition. For your information, the code was run on linux.

Yours,
X. Shen

VPS Training dataset problem

I am trying to train a VPS module using config file from guide (VPS_Module/configs/COCO-PanopticSegmentation/panoptic_fpn_R_50_3x_vkitti_511.yaml). And it contains

DATASETS:
  TRAIN: ("vkitti_511_train",)
  TEST: ("vkitti_511_val",)

But VPS_Module/detectron2/data/datasets/others.py doesn't contain vkitti_511_train

def register_others(root):
    register_vkitti_511_val()
    register_vkitti_clone()
    register_vkitti_test()

How i can fix this problem? thx

Definition of dynamic objects

Hello, thanks for your excellent work!
I'd like to know how the Panoptic-Enhanced VO Module detects the dynamic objects? It seems paper didn't illustrated on this part.

Code release

@zehongs thanks for share the wonderful work , can u please let us knw when is the code base shall be release along with weights

Did we ever use segmentation masks in training?

I noticed the line was removed:
weight = self.do_filter(lay, weight, delta_dy, delta_m, segments)
In driod_net.py, I couldn't find other functions which used seg masks.
Should we uncomment this line to enable segmentation masks?

Installation Issue for VO_Module

I am facing an issue where running "python setup.py install" in the VO_Module doesn't seem to be working and exits with an error.

The first error is:
=> => # /PVO/VO_Module/thirdparty/eigen/Eigen/SparseCore:11:10: fatal error: Core: No such file or directory
=> => # 11 | #include "Core"
=> => # | ^~~~~~
=> => # compilation terminated.

Then I get:
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1. (I installed ninja and still getting this issue.)
RuntimeError: Error compiling objects for extension

Can you look into this and give some advice on how to resolve this? I will send more detailed logs if required.

P.S. I have been able to install the VPS_Module without any issues.

AttributeError: module 'numpy' has no attribute 'range'

为啥我的numpy没有range函数呢
PVO/VO_Module/droid_slam
/droid_net.py

里面有一个 lay = torch.as_tensor(np.range(1, num+1).repeat(ht*wd).reshape(num,ht,wd)).unsqueeze(0)
会让我报错。

我试着直接写再我的python里试图调用numpy.range，确实不太work

这是我跑到sh tools/test_vo_scene.sh 的时候报的错误。之后还报了若干个类似

Traceback (most recent call last):
  File "VO_Module/evaluation_scripts/test_vo2.py", line 93, in <module>
    for i_batch, item in enumerate(train_loader):
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/xbh/.local/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 302, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/home/xbh/TIM/PVO/PVO/VO_Module/droid_slam/data_readers/base.py", line 201, in __getitem__
    mask, v = self.__class__.dymask_read(dymask_list[i])
IndexError: list index out of range

这样的错误

About ego pose of VKITTI2

Thanks a lot for your wonderful work! I want to find how to get ego pose in VKITTI2 dataset, firstly I use “extrinsics.txt” as own car’s pose, but when I try to accumulate the point cloud(derived from depth) using that pose, it dosen’t look right, so can you help me with that? Thanks a lot!

No module named 'VO_Module'

python tools/datasets/generate_dynamic_masks.py
Traceback (most recent call last):
File "/home/sy/pvo/tools/datasets/generate_dynamic_masks.py", line 10, in
from VO_Module.droid_slam.data_readers.factory import dataset_factory
ModuleNotFoundError: No module named 'VO_Module'

In generate_ Dynamic_ In masks.py
from VO_ Module. droid_ slam.data_ readers.factory import dataset_ factory

Set VO_ Module for root and add_ Init_ Py. It's useless

IndexError: list index out of range

python tools/datasets/generate_dynamic_masks.py
['Scene18', 'Scene20', 'Scene06', 'Scene02', 'Scene01']
cache_path: /media/happy507/DataSpace1/sy/pvo/VO_Module/droid_slam/data_readers/cache/VKitti2.pickle
Building VKitti2 dataset
0%| | 0/5 [00:00<?, ?it/s]True
True
True
True
True
100%|█████████████████████████████████████| 5/5 [00:00<00:00, 53.51it/s]
Reserving datasets/Virtual_KITTI2/Scene02 for validation
Reserving datasets/Virtual_KITTI2/Scene06 for validation
Reserving datasets/Virtual_KITTI2/Scene18 for validation
Reserving datasets/Virtual_KITTI2/Scene20 for validation
Dataset vkitti2 has 445 images
Segments List (Flow): []
i (Flow): 0
Segments List (Flow): []
i (Flow): 0
Segments List (Flow): []
i (Flow): 1
Segments List (Flow): []
i (Flow): 1
Segments List (Flow): []
i (Flow): 2
Traceback (most recent call last):
File "tools/datasets/generate_dynamic_masks.py", line 32, in
for i_batch, item in enumerate(train_loader):
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/_utils.py", line 434, in reraise
raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/shaying/.virtualenvs/droid/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 308, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/media/happy507/DataSpace1/sy/pvo/VO_Module/droid_slam/data_readers/base.py", line 196, in getitem
seg = self.class.segment_read(segments_list[i])
IndexError: list index out of range
在运行python tools/datasets/generate_dynamic_masks.py时报错，谁能帮帮我？

Asking for use of segment results

Dear author,

when I want to implement the code, I have read it and find that in the file VO_Module/droid_slam/droid_net.py, in Class DroidNet forward, there's no use of the Panoptic segmentation results. Could you please tell me how the segmentation results can be adopted for the panotic-aware dynamic masks?

IndexError: list index out of range

I try to run generate_dynamic_masks.py
and it looks like the segment fileis missing
seg = self.__class__.segment_read(segments_list[i])

我装了好久的环境，环境已经能在droid-slam上正常运行，也配置了panopticapi-0.1这个包。
我debug了一下，这里越界好像是因为没有加载到segment的数据，但是我并没有找这个数据在哪里。。。
请问一下这里segment应该在哪里生成呢？

(droidslam) lwy@lwy-WS:~/research/PVO/tools/datasets$ python generate_dynamic_masks.py 
['Scene18', 'Scene20', 'Scene06', 'Scene02', 'Scene01']
cache_path:  /home/lwy/research/PVO/VO_Module/droid_slam/data_readers/cache/VKitti2.pickle
Building VKitti2 dataset
  0%|                                                                                                               | 0/5 [00:00<?, ?it/s]True
True
True
True
True
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 122.42it/s]
Reserving /home/lwy/research/PVO/datasets/Virtual_KITTI2/Scene02 for validation
Reserving /home/lwy/research/PVO/datasets/Virtual_KITTI2/Scene06 for validation
Reserving /home/lwy/research/PVO/datasets/Virtual_KITTI2/Scene18 for validation
Reserving /home/lwy/research/PVO/datasets/Virtual_KITTI2/Scene20 for validation
Dataset vkitti2 has 445 images
Traceback (most recent call last):
  File "generate_dynamic_masks.py", line 28, in <module>
    for i_batch, item in enumerate(train_loader):
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/lwy/miniconda3/envs/droidslam/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 308, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "../../VO_Module/droid_slam/data_readers/base.py", line 185, in __getitem__
    seg = self.__class__.segment_read(segments_list[i])
IndexError: list index out of range

Asking for data_readers

Dear author,

When I run python tools/datasets/generate_dynamic_masks.py, I find there no module named data_readers, Do you know why? Maybe there's a lost for the data_readers file(cause after pip install data_readers this problem still happens)?

Kind regards

dymask_list为空

Traceback (most recent call last):
File "/mnt/share_disk/lvnuo/LN/code/pvo/VO_Module/evaluation_scripts/test_vo2.py", line 93, in
for i_batch, item in enumerate(train_loader):
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/_utils.py", line 434, in reraise
raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/envs/pvoenv/lib/python3.9/site-packages/torch/utils/data/dataset.py", line 308, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/mnt/share_disk/lvnuo/LN/code/pvo/VO_Module/droid_slam/data_readers/base.py", line 189, in getitem
mask, v = self.class.dymask_read(dymask_list[i])
IndexError: list index out of range

我在运行test_vo_scene.sh时报如下的错误，请帮帮我

How to train the PVO model on kitti dataset?

There is no data reading part during VPS training

There is no training mode for reading data in VPS mode. Can you provide the following information