zju3dv/autodecomp

3D object discovery from casual object captures

AutoDecomp: 3D object discovery from casual object captures

This is the coarse decomposition part of the method proposed in AutoRecon: Automated 3D Object Discovery and Reconstruction. It can be used to preprocess a casual capture (object-centric multi-view images or a video) which estimate the camera poses with SfM and localize the salient foreground object for further reconstruction.

Install

Please install AutoDecomp following INSTALL.md.

Inference

Inference with demo data

Here we takes assets/custom_data_example/co3d_chair as an example. You can run automatic foreground scene decomposition with: scripts/run_pipeline_demo_low-res.sh. You should get a similar visualization as in assets/custom_data_example/co3d_chair/vis_decomposition.html.

You can take the data structure and the script as a reference to run the pipeline on your own data.

Inference with CO3D data

1. Download the demo data from Google Drive and put them under data/.
2. Run one of the script in scripts/test_pipeline_co3d_manual-poses/cvpr (use low-res images for feature matching and DINO features) or scripts/test_pipeline_co3d_manual-poses (use high-res images for feature matching and DINO features) to run the inference pipeline.
3. We save camera poses, decomposition results and visualization to path_to_the_instance/auto-deocomp_sfm-transformer.

Inference with annotated data in the IDR format

We also support import camera poses saved in the IDR format and localize the foreground object. You can run one of the script in scripts/test_pipeline_bmvs/cvpr or scripts/test_pipeline_bmvs for reference.

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{wang2023autorecon,
  title={AutoRecon: Automated 3D Object Discovery and Reconstruction},
  author={Wang, Yuang and He, Xingyi and Peng, Sida and Lin, Haotong and Bao, Hujun and Zhou, Xiaowei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21382--21391},
  year={2023}
}

autodecomp's Issues

ncut segmentation

Hi,

Thank you for the great project! I see that there is only inference_transformer in the config folder. Do you plan to publish inference_ncut and your ncut segmentation algorithm to generate the training data?

pycolmap version?

Thanks for your great job! I met some problems about pycolmap when running your code, could you offer some help? When I run
python third_party/AutoDecomp/auto_decomp/cli/inference_transformer.py --config-name=cvpr data_root=$DATA_ROOT inst_rel_dir=$INST_REL_DIR sparse_recon.n_images=40 sparse_recon.force_rerun=$FORCE_RERUN sparse_recon.n_feature_workers=1 sparse_recon.n_recon_workers=1 triangulation.force_rerun=$FORCE_RERUN triangulation.n_feature_workers=1 triangulation.n_recon_workers=1 dino_feature.force_extract=$FORCE_RERUN dino_feature.n_workers=1. An error occurred like following

pycolmap version is 0.5.0.

I have some problems when running the demo

Hello! Many thanks for this great project!

When I run the scripts/run_pipeline_demo_low-res.sh，a terrible error was encountered, as

(extract_and_match pid=6843) /home/xiaosongwei/anaconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/nn/functional.py:4227: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
(extract_and_match pid=6843) warnings.warn(
2%|▎ | 1/40 [00:01<00:53, 1.38s/it]
18%|█▊ | 7/40 [00:01<00:05, 6.25it/s]
32%|███▎ | 13/40 [00:01<00:02, 12.35it/s]
48%|████▊ | 19/40 [00:01<00:01, 19.00it/s]
62%|██████▎ | 25/40 [00:01<00:00, 25.75it/s]
78%|███████▊ | 31/40 [00:01<00:00, 32.29it/s]
92%|█████████▎| 37/40 [00:02<00:00, 38.07it/s]
2023-08-14 09:41:13,717 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::FeatureActor.run() (pid=6688, ip=10.96.33.83, actor_id=129ef46931ba20e25c757a0801000000, repr=<auto_decomp.sfm.sfm.FeatureActor object at 0x7f79fc0fe2e0>)
File "/home/xiaosongwei/AutoDecomp/auto_decomp/sfm/sfm.py", line 312, in run
_ = ray.get(obj_refs)
ray.exceptions.RayTaskError(SyntaxError):

Can you give me some solutions?