Xueting Li Sifei Liu Kihwan Kim Shalini De Mello Varun Jampani Ming-Hsuan Yang Jan Kautz
NVIDIA UC Merced
ECCV 2020
Please cite our paper if you find this code useful for your research.
@inproceedings{umr2020,
title={Self-supervised Single-view 3D Reconstruction via Semantic Consistency},
author={Li, Xueting and Liu, Sifei and Kim, Kihwan and De Mello, Shalini and Jampani, Varun and Yang, Ming-Hsuan and Kautz, Jan},
booktitle={ECCV},
year={2020}
}
-
Download code & pre-trained model:
Git clone the code by:
git clone https://github.com/NVlabs/UMR $ROOTPATH
The pre-trained weights and semantic template can be downloaded from Google drive and unzip at
$ROOTPATH/UMR/
. -
Install packages:
Virtual environments are not required but highly recommended:
conda create -n umr python=3.6 source activate umr
Then run:
cd $ROOTPATH/UMR sh install.sh
Run the following command from the $ROOTPATH
directory:
python -m UMR.experiments.demo --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --img_path UMR/demo_imgs/birdie.jpg --out_path UMR/cachedir/demo
The results will be saved at out_path
in the above command.
Download CUB bird images by:
wget http://www.vision.caltech.edu/visipedia-data/CUB-200-2011/CUB_200_2011.tgz && tar -xf CUB_200_2011.tgz
Command for keypoint transfer evaluation using predicted texture flow:
python -m UMR.experiments.test_kp --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --split test --number_pairs 10000 --cub_cache_dir UMR/cachedir/cub/ --cub_dir CUB_200_2011/
Command for keypoint transfer evaluation using predicted camera pose:
python -m UMR.experiments.test_kp --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --split test --number_pairs 10000 --mode cam --cub_cache_dir UMR/cachedir/cub/ --cub_dir CUB_200_2011/
For keypoint transfer result visualization, add --visualize
. The visualizations will be saved at $ROOTPATH/UMR/cachedir/results_vis/
Command for evaluating mask IoU:
python -m UMR.experiments.test_iou --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --split test --cub_cache_dir UMR/cachedir/cub/ --cub_dir CUB_200_2011/ --batch_size 32
To train the full model with semantic consistency constraint:
-
Prepare SCOPS predictions following instructions here.
-
Download semantic template as described above if haven't done so.
-
Run the following command from the
$ROOTPATH
folder:python -m UMR.experiments.train_s2 --name=cub_s2 --batch_size 16 --cub_dir CUB_200_2011/ --cub_cache_dir UMR/cachedir/cub --scops_path SCOPS/results/cub/ITER_60000/train/dcrf_prob/ --stemp_path UMR/cachedir/cub/scops/
If the code is ran on multiple GPUs, add
--multi_gpu --gpu_num GPU_NUMBER
to the above commands.
If you wish to train from scratch (i.e. learn the semantic template from scratch):
-
Run the following command from the
$ROOTPATH
folder:python -m UMR.experiments.train_s1 --name=cub_s1 --gpu_num 4 --multi_gpu -cub_dir CUB_200_2011/ --cub_cache_dir UMR/cachedir/cub
-
Then compute the semantic template by:
python -m UMR.experiments.avg_uv --model_path UMR/cachedir/snapshots/cub_s1/pred_net_latest.pth --batch_size 16 --out_dir UMR/cachedir/snapshots/cub_s1 --cub_dir CUB_200_2011/ --cub_cache_dir UMR/cachedir/cub --scops_path SCOPS/results/cub/ITER_60000/train/dcrf_prob/
Now you can use the computed semantic template in the model training by setting
stemp_path
intrain_s2.py
toout_dir
in the above command.
Unless otherwise indicated, all code is Copyright (C) 2020 NVIDIA Corporation. All rights reserved. Licensed under the NVIDIA Source Code License-NC.
This code is built on CMR, CSM, project_skeleton. We modified the SoftRas for the proposed texture cycle consistency objective and the PerceptualSimilarity to support multiple GPU training. We included the modified codes in the external
folder for convenience. If you find these modules useful, please cite the corresponding paper.