iglict/neudf

The code for "NeUDF: Leaning Neural Unsigned Distance Fields with Volume Rendering (CVPR 2023)"

NeUDF

We present an open surface reconstruction method using UDF, called NeUDF, for reconstructing objects with high-fidelity open surfaces from 2D image inputs.

Project page | Paper

This is the official repo for the implementation of NeUDF: Leaning Neural Unsigned Distance Fields with Volume Rendering (CVPR 2023).

Usage

Setup environment

Installing the requirements using:

pip install -r requirements.txt

To compile MeshUDF to extract open mesh from the learned UDF field, please run:

cd custom_mc
python setup.py build_ext --inplace
cd ..

To use PyMeshLab and a customized Screened poisson to extract open mesh from the learned UDF field, please run:

pip install pymeshlab

To build PyMeshLab from source, please refer to PyMeshLab.

Running

• Train without mask

python exp_runner.py --mode train --conf ./confs/womask_open.conf --case <case_name>

• Train with mask

python exp_runner.py --mode train --conf ./confs/wmask_open.conf --case <case_name>

• Extract surface using MeshUDF

python exp_runner.py --mode validate_mesh_udf --conf <config_file> --case <case_name> --is_continue

The corresponding mesh can be found in exp/<case_name>/<exp_name>/meshes/mu<iter_steps>.ply.

• Extract surface using Screened Poisson

python exp_runner.py --mode validate_mesh_spsr --conf <config_file> --case <case_name> --is_continue

The corresponding mesh can be found in exp/<case_name>/<exp_name>/meshes/spsr<iter_steps>.ply.

• Extract surface using MarchingCubes

python exp_runner.py --mode validate_mesh --conf <config_file> --case <case_name> --is_continue

The corresponding mesh can be found in exp/<case_name>/<exp_name>/meshes/<iter_steps>.ply.

Evaluation using pretrained models

• Without mask setting

python exp_runner.py --mode validate_mesh_udf --conf ./confs/womask_open.conf --case my_rack_test --is_continue --resolution 512

• With mask setting

python exp_runner.py --mode validate_mesh_udf --conf ./confs/wmask_open.conf --case my_whale_test --is_continue --resolution 512

Datasets and results

You can download the full datasets and results here and put them in ./public_data/ and ./exp/, respectively.

The data is organized as follows:

public_data
|-- <case_name>
    |-- cameras_xxx.npz
    |-- image
        |-- 000.png
        |-- 001.png
        ...
    |-- mask
        |-- 000.png
        |-- 001.png
        ...
exp
|-- <case_name>
    |-- <conf_name>
        |-- checkpoints
            |-- ckpt_400000.pth

Train NeUDF with custom data

Please refer to the Data Conversion in NeuS.

Citation

Cite as below if you find this repository is helpful to your project:

@inproceedings{Liu23NeUDF,
    author = {Liu, Yu-Tao and Wang, Li and Yang, Jie and Chen, Weikai and Meng, Xiaoxu and Yang, Bo and Gao, Lin},
    title = {NeUDF: Leaning Neural Unsigned Distance Fields with Volume Rendering},
    booktitle={Computer Vision and Pattern Recognition (CVPR)},
    year = {2023},
}

Acknowledgement

Some code snippets are borrowed from NeuS, NDF and MeshUDF. Thanks for these great projects.

neudf's Issues

Leaning UDF or Learning UDF?

The title of paper is "Leaning Neural UDF", rather than "Learning Neural UDF". Do you have some special design for choosing this word leaning to express some tendency? Or is that a typo?

How to solve error _marching_cubes_lewiner_cy

Hi,
Thank you for your work.

I got an error when run training
No module named '_marching_cubes_lewiner_cy'

I found in custom_mc, but did not see.

How to solve this?
Thank you

Code Release

Thanks for your excellent work!
When should we expect the code release to have a go?
Thank you :)

RuntimeError: CUDA error: device-side assert triggered

Thanks for your excellent work!
But when I run the command python exp_runner.py --mode train --conf ./confs/wmask_open.conf --case real_capture_fan I got the error:
/opt/conda/conda-bld/pytorch_1614378124864/work/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [0,0,0], thread: [0,0,0] Assertion input_val >= zero && input_val <= onefailed. …… Traceback (most recent call last): File "exp_runner.py", line 934, in <module> runner.train() File "exp_runner.py", line 204, in train loss.backward() File "/home/zxy/.conda/envs/neus/lib/python3.7/site-packages/torch/tensor.py", line 245, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/zxy/.conda/envs/neus/lib/python3.7/site-packages/torch/autograd/__init__.py", line 147, in backward allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag RuntimeError: CUDA error: device-side assert triggered
Is this a problem with the cuda device? What parameters can I adjust if I want to get it running?

Training Code Issues

Congratulations and thanks for such a great work！
I found that for the DTU dataset (probably other datasets as well?), the norm of normals may be 0 when calculating the normal error for supervising, which causes the code stop running, so I add the following line:
norm = torch.where(torch.eq(norm,0.0), torch.tensor(1.), norm)
in the render_core function in ./models/renderer.py to prevent from such issues and the code works now.
But I'm not sure that the modification of code is correct, so I just raise this issue.

Reconstructed mesh output for results in paper

Can you upload the reconstructed mesh outputs for the results displayed in the project page / paper ?