knut0815/neural_renderer

"Neural 3D Mesh Renderer" (CVPR 2018) by H. Kato, Y. Ushiku, and T. Harada.

Neural 3D Mesh Renderer (CVPR 2018)

This is code for the paper Neural 3D Mesh Renderer by Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.

For more details, please visit project page.

This repository only contains the core component and simple examples. Related repositories are:

• Neural Renderer (this repository)
  • Single-image 3D mesh reconstruction
  • 2D-to-3D style transfer
  • 3D DeepDream

For PyTorch users

This code is written in Chainer. For PyTorch users, there are two options.

• Angjoo Kanazawa & Shubham Tulsiani provides PyTorch wrapper of our renderer used in their work "Learning Category-Specific Mesh Reconstruction from Image Collections" (ECCV 2018).
• Nikos Kolotouros provides PyTorch re-implementation of our renderer, which does not require installation of Chainer / CuPy.

I'm grateful to these researchers for writing and releasing their codes.

Installation

sudo python setup.py install

Running examples

python ./examples/example1.py
python ./examples/example2.py
python ./examples/example3.py
python ./examples/example4.py

Example 1: Drawing an object from multiple viewpoints

Example 2: Optimizing vertices

Transforming the silhouette of a teapot into a rectangle. The loss function is the difference between the rendered image and the reference image.

Reference image, optimization, and the result.

Example 3: Optimizing textures

Matching the color of a teapot with a reference image.

Reference image, result.

Example 4: Finding camera parameters

The derivative of images with respect to camera pose can be computed through this renderer. In this example the position of the camera is optimized by gradient descent.

From left to right: reference image, initial state, and optimization process.

FAQ

CPU implementation?

Currently, this code has no CPU implementation. Since CPU implementation would be probably too slow for practical usage, we do not plan to support CPU.

Python3 support?

Code in this repository is only for Python 2.x. PyTorch port by Nikos Kolotourosr, supports Python 3.x.

If you want to install neural renderer using Python 3, please add ./neural_renderer to $PYTHON_PATH temporarily as mentioned in issue #6. However, since we did not tested our code using Python 3, it might not work well.

Citation

@InProceedings{kato2018renderer
    title={Neural 3D Mesh Renderer},
    author={Kato, Hiroharu and Ushiku, Yoshitaka and Harada, Tatsuya},
    booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2018}
}