tobias-kirschstein/nersemble

[Siggraph '23] NeRSemble: Neural Radiance Field Reconstruction of Human Heads

nersemble's Issues

How to do color correction in other sequences?

There is only color correction on the 10 sequences used in the paper while for the majority of other data, there is no such file.

I found in your paper that color correction involves a quite complex procedure, including face segmentation and optimal transport solving, which appears to be hard to reproduce.

I wonder if you could release (or point to) the code of doing such preprocessing, or release the correction data for the rest of the data?

Thank you very much.

Hardware Requirements

Hello,

I'm interested in running your code to reproduce the results presented in your paper. Could you please specify the hardware requirements? Specifically, I'm interested in the following:

• RAM requirements
• GPU memory (VRAM) requirements
• Any other hardware dependencies

Would it be possible to achieve the paper's results using an NVIDIA RTX 3090 or 4090?

Thanks!

How to train on custom dataset?

Hey, awesome work! Could you please release the code of data preprocess to train on custom dataset?

how to get the depth map

Thanks for your excellent work. How can I preprocess my videos if I want to train my own datasets. In your paper mentions that the depth map is calculated by colmap. I want to make sure if colmap is enough?

About camera coordinate system

Hi @tobias-kirschstein , I'm confused with the cam extrinsic and intrinsic.
The paper wrote:

We estimate an individual extrinsic and a shared intrinsic camera
matrix by employing a fine checkerboard in combination with a
bundle adjustment optimization procedure.

I use colmap to estimate extrinsic and intrinsic to reproduce reconstruct on other dataset, but I canot get correct params.
So I did an experiment on nersemble using colmap, and I got roughly same numerical value but positive and negative differs a lot.
I visualized the estimate results and it looks same with the cam array in the paper, so I suggest that might be a coordinate system problem.

Colmap outputs should be opencv, what's the coordinate system in NeRSemble?

Thank you.

Pretrained Model

Thank you for sharing your code. I just wanted to know if you have pre-trained weight for nersemble. Thank you. Please do share it if you have a pre-trained model.

Resume Training from Checkpoint

Is there functionality to load a previous model checkpoint to continue training?

Matting masks for free part?

Thank you for your excellent work. Could you please provide the alpha maps for the free parts? All emotional segments, with the exception of this particular part, already have their alpha maps for background removal. In case they are not available, could we obtain the background images for each camera?

Best regards,

DataManager compatible Extraction script

Hey Tobias,

First of all, I want to congratulate you on your awesome work ...
I am wondering if you can share the script which turns the default distribution of the dataset (i.e., mp4 files without extracted .png to the multi-view images, colmap and alpha) to the version that is supported by your datamanger?

As an other idea, I think it would be nice to being able to extract the images and alpha maps on the fly from the mp4 files directly instead of storing them ... let me know if I can help you with this

Thank you
BR

Coordinate System of Camera Matrix

Hi authors, in your dataset, the camera extrinsics are stored in camera_params.json. Could you share which coordinate system are those matrices in? Are they in OpenGL or OpenCV or Pytorch3D style?

Expressions Transfer

Great job with this - how would you suggest doing expression transfer between 2 different rendered human heads?

Extract .ply or .obj file

Can we extract .ply or .obj (any 3d model)?

Training Requirement

Hello, this is really a fascinating job.
I have a question. I understand that this work requires at least A6000 48GB for training, but I currently only have a few 24GB 3090 graphics cards. Can I use distributed training to solve this problem? If so, can you give me some guidance?