haian-jin / tensoir Goto Github PK
View Code? Open in Web Editor NEW[CVPR 2023] TensoIR: Tensorial Inverse Rendering
Home Page: https://haian-jin.github.io/TensoIR/
License: MIT License
[CVPR 2023] TensoIR: Tensorial Inverse Rendering
Home Page: https://haian-jin.github.io/TensoIR/
License: MIT License
Thank you first for the great work.
I tried the model on NeRFactor dataset (512x512 with low res. env. map). The results of the Lego scene make me confused.
Result of the standard Lego config, the result is blur and has an unexcepted 'bottom' part:
Result of the 'tricky' Lego config (enable the those three lines), the result looks better but still it has 1. tricky normal direction in some part, 2. noisy normal for the flat surface and 3. not correct surface reconstruction inside the digging bucket:
As can be seen, training with either standard lego config or the tricky one can not produce results as good as in the paper. One reason is that I trained it with a smaller image size (512x512, in the paper 800x800). Do you have any suggestions for getting a better geometry?
An amazing work and congrats! I have a question in the code and hope for a reply if you're convenient. In the file renderer.py
Line 37, the TensoIR model is warped by torch.enable_grad()
, so I wonder why we need the gradient here?
Hello,
thanks for your great work and for open sourcing your code. It is really a nice paper.
For my use case, I need to learn (and relight accurately) an outbounded scene (such as the ones in Mip-NeRF360, although with multiple lights). Have you tried running your algorithm on such scenes? Did it perform well, or should I look for other works ?
Thank you again for this insightful research project and for providing us with the source code.
I was trying to track down an issue related to glossy estimation. I did a simple test where I optimize a glossy scene and then render novel views by setting the roughness value (after volume rendering is done) such that every 800x800 pixels has a roughness of 0
. I tried other values but it seems that values lower than 0.5
cause issues. Below, I am showing results for different r
values:
I did an ablation for r=0.2
, the strange highlights seem to be coming from the directly lit specular component:
Do you know why there happens to be artefacts for r=0.2
; while r=0
looks very matte? Let me know if I overlooked something.
Hello, I am trying to experiment with applying TensoIR to an unbounded scene.
I started with trying the well-known "Garden" scene from the MipNeRF-360 dataset.
I'm not sure if this will be the only issue, but one problem that I have run into is the following, where it seems that the scene geometry is limited to a bounding box that is much smaller than the full scene. This also seems like it leads to incorrect optimization of the geometry/normals:
Do you have any ideas as to what I may do to resolve this bounding box issue and improve results?
Thanks very much!
Hi, thank you very much for making your work publicly available.
Unfortunately, while I was trying to reproduce your experimental results on synthetic hotdog, a strange artifact was observed in a specific area in the scene as shown in the pictures below.
I followed your instructions for setting up python environment and used "configs/single_light/hotdog.txt" configuration provided in the codebase. Running the experiment multiple times or setting a different random seed value doesn't seem to resolve this issue. So I was wondering if I can get some help here.
Thanks in advance!
Thank you very much for your excellent work. I get an error when I train the synthetic dataset for nerf:
The format of the nerf synthetic dataset is as follows:
Looking at the error, it seems to ask for a metadata.json file.
But when I look at the metadata.json file in the dataset you provided, it looks like HDR is required!
Is HDR a required input?
Hi, thank you for sharing the great work.
I want to run the TensoIR on our own dataset. The data format of our own data is similar to kitti.
Can you give me some advice?
Hello! Thanks for your great work. We are really impressed by reconstruction quality has achieved.
We had followed 'Generating your own synthetic dataset' for the code generating out new scenes with a single small cube, and try to reconstruct it by this project.
However we found it doesn't work and the reuslt is pretty blurry. We had made the backgroud transparency like the armadillo in TensoIR-Synthetic.(using train_tensoIR.py )
After diving into the code about photos genreation and this project, we found the code to genreate new pictures with blender reamin some inconsistency with this project, especially generating camera params in the json file(camera_angle_x, rotation, rgba_xxsscene_000.png etc) I had fixed some, but it still remain some inconsistency.
Would you give me some advice about how to import my own scenes description into this project?(orientation of Z-axis, or is 'rotation' used in json? ) Or maybe could you share me you latest dataset generation code for blender?(the code in 'Generating your own synthetic dataset' dosen't seems to be latest)
Thanks!
(I am sorry maybe I am not a good English Speaker, if you feeled confused what I am talking about, maybe I can use Chinese
Thanks to your code!!
I am a student in another university and interesting in nerf and computer graphics and want to follow your work.But I have no zju email and can't regist.
Can you share the dataset by other ways?
Thanks!!
Hi,
I am a student with a keen interest in the topic at hand. While exploring your repository, I came across the "relight_importance.py" script and noticed the presence of a "compute_rescale_ratio" function.
It appears that this computed ratio is intended for the purpose of rescaling the diffuse color.
I am curious to understand the specific rationale behind this rescaling process. In my view, it might be possible to achieve the desired relighting results using the original, unaltered diffuse color.
Thanks for your amazing work and look forward to your reply.
Great job. I have seen that it is possible to export mesh, but I would like to know how to export texture maps so that they can be used in UE rendering engines effectively. Thanks a lot.
Hello, thank you for your excellent work.
I would like to add the SDF field to your work for further reconstruction. I used the geometric representation of neus while retaining the density tensor. I tried it on the 'hotdog' scene of the Blender dataset.
(1) The estimation of the environment map was found to be unreasonable, and the estimated albedo was severely darkened. Here is the environment map.
(2) Besides, the estimated RGB image (nvs with radiance field) seems to have grids, which is not smooth and continuous.
Could you give me some suggestions about it? Any reply will be appreciated.
Hi, thank you for this exciting paper.
I was looking to run the training on a custom blender scene. I used the utility files to render the full dataset from a user defined blender file. While I manage to train the full 80k steps on the single light armadillo experiment, I am unable to do so on a custom dataset because of a memory issue (gpu used is also 11GB). This happens at the 10k mark. From what I understand, 10k is when you start upsampling the vector and matrix sizes. Just after the 10k steps, the training fails with an out of memory error. What I don't understand is why this happens on my custom scene while the provide scene works correctly.
Is this related to the fact the alpha mask is being computed at that point?
Do you have any pointers as what would be causing this issue?
Thanks for your great work!
Will the code be released today? really looking forward to it!
Excellent work! Here is my question:
Is the radiance field color (computing by volume rendering) uesd as the indirect lighting, same as the radiance field color used in the loss L_RF? If so, Why does the indirect lighting need to be consistent with GT color?
I apologize for the possible error in my understanding. Looking forward to your answer and thank you for your excellent work!
could you provide Armadillo.blend
Thank you for your great work! Your method is truly impressive and very helpful to me. Can it directly export high-quality textured meshes?
Which setting using and what should I be careful?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.