Why use NLT at all?

[google]

Why use NLT at all?

[xiumingzhang]

Thanks for your comments.

We made it clear that the model is per person, meaning that you won't be able to relight person B by training on person A. Many other projects aim to do "one/zero-shot relighting," as you called it.

For a synthetic scene like the dragon Blender file, sure, you can render whatever you want with the 2 MB .obj file, similar to how you can render AAA game-level synthetic scenes from any viewpoint under any illumination, but that's not the point of the paper. The point here is to interpolate lights and views for a person (whose reflectance is hard to specify by hand, as opposed to the Blender dragon that has just a simple BRDF) when given an OLAT capture that includes an imperfect shape proxy (as opposed to the perfect dragon geometry). We couldn't release our human dataset due to privacy issues; the dragon dataset is just an example people can follow if they want to adapt our code for their own data. To summarize, if you have a synthetic scene over which you have complete control in every aspect (shape, reflectance, illumination), then you are right about just rendering it with Blender. But when you have only a shape proxy (with artifacts from scanning) and some simplified reflectance model, and you throw them into Blender, you will end up with artifacts due to imperfect geometry (e.g., bumps) and reflectance (e.g., missing SSS unless you specify that manually to your reflectance model).

A concrete example of what I said above: Suppose you want to relight a specific static actor from any viewpoint. Given your experience in photogrammetry and rendering, you can "scan" the shape of this actor using MVS or similar techniques and also acquire some PBR maps by assuming certain reflectance properties. Because your "scanned" geometry is imperfect, and your reflectance models are simplified (up to what PBR maps you acquired and what effects these maps can represent), if you import this model into Blender and just render that, then you will end up with artifacts due to the imperfect shape and reflectance. On the contrary, NLT doesn't explicitly model BRDFs, is tolerant of geometric errors, and therefore may avoid the said artifacts.