Question of custom data

[zju3dv]

Question of custom data

[dendenxu]

Hi! Thank you for your feedback!

Actually, we've only implemented code to run Animatable NeRF on fitted SMPL and ground truth images. I haven't tested whether using other model works under current pipeline. Although at first glance CAPE and SMPL have similar topology, thus it would probably work without too much pain (and probably improve the reconstruction since CAPE fits clothed human better).

prepare_lbs_meta.py will prepare input files under the lbs folder of the corresponding dataset. And as the script's name suggests, it will prepare metadata to perform LBS on the fitted SMPL of the dataset in question. The outputs weights.npy, joints.npy, tvertices.npy and bigpose_vertices.npy correspond to the SMPL model fitted to the data instead of the dataset itself.

I'm not sure I fully understand what your problem is so it would be great if you elaborate on what do you mean by:

But I find the result of get_tpose_blend_weights() seems not right

From the image you posted I can see that the posed vertices actually contains pose-dependent deformation (Notice the bulging on the elbows). Thus when you unpose the posed mesh, you should also remove the pose-dependent deformation of CAPE to get the original canonical mesh. In Animtable NeRF we simply ignore the pose-dependent effect of SMPL (When converting SMPL from canonical to world space, we only apply LBS without pose-dependent deformation), so this phenomenon of bulging elbow is kind of expected. I would advise using the provided GT canonical mesh and pose it to world space without pose-denpendent effect (you'll notice some bulging or contraction on the posed mesh).

We're planning on improving documentation about preparing your custom dataset but I haven't been able to find time for that.

[tpzou]

I see... So the data param processing is correct, right?

[dendenxu]

Looks good enough to me. But you should be aware of the caveats mentioned in my previous comment #47 (comment) when proceeding.

[tpzou]

Hi, @dendenxu ， sorry to bother you again. I'm confused about the z_vals in the code. Whether it refers to the distance along the ray or the distance on the z-coordinate axis？Its performance in depth_map and tpose_nerf_network.py looks contradictory.

[dendenxu]

@tpzou No big deal. z_vals means distance along the ray (consider world coordinates). We called it z_vals (and use it to compute the depth map) because we consider depth as originating from the camera center, instead of the camera plane (perspective v.s. orthogonal projection).