[Question] How to get the 3d reconstruction at inference time?

[chisarie]

[Question] How to get the 3d reconstruction at inference time?

[Steve-Tod]

Hi, thanks for your interest in our work! The composed_scene here is the ground truth mesh and the added grasps. It's for visualization. If you want to get the 3D reconstruction, you can refer to this line: https://github.com/UT-Austin-RPL/GIGA/blob/d67c4388d334babe6c11c1555a6e848fb4828c84/scripts/eval_geometry_voxel.py#LL80C32-L80C32.

[chisarie]

Thank you for the fast response. What is the argument pc_in exactly? Is it the raw pointcloud from the camera, including all objects and the table they are sitting on?
I am trying to do inference from a depth camera I have in real life, not from the simulation

[Steve-Tod]

Hi, we crop the raw point cloud to remove the table surface and only keep the objects.

GIGA/scripts/construct_dataset_parallel.py

Line 34 in d67c438

pc = pc.crop(bounding_box)

[chisarie]

Hi, so I think I managed to put together a working script. Could you please let me know if this (pseudocode) looks correct? Thank you very much in advance.

from vgn.detection_implicit import VGNImplicit  # GIGA
from vgn.perception import TSDFVolume, create_tsdf
from vgn.ConvONets.conv_onet.generation import Generator3D

O_RESOLUTION = 40
O_SIZE = 0.3  # Meters --> I saw this somewhere in the repo
O_VOXEL_SIZE = O_SIZE / O_RESOLUTION

giga_model = VGNImplicit(
    model_path=model_path,
     model_type="giga",
     visualize=vis,
     best=True,
     qual_th=0.75,
     force_detection=True,
     out_th=0.1,
     select_top=False,
)
generator = Generator3D(
    self.giga_model.net,
    device=self.giga_model.device,
    input_type='pointcloud',
    padding=0,
)
tsdf_volume = create_tsdf(
    size=O_SIZE,
    resolution=O_RESOLUTION,
    depth_imgs=np.expand_dims(depth, axis=0),
    intrinsic=self.camera_intrinsic,
    extrinsics=np.expand_dims(camera_pose, axis=0),
)

pc_torch = torch.tensor(tsdf_volume.get_grid())
pred_mesh, _ = generator.generate_mesh({'inputs': pc_torch})

# scale back to original size
scale_matrix = np.eye(4)
scale_matrix[:3, :3] *= O_SIZE
pred_mesh.apply_transform(scale_matrix)

[Steve-Tod]

Yes, I think it's correct. I just re-ran the code and found input to generate_mesh is with size torch.Size([1, 40, 40, 40]), so it's a tsdf grid.

[chisarie]

Great, thank you!