pku-epic/graspnerf

[ICRA 2023] GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF

graspnerf's Issues

No texture and color in output images

I am very appreciate your work and run the demo file 'run_simgrasp.sh'. But images in 'rendered_results' only have grey and transparent objects. Do you know how to output images with color objects?

no moudle named torch

whatever GPU or pytorch i use,it

code releasing date

Hi!

Was wondering approximately when you're planning to release the code.

Thanks!

Unable to install pybullet==2.7.9 in Blender's bundled Python

Thank you for sharing such a nice work.

I cannot install pybullet==2.7.9. How do you install it and what version of python do you use?

scripts/sim_grasp.py is executed by Blender's bundled python in run_simgrasp.sh and binary file path for Blender's bundled python is /usr/local/blender-2.93.3-linux-x64/2.93/python/bin/python3.9.

$BLENDER_BIN $BLENDER_PROJ_PATH --background --python scripts/sim_grasp.py \

So I use Blender's bundled python (python3.9) but the following installation command does not work.

/usr/local/blender-2.93.3-linux-x64/2.93/python/bin/python3.9 -m pip install pybullet==2.7.9` 

A part of the error message during installation is as follows.

      In file included from src/btBulletCollisionAll.cpp:77:0:
      src/BulletCollision/CollisionShapes/btOptimizedBvh.cpp: In member function ‘void btOptimizedBvh::updateBvhNodes(btStridingMeshInterface*, int, int, int)’:
      src/BulletCollision/CollisionShapes/btOptimizedBvh.cpp:296:9: warning: ‘graphicsindex’ may be used uninitialized in this function [-Wmaybe-uninitialized]
           int graphicsindex;
               ^~~~~~~~~~~~~
      In file included from src/btBulletCollisionAll.cpp:85:0:
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp: In member function ‘bool btMiniSDF::load(const char*, int)’:
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:72:18: warning: ‘buf[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_cell_size[0] = buf[0];
         ~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:73:18: warning: ‘buf[1]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_cell_size[1] = buf[1];
         ~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:74:18: warning: ‘buf[2]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_cell_size[2] = buf[2];
         ~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:86:13: warning: ‘cells’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_n_cells = cells;
         ~~~~~~~~~~^~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:91:14: warning: ‘fields’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_n_fields = fields;
         ~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:143:22: warning: ‘n_cell_maps1’ may be used uninitialized in this function [-Wmaybe-uninitialized]
          for (int j = 0; j < n_cell_maps1; j++)
                          ~~^~~~~~~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:137:21: warning: ‘n_cell_maps0’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         for (int i = 0; i < n_cell_maps0; i++)
                         ~~^~~~~~~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:125:21: warning: ‘n_cells1’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         for (int j = 0; j < n_cells1; j++)
                         ~~^~~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:119:20: warning: ‘n_cells0’ may be used uninitialized in this function [-Wmaybe-uninitialized]
        for (int i = 0; i < n_cells0; i++)
                        ~~^~~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:108:15: warning: ‘n_nodes1’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         nodes.resize(n_nodes1);
         ~~~~~~~~~~~~^~~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:94:25: warning: ‘nodes0’ may be used uninitialized in this function [-Wmaybe-uninitialized]
        unsigned long long int nodes0;
                               ^~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:81:22: warning: ‘buf[2]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_inv_cell_size[2] = buf[2];
         ~~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:80:22: warning: ‘buf[1]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_inv_cell_size[1] = buf[1];
         ~~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:79:22: warning: ‘buf[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_inv_cell_size[0] = buf[0];
         ~~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:67:19: warning: ‘buf2[2]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_resolution[2] = buf2[2];
         ~~~~~~~~~~~~~~~~^~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:66:19: warning: ‘buf2[1]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_resolution[1] = buf2[1];
         ~~~~~~~~~~~~~~~~^~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:65:19: warning: ‘buf2[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_resolution[0] = buf2[0];
         ~~~~~~~~~~~~~~~~^~~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:59:21: warning: ‘buf[5]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_domain.m_max[2] = buf[5];
         ~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:58:21: warning: ‘buf[4]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_domain.m_max[1] = buf[4];
         ~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:57:21: warning: ‘buf[3]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_domain.m_max[0] = buf[3];
         ~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:55:21: warning: ‘buf[2]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_domain.m_min[2] = buf[2];
         ~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:54:21: warning: ‘buf[1]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_domain.m_min[1] = buf[1];
         ~~~~~~~~~~~~~~~~~~^~~~~~~~
      src/BulletCollision/CollisionShapes/btMiniSDF.cpp:53:21: warning: ‘buf[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         m_domain.m_min[0] = buf[0];
         ~~~~~~~~~~~~~~~~~~^~~~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pybullet
  Running setup.py clean for pybullet
Failed to build pybullet
ERROR: Could not build wheels for pybullet, which is required to install pyproject.toml-based projects

Thanks.

scenes tsdf depth normal generation

Hi,

This is a wonderful work, I am recently trying to train the GraspNeRF network. But facing an issue with scenes tsdf depth normal generation for train data.

There is no script to generate this data. can you guide on generating this data whether it uses single view depth images or mutli view images. How to generate or when can I expect the github to update with files to support training.

Thank you so much in advance. Appreciate your help!

finished

not found glibc2.18

I have a problem when I run the train.sh file, it shows that not found glibc2.18, can I compile the program with a lower version of glibc2.17 and then run it?
我在运行train.sh文件的时候显示我缺失glibc2.18,这是一个底层的编译库文件，但是我不太方便升级，麻烦问一下用低版本的glibc2.17编译后可以运行程序吗？麻烦您了，期待您的答复

Franka Emika Panda

Hi!
Thank you for the work you've done!

I was wondering if you will be adding any instructions about how to use your code with Panda?
I'm doing a project using a Panda arm robot and I'm a newbie on working with a robot.
I suppose you used ROS for the experiment?

Thanks!

realsense holder

Hi!
Thanks for sharing your work!

Do you have a 3d file for a realsense holder by any chance?

Torch error during training and testing

Hello,
A really impressive contribution.
I am running the code on an RTX A4000
I use Torch 2.1 with cuda 12.1
after launching both

bash run_simgrasp.sh

or

bash train.sh GPU_ID

I get the following error

File "/home/user/miniconda3/envs/GRASP/lib/python3.9/site-packages/torch/init.py", line 235, in
from torch._C import * # noqa: F403
ImportError: /home/user/miniconda3/envs/GRASP/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: cblas_gemm_bf16bf16f32

Do you have an idea, what can I do to fix that ?
Thanks in advance

Testing custom images

Hi! Thanks for sharing your work!

I'm trying to use custom images and output grasping information (using your model's pretrained weights). This is my first time reading research paper repo so I'm not sure which script I should look at and how to run it to achieve my goal.
I've seen other papers you mentioned (VGA and GAGA).

I'll use Panda eventually to grasp objects but for now I just want to use your model with my custom images without using Panda and output grasping information.

Am I supposed to use simulation experiment script? or robotics experiment script?
I'm a bit confused because in simulation experiment I think the model is taking pictures of a simulated scene. And in robot experiment, I have to use Panda to take pictures and use them as input.

Could you briefly explain the process to achieve my goal?

Sorry for asking basic questions!

Simulating grasp using pybullet and blender

Hi. Thanks for your wonderful work and sharing the code. It benefits me a lot. It would be highly appreciated if you could provide more information about simulating grasp using pybullet and blender. How to use your scripts to generate simulation scenes and grasps? Appreciate your help!

How to use it in real life？

Great job!
I want to directly judge the grab points through photos from some angles. How should I do this?

Grasping TSDF on custom images

Hi,

Thanks for sharing your wonderful work!
I'm looking to generate TSDF and grasp samples for custom images!

Could you point me to the relevant script!
Thanks

Which version of the packages should i use ,Can anyone share your runable settings?

torch
tensorflow
easydict
inplace-abn
plyfile
numpy
scikit-image
pyyaml
h5py
opencv-python
tqdm
matplotlib
scipy
lpips
transforms3d
kornia
sklearn
catkin_pkg
black
jupyterlab
pandas
mpi4py
open3d
pybullet==2.7.9
pytorch-ignite
tensorboard

Blender version?

Hi there,
Cool project!

I was wondering what version of blender you guys used?

Thanks :)

Does there exists dictionary file utils?

Can the simulation work in windows?

File not found error,anyone have the same problems?

FileNotFoundError: [Errno 2] No such file or directory: './log/20230915-063503/exp_results/0'
run_simgrasp.sh: line 22: blender: command not found

pile_full, packed_full and scenes_tdsf_dep-nor generation scripts for 100 scenes are not present to train the GraspNeRF network

Hi,

Thanks for your wonderful work and sharing the code.

The training script for graspnerf might be missing src/vgn, render_packed_STD_rand.py, scenes_tsdf_dep-nor_packed.py and scenes_tsdf_dep-nor_pile.py.

The loss convergence for the training script is not able to happen and giving poor results.

Appreciate your help!

bad TSDF volume result from example data

Hi, thank you for the great work.

I'm trying to output TSDF volume of the transparent object scene using pre-trained weights.
But the result seems not good compared to the ground truth one.

The details of the configuration are as follow:

downSample: 0.8
images: resized (512x288) images in "traindata_example/giga_hemisphere_train_demo/packed_full/packed_0-170/000fc0562d2a4881b24921a424ef9175/rgb" directory
extrinsics: "traindata_example/giga_hemisphere_train_demo/packed_full/packed_0-170/000fc0562d2a4881b24921a424ef9175/camera_pose.npy" file
intrinsics: downsized(K[:2] * 0.8 * 0.5) K (https://github.com/PKU-EPIC/GraspNeRF/blob/30e28971c95d3a9bacb351c5ab8a55dac3ed004d/src/nr/main.py#L107C49-L107C49)
depth range: fixed to [0.2, 0.8]
bbox3d: [[-0.15, -0.15, -0.0503], [0.15, 0.15, 0.2497]]
TSDF value thresholds: [0.0 (low), 1.0 (high)]
Model: NeuralRayRenderer
number of images: 6

Here is the visualization result of TSDF volume (red dots are camera origins)

This is from my code.

This is from "traindata_example/giga_hemisphere_train_demo/scenes_tsdf_dep-nor/000fc0562d2a4881b24921a424ef9175.npz" file.

Can you tell me which part I'm missing?
Appreciate your help!

no moudle named vgn.utils,anyone have the same problem?

May I know when the official open source release is approximately planned?