suyz526/zebrapose

The implementation of the paper 'ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation' (CVPR2022)

ZebraPose

The implementation of the paper 'ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation' (CVPR2022). ArXiv

System Requirement

Tested Environment

• Ubuntu 18.04
• CUDA 11.1
• Python 3.6

Main Dependencies:

• bop_toolkit
• Pytorch 1.10
• torchvision 0.11.0
• opencv-python
• Progressive-X

Download with git clone --recurse-submodules so that bop_toolkit will also be cloned.

Training with a dataset in BOP benchmark

Training data preparation

1. Download the dataset from BOP benchmark

2. Download required ground truth folders of zebrapose from owncloud. The folders are models_GT_color, XX_GT (e.g. train_real_GT and test_GT) and models (models is optional, only if you want to generate GT from scratch).

3. The expected data structure:

   .
   └── BOP ROOT PATH/
       ├── lmo   
       ├── ycbv/
       │   ├── models
       │   ├── models_eval
       │   ├── models_fine
       │   ├── test
       │   ├── train_pbr
       │   ├── train_real
       │   ├── ...               #(other files from BOP page)
       │   ├── models_GT_color   #(from last step)
       │   ├── train_pbr_GT      #(from last step)
       │   ├── train_real_GT     #(from last step)
       │   ├── test_GT           #(from last step)
       │   ├── train_pbr_GT_v2   #(from last step, for symmetry aware training)
       │   ├── train_real_GT_v2  #(from last step, for symmetry aware training)
       │   └── test_GT_v2        #(from last step, for symmetry aware training)
       └── tless
   

4. Download the 3 pretrained resnet, save them under zebrapose/pretrained_backbone/resnet, and download pretrained efficientnet from "https://download.pytorch.org/models/efficientnet_b4_rwightman-7eb33cd5.pth", save it under zebrapose/pretrained_backbone/efficientnet

5. (Optional) Instead of download the ground truth, you can also generate them from scratch, details in Generate_GT.md.

Training

Adjust the paths in the config files, and train the network with train.py, e.g.

python train.py --cfg config/config_BOP/lmo/exp_lmo_BOP.txt --obj_name ape

The script will save the last 3 checkpoints and the best checkpoint, as well as tensorboard log. To enable sym. aware training, with --sym_aware_training True

Test with trained model

For most datasets, a specific object occurs only once in a test images.

python test.py --cfg config/config_BOP/lmo/exp_lmo_BOP.txt --obj_name ape --ckpt_file path/to/the/best/checkpoint --ignore_bit 0 --eval_output_path path/to/save/the/evaluation/report

To use ICP for refinement, use --use_icp True

For datasets like tless, the number of a a specific object is unknown in the test stage.

python test_vivo.py --cfg config/config_BOP/tless/exp_tless_BOP.txt --ckpt_file path/to/the/best/checkpoint --ignore_bit 0 --obj_name obj01 --eval_output_path path/to/save/the/evaluation/report

To use ICP for refinement, use --use_icp True

Download our trained model from this link. The ProgressiveX can not set random seed in its python API. The ADD results can be +/- 0.5%.

Evaluate for BOP challange

Merge the .csv files generated in the last step using tools_for_BOP/merge_csv.py, e.g.

python merge_csv.py --input_dir /dir/to/pose_result_bop/lmo --output_fn zebrapose_lmo-test.csv

And then evaluate it according to bop_toolkit

Difference between ArXiv v1 and v2

The results were reported with the same checkpoints. We fixed a bug that only influence the inference results:

The PnP solver requires the Bbox size to calculate the 2D pixel location in the original image. We modified the Bbox size in the dataloader. The bug is that we didn't update this modification for the PnP solver. If you remove the get_final_Bbox in the dataloader, you will get the results reported in v1.

The bug has more influence if we resize the Bbox using crop_square_resize. After we fixed the bug, we used crop_square_resize for BOP challange (instead of crop_resize in the config files in config_paper). We think this resize method should work better since it will not introduce distortion. However, we didn't compare resize methods with experiments.

Acknowledgement

The original code has been developed together with Mahdi Saleh. Some code are adapted from Pix2Pose, SingleShotPose, GDR-Net, and Deeplabv3.

Citation

@article{su2022zebrapose,
  title={ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation},
  author={Su, Yongzhi and Saleh, Mahdi and Fetzer, Torben and Rambach, Jason and Navab, Nassir and Busam, Benjamin and Stricker, Didier and Tombari, Federico},
  journal={arXiv preprint arXiv:2203.09418},
  year={2022}
}

zebrapose's Issues

loss for predicted binary coding

Hi，

If I want to compare the entire mask to obtain a loss function, what part of the code do I need to modify? Or do I need to retrain the ResNet34 model?

if predict_entire_mask:
loss_m = maskLoss(pred_mask_prob, entire_masks)
else:
loss_m = maskLoss(pred_mask_prob, masks)

Best,
Joa

Download of Trained models

First, thanks for the wonderful paper. It was a pleasure to read and I look forward to working with the code to understand the approach better.

Can you kindly explain again how to download the pre-trained models? The README links to https://cloud.dfki.de/owncloud/index.php/s/EmQDWgd5ipbdw3E. For example the ape of lmo folder contains a version file, a data.pkl file, and a subfolder data which contains several files but it is not clear to me that the checkpoints are among them.

difference between bop version and paper version for YCB-V dataset

Hi,

Thanks again for sharing your great work! I am a bit confused about the bop version and paper version of YCB-V dataset. The configs looks the same, except the option bop_challange. Does bop_challange only affect the test data? Thanks!

visualization of pose estimation results

Hello, I would like to ask where the implementation for loss curves and visualization of pose estimation results is located?

when i try to run python scripts/vis_est_poses.py，it encountered this error.

I looked at the code, and there are options to choose between 'flat' and 'phong'. It seems that 'flat' may not require normals, but 'phong' appears to provide better visual effects.

How would you like to address this issue?

Error building "Generate_Mesh_with_GT_Color"

When building "Binary_Code_GT_Generator/Generate_Mesh_with_GT_Color" i received a lot of errors.

I fixed the issue by changing the required c++ version from "-std=c++11" to "-std=c++14" In this cmake file https://github.com/suyz526/ZebraPose/blob/main/Binary_Code_GT_Generator/Generate_Mesh_with_GT_Color/CMakeLists.txt

Is this correct?

Ask about your training time

I want to know how long to train one object, and do you train all object in total iteration 380000? And do you train on one GPU.

2D object detection

Hello, I want to ask Where can I obtain the JSON file of the 2D object detection results for GDRNPPDet?

Code for visulize_input_data_and_network?

I'm trying to use my own dataset, and i'm troubleshooting some training problems

I hacked together a way to visualize the input data as well as the masks, which works okay.

I noticed that the in train.py there is the following uncommented code.
#visulize_input_data_and_network(writer, train_loader, net)
Does anyone still have this code? The function doesn't exist in this repo

wget from DFKI cloud

Hi, thanks for your great work! I wonder how can I download files from DFKI cloud directly via commands like wget? Thanks

Run test.py, it raise No module named 'Render'

Hi, I want to run test.py with ycbv data, I use (resnet):

python test.py --cfg config/config_BOP/lmo/exp_ycbv_BOP.txt --obj_name banana --ckpt_file xxx/checkpoint --ignore_bit 0 --eval_output_path xxx/report

it will raise no module named 'Render' in ICP_Render.py line 1

experiment results: separate networks or unified?

Hi,

I have read your paper and the idea is really interesting! But I am not sure whether you train separate networks for each object (i.e. P.E.=N defined in GDR-Net) or train one unified network for all objects (i.e. P.E.=1). Could you help to clarify this? Thanks!

Also, in my understanding, the results of GDR-Net are P.E.=N while those of SO-Pose are P.E.=1.Since P.E.=N tends to outperform P.E.=1 for the same method, putting these results together without additional remarks may look a bit weird.

Obtaining models_GT_color for custom dataset

Hi,

Thank you for your work. I would like to try to train with my own dataset but I can't seem to understand what are the "Class_CorresPoint0000XX.txt". How do you obtain them and what do they represent?

Thank you for your response.

Will windows10 run your project?

Dear author, can windows10 run your project?

Does loss curves look right?

The following is the loss curve of my own dataset. Does it look right? it seems that binary code loss didn't decrease.

something about training dataset(LMO)

how many pics(come from PBR) do you use when training LMO dataset? all of them or the part that contain the object

Asking about the 'efficientnet_key'

Hi!
Thank you for your amazing work!
I have some questions about the parameter "efficientnet_key"， what is the purpose of this parameter? Is it to improve the model's inference time and reduce GPU memory usage? In your code, you have implemented two options, 'b5' and 'b4'. What do they represent respectively? Does using this parameter lead to a decrease in the accuracy of model inference?

visualization of pose estimation results

Hi, I try to run python scripts/vis_est_poses.py for the visualization of pose estimation results. It ran successfully, but the outputs images don't show the pose estimation:

(zebrapose) simoc@IITICB002LW012:~/ZebraPose/bop_toolkit$ python scripts/vis_est_poses.py 
3/1|13:06:11: Processing: zebrapose_lmo-test.csv
vispy uses app: egl, gl: gl2
3/1|13:06:12: Loading 3D model of object 1...
3/1|13:06:12: Loading 3D model of object 5...
3/1|13:06:12: Loading 3D model of object 6...
3/1|13:06:12: Loading 3D model of object 8...
3/1|13:06:12: Loading 3D model of object 9...
3/1|13:06:12: Loading 3D model of object 10...
3/1|13:06:12: Loading 3D model of object 11...
3/1|13:06:12: Loading 3D model of object 12...
3/1|13:06:12: Loading pose estimates...
3/1|13:06:12: Organizing pose estimates...
3/1|13:06:13: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 3
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
3/1|13:06:16: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 62
3/1|13:06:19: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 107
3/1|13:06:22: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 166
3/1|13:06:25: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 254
3/1|13:06:28: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 342
3/1|13:06:31: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 434
3/1|13:06:34: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 480
3/1|13:06:37: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 549
3/1|13:06:40: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 649
3/1|13:06:43: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 737
3/1|13:06:47: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 770
3/1|13:06:50: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 809
3/1|13:06:53: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 883
3/1|13:06:56: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 942
3/1|13:06:58: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 1038
3/1|13:07:02: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 1107
3/1|13:07:05: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 178
3/1|13:07:07: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 611
3/1|13:07:10: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 840
3/1|13:07:12: Done.

In this issue, I have seen that @hrkita produced some results, so, am I missing something?

Thanks,
Simone

about progressive-x

Hello,

I would like to ask why progressive-x is disabled during training and only enabled during testing?

Code ETA

Hi @suyz526 ,

do you have any estimate on when your code could be published?

Best,
D

module 'cv2' has no attribute 'ppf_match_3d_ICP'

Hi suyz526,

Thanks, thanks a lot work your great work. I have big hopes with our dataset for your project if I can make it work of course. I have been working for 3 days to make it work. Especially render module became a torture for me : )

Anyway, I am very close to see the refined positions on renderer but I am getting error like below

AttributeError: module 'cv2' has no attribute 'ppf_match_3d_ICP'

on code line 129 on icp_refiner_modified.py.

icp_fnc = cv.ppf_match_3d_ICP(n_iters, tolerence=tolerence, numLevels=4)

I am misssing something but I couldnt figure out.

I have installed the below modules:
pip install opencv-python==4.5.4.58
pip install opencv-contrib-python==4.5.4.60

but I am still having the same problem. cv2 cannot find surface matching module and ppf.

Do you have any suggestion for this problem?
thanks thanks in advance

release your rusults on other datasets

Do you prepare to release your results (eg. add-s) on other datasets like tless?

Not Getting The Same Results

Hi all, I was trying to re-run your baseline results using the provided checkpoints on the LMO dataset. What I found is that the results were closer to V1 of your paper. I even tried to change the resize methods in inference but it's not giving me the same results. Can someone help me on this?

Expected to get a single numpy array of shape (H,W) or (H,W,C)

Hello. I am trying out zebrapose and I'm having some issues when training on the example given in the readme.

I get this error

Neighborhood calculation time = 0.243874 secs.
Neighborhood calculation time = 0.249840 secs.
Neighborhood calculation time = 0.193178 secs.
Neighborhood calculation time = 0.205201 secs.
Neighborhood calculation time = 0.230796 secs.
Neighborhood calculation time = 0.317984 secs.
Neighborhood calculation time = 0.238225 secs.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [02:52<00:00, 7.83s/it]
ADD_passed 0.0
best_score 0.0
best check point saved in /home/robotlab/Desktop/HAOV/ZebraPose/6dof_pose_experiments/experiments/checkpoints/exp_lmo_BOPape/best_score/0_0000step0
exp_lmo_BOP iteration_step: 1 loss_b: 0.7003157009678205 loss_m: 0.4887707233428955 loss: 2.589717826246357
exp_lmo_BOP iteration_step: 2 loss_b: 0.69137206054109 loss_m: 0.3945087790489197 loss: 2.46862496067219
exp_lmo_BOP iteration_step: 3 loss_b: 0.694508611202345 loss_m: 0.31038543581962585 loss: 2.3939112694266607
exp_lmo_BOP iteration_step: 4 loss_b: 0.6951448561501329 loss_m: 0.2788199484348297 loss: 2.3642545168852283
exp_lmo_BOP iteration_step: 5 loss_b: 0.6913055153449651 loss_m: 0.22996273636817932 loss: 2.3038792824030745
exp_lmo_BOP iteration_step: 6 loss_b: 0.6898123773388913 loss_m: 0.25095099210739136 loss: 2.320388124124065
exp_lmo_BOP iteration_step: 7 loss_b: 0.6898932232178236 loss_m: 0.22556562721729279 loss: 2.295245296870764
exp_lmo_BOP iteration_step: 8 loss_b: 0.689580717264042 loss_m: 0.21235992014408112 loss: 2.281102071936207
exp_lmo_BOP iteration_step: 9 loss_b: 0.6850530220622502 loss_m: 0.20650124549865723 loss: 2.2616603116854077
exp_lmo_BOP iteration_step: 10 loss_b: 0.6861161840507828 loss_m: 0.19408346712589264 loss: 2.252432019278241
exp_lmo_BOP iteration_step: 11 loss_b: 0.685705219347128 loss_m: 0.19636832177639008 loss: 2.253483979817774
exp_lmo_BOP iteration_step: 12 loss_b: 0.6799169770450811 loss_m: 0.19241726398468018 loss: 2.2321681951199235
exp_lmo_BOP iteration_step: 13 loss_b: 0.6819430279221167 loss_m: 0.18748752772808075 loss: 2.233316611494431
exp_lmo_BOP iteration_step: 14 loss_b: 0.6840540645380042 loss_m: 0.18512609601020813 loss: 2.237288289624221
Traceback (most recent call last):
File "train.py", line 383, in
main(configs)
File "train.py", line 225, in main
for batch_idx, (data, entire_masks, masks, Rs, ts, Bboxes, class_code_images, cam_Ks) in enumerate(train_loader):
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/robotlab/Desktop/HAOV/ZebraPose/zebrapose/bop_dataset_pytorch.py", line 263, in getitem
x = self.apply_augmentation(x)
File "/home/robotlab/Desktop/HAOV/ZebraPose/zebrapose/bop_dataset_pytorch.py", line 351, in apply_augmentation
x = augmentations.augment_image(x)
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/imgaug/augmenters/meta.py", line 763, in augment_image
"Expected to get a single numpy array of shape (H,W) or (H,W,C) "
TypeError: %d format: a number is required, not str

The config is the same as "exp_lmo_BOP.txt" but because i have a 2080Ti with 11GB of vram i have chosen my number of workers to 4 using a batch size of 16.

I believe my file structure is correct, but I'm not entirely sure. Here it is for lmo.

Any ideas as to what might cause this?

Inconsistent and slow execution of "test_network_with_single_obj" during training

As the title says i am having some performance issues when training.

training 1000 iterations takes on average 60 minutes. Of these 60min around 50min are executing test_network_with_single_obj. The weird thing is that sometimes test_network_with_single_obj will finish within 20 seconds

I have tried training on my own dataset and lmo the problem persists with both.
The computer specs are 2080 ti and a 9900k, 32GB of ram
I have tried different combinations of batch_sizeand num_workers.
progressive X has also been disabled.

Any other suggestions as to what might cause this issue?

large drop of performance when using icp

Dear Authors,

I hope this message finds you well. I am writing to express my appreciation for your remarkable work. I have been utilizing your pretrained weights of LM-O in an attempt to reproduce the results presented in your paper. My results closely align with yours, with a minor difference (75.4 vs. 76.9).

While examining your scripts, I noticed the argument “--use_icp True”. I was intrigued by the potential impact of using ICP to refine the pose. However, when I incorporated “--use_icp True” during evaluation, I observed a significant drop in the results ( can, driller, glue ). I am curious to know if you have previously encountered this issue and if you could possibly shed some light on why this might be happening.

Here are the results for your reference:

Method	Ape	Can	Cat	Driller	Duck	Eggbox	Glue	HolePuncher	Mean
ZebraPose(in your paper)	57.9	95.0	60.6	94.8	64.5	70.9	88.7	83.0	76.9
ZebraPose(reproduced)	55.0	94.5	57.9	93.2	61.7	70.6	89.0	81.2	75.4
ZebraPose+ICP(reproduced)	60.4	57.4	62.1	62.2	63.8	73.8	69.9	88.4	68.2

Yours,
Leroy Chou

AUC_posecnn_ADD/ADI

Hi thanks for sharing the code. In terms of the reported results of Ycbv dataset in the paper, I think the reported AUC_ADD(-S)/AUC_ADD-S corresponds to the AUC_PoseCNN_ADD/ADI in the code. I want to confirm whether my understanding is correct or not. Thanks!

difference between two arXiv versions

Hi,

I find out that you have uploaded a new version of your paper in arXiv, and the performances are greatly improved compared with v1. However, I could not figure out what contributed to such significant improvement. Could you help clarify this? Thanks very much!

ycbv_symmetry_obj in common_dataset_info.py

the definition of ycbv_symmetry_obj in common_dataset_info.py is
ycbv_symmetry_obj = {
'bowl',
'wood_block',
'large_clamp',
'extra_large_clamp',
'foam_brick'
}
but large_marker is symmetry_obj in models_info.json, why it is not in ycbv_symmetry_obj?

extract features from the ROI

Hi,

I want to ask, after the Fcos detector obtains the results of Bboxes, how does the ResNet-34 network extract features from the ROI region of interest（e.g. ape）? Is this implemented by the pre-trained model in the pretrained_backbone folder?

Best,
Joa

lmo download speed is very slow

dear author,thanks for your work.When i try to download the gt datasets,the download speed is very slow

OpenGL shader issue when generating GT training data

Dear Yongzhi,

I have compiled and installed the GT generation part of the code and dependencies.
The GT object mesh is successfully generated, but I am running into an error when trying to render the GT training data.

I changed the line 39 in the opengl_render.cpp to my local path to the render_related_source folder.
It happens on both my own dataset and on LM-O.

I also noticed that this portion of the code received major updates 15 days ago. Is this a working version on your system?

Below is the resulting output.

$ python generate_training_labels_for_BOP.py --bop_path $BOP_PATH --dataset_name lmo --force_rewrite True --is_training_data True --data_folder train --start_obj_id 1 --end_obj_id 1
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000001.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000005.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000006.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000008.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000009.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000010.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000011.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000012.ply
if models are not fully listed above, please make sure there are ply files available
[640.      480.      572.4114  573.57043 325.2611  242.04899]
ERROR::SHADER::FILE_NOT_SUCCESFULLY_READ
ERROR::SHADER_COMPILATION_ERROR of type: VERTEX
0:1(1): error: syntax error, unexpected $end

 -- --------------------------------------------------- -- 
ERROR::SHADER_COMPILATION_ERROR of type: FRAGMENT
0:1(1): error: syntax error, unexpected $end

 -- --------------------------------------------------- -- 
ERROR::PROGRAM_LINKING_ERROR of type: PROGRAM
error: linking with uncompiled/unspecialized shadererror: linking with uncompiled/unspecialized shader
 -- --------------------------------------------------- -- 

Can you maybe point me in the right direction?

Best,
Thijs

the JSON results of GDRNPPDet

Hello,
I would like to ask if there is the JSON results of GDRNPPDet?

CUDA out of memory (on 16GB GPU with minimal batch size)

Dear authors,

Thank you for your contribution and work.

I am trying to run your code for my own research. However, CUDA keeps running out of memory.
I have tried to make the following modifications, some have some effect of the memory running out a bit slower:

• Set the batch size to 2 (it will not go to 1)
• Set resnet to 18 layers
• Switched to Google Colab (16GB GPU instead of the 4GB on my computer)
• Switched progressive-x off.

I am trying to train on a small subset of TUD-L dataset, using the GTs and pre-trained backbones provided through the links on this repo.

Here is a screenshot from the Colab environment:

Suddenly, there is a huge spike in GPU RAM usage, seen on the right in the dashboard.

Is this an issue or is this expected behaviour? Your help would be much appreciated.

Best regards,
Thijs

Asking for training details

Hi,

Thanks for your great job and your kindness in sharing your pretrained models!

I'm wondering if you could share the training details of other datasets in BOP besides YCBV and LM/LMO. Such as the choice of training data, the augmentation skills, etc.

It would be really helpful for reproducing your results and develop your method.

Error when testing the trained model for eggbox on LMO

Hi, when I evaluate your trained model for eggbox on LMO, I encountered the following error. How can I fix this issue? Thanks!

Score for Prediction and refinement

Hi Yongzhi,

I couldnt find a score for prediction and refinement in test_vivo.py . I think I am again missing something there. How do we calculate the score for both predictions and icp refinements?

Thanks in advance

Artifacts in saved images

Hi all,

I'm currently trying to apply Zebrapose to my own dataset. I've sucessfully generated the c++ executable and the python wrapper, and I can run the generator to render the zebrapose annotated images

python3 generate_training_labels_for_BOP.py --bop_path /home/user/datasets/bop/ --dataset_name myDataset --force_rewrite True --is_training_data False --data_folder test --start_obj_id 0 --end_obj_id 1

The script runs fine and I can see that correct images are being generated, see the following screenshot:

Screenshot

However, when I open the saved images, all of them have some artifacts on the top border, and the rendered image content is shifted by the same height along the y axis. See this example:

Saved Image

My specs:

• Ubuntu 20.04 LTS
• Python 3.6.15
• OpenGL 1.3.2
• glfw 3.3.2
• OpenCV 4.8.0
• glad was downloaded with the default settings (C/C++, OpenGL, compatibility profile)

Has anyone experienced a similar issue or a good guess what could be the issue? Thanks in advance!

Pretrained LM-O weights

Hi,
Thank you for your great work. I'm very interested in your proposed method and wanna reproduce the results on LM-O, but it seems that you didn't provide pretrained weight of each object. It would be very kind if you make the pretrained weights publicly available.
Best regards
Leroy

Installing the dependencies for Progressive-X

Hello, I am working on reproducing this project. I encountered the issue shown in the image while installing the dependencies for Progressive-X. I have searched the search engine for solutions to the related problem but couldn't resolve it. I would like to ask for guidance on how to address this issue. My OpenCV version is 4.3.0.36.