thodan/bop_toolkit

A Python toolkit of the BOP benchmark for 6D object pose estimation.

bop_toolkit's Issues

3D bounding box labels around rendered image ?

Hi @thodan ,

is there a script for 3D bounding box corner generation of rendered image ?
or
any idea on how to generate 3d bbox corner labels ?
Thanks!
(sorry for stupid questions)

collecting parameters in a single config file (like config.py)

avoid duplicated definitions and multiple argparsers but retain standalone scripts.

TUD_L render issue

Hi,

I use the renderer provided in this git to project the ground truth label to RGB image, but the projection seems not right. Then I projected the object model with ground truth pose to the depth, I found they are not aligned. The detailed information:

tudl_train_real/000001/depth/000000.png (the target depth image).

The projected result, the blue one is projected mesh model with GT pose, the yellow points are depth which transformed to point cloud with known camera matrix.

How to generate mask images

I have generated my own dataset in BOP format.
And I am using 'bop_toolkit/scripts/calc_gt_masks.py ' to generate masks and visib_mask images.
However, our generated mask images are totally black. I am wondering whether there are some parameters we should config? (I have config the dataset path)

Thanks!
@thodan

looking forward to the doc support to the vis_gt and vis_test

Symmetry information in model_info.json ?

I am trying to use my own models and I used 'calc_model_info.json' script. However, The script does not generate some property called "symmetry_discrete" that I do not know what it means. Do you know what it is or how can I generate this information for my own model?

#48

KeyError: 'total_recall' in show_performance_bop19.py

It looks like in, for example, error=vsd_ntop=-1_delta=15.000_tau=0.050/matches_th=0.050_min-visib=-1.000.json, there is no key named total_recall. I run the script by simply typing: python scripts/show_performance_bop19.py Am I missing something?

Handling stray '/' in the path of a dataset.

E.g. importing ycbv and the path is <path>/ycbv/ this doesn't work, but <path>/ycbv works.

I would it would be nice if this is handled in the code, some users might not figure out that this the problem very quickly.

Scripts/calc_model_info.py maybe have error!

In the file Scripts/calc_model_info.py, the last code
inout.save_json(dp_split['models_info_path'], models_info)
maybe place
inout.save_json(dp_model['models_info_path'], models_info).

Does bop_toolkit have an example that converts from depth to point cloud?

Hello,

As the title said, Does bop_toolkit have an example that converts from depth to point cloud?

Could I use this dataset( PBR images) for object detection?

hello, @thodan . Could I use the PBR dataset for object detection? I noticed you provide the bbox GT(ground truth), but the format isn't compatible with Faster RCNN, YOLO, etc.
And I noticed that the PBR dataset GT are different from other datasets(LM, LM-O, etc.). How can I convert PBR GT format to LM format?

b'The GLFW library is not initialized'

when i run render_python , i have the error " b'The GLFW library is not initialized'

AttributeError: 'list' object has no attribute 'flatten' when rendering RGBD data

When I set up config.py and prepare lm dataset already.
I got the following error when run render_train_imgs.py to get RGB-D images.

---

Traceback (most recent call last):
File "render_train_imgs.py", line 226, in
out_path=out_path, obj_id=obj_id), scene_camera)
File "/home/ziqi/bop_toolkit/bop_toolkit_lib/inout.py", line 175, in save_scene_camera
im_camera['cam_K'] = im_camera['cam_K'].flatten().tolist()
AttributeError: 'list' object has no attribute 'flatten'

---

After looking deeper into the code, I see scene_camera has .flatten().tolist() method and inout.py also has .flatten().tolist() method in save_scene_camera(path, scene_camera) function.

---

codes in render_train_imgs.py:

scene_camera[im_id] = {
  'cam_K': dp_camera['K'].flatten().tolist(),
  'depth_scale': dp_camera['depth_scale'],
  'view_level': int(views_level[view_id])
}

codes in inout.py:

if 'cam_K' in im_camera.keys():
  im_camera['cam_K'] = im_camera['cam_K'].flatten().tolist()

---

I fixed this error by remove all .flatten().tolist() in render_train_imgs.py to make sure it's not converted to list before save_scene_camera() function.
I want to know is it OK to do like this? Or, the code itself is right while I missed something?

model info and gt masks errors

Hi there,

I'm trying to rebuild a dataset similar to those of the BOP challenge. For that I generated synthetic images with BlenderProc and now I try to generate the json files I need and the masks. Before executing these scripts on my dataset, i run it on T-LESS dataset to make sure I can use his scripts. After running two scripts, I get the following errors that I can't solve:

• python scripts/calc_gt_masks.py

8/10|09:00:09: Initializing renderer...
[w] Backend (<module 'glumpy.app.window.backends.backend_glfw' from '/home/imr/.local/lib/python3.8/site-packages/glumpy/app/window/backends/backend_glfw.py'>) not available
[w] Backend (<module 'glumpy.app.window.backends.backend_pyglet' from '/home/imr/.local/lib/python3.8/site-packages/glumpy/app/window/backends/backend_pyglet.py'>) not available
[w] Backend (<module 'glumpy.app.window.backends.backend_sdl' from '/home/imr/.local/lib/python3.8/site-packages/glumpy/app/window/backends/backend_sdl.py'>) not available
[w] Backend (<module 'glumpy.app.window.backends.backend_sdl2' from '/home/imr/.local/lib/python3.8/site-packages/glumpy/app/window/backends/backend_sdl2.py'>) not available
[w] Backend (<module 'glumpy.app.window.backends.backend_osxglut' from '/home/imr/.local/lib/python3.8/site-packages/glumpy/app/window/backends/backend_osxglut.py'>) not available
[w] Backend (<module 'glumpy.app.window.backends.backend_freeglut' from '/home/imr/.local/lib/python3.8/site-packages/glumpy/app/window/backends/backend_freeglut.py'>) not available
8/10|09:00:12: Calculating masks - dataset: tless (train, pbr), scene: 0, im: 0
QOpenGLContext::swapBuffers() called with non-exposed window, behavior is undefined

• python scripts/calc_model_info.py

8/10|08:53:40: Processing model of object 1...
Traceback (most recent call last):
File "scripts/calc_model_info.py", line 40, in
size = map(float, (model['pts'].max(axis=0) - ref_pt).flatten())
TypeError: unsupported operand type(s) for -: 'float' and 'map'

Has anyone encountered this kind of error before, or would know how to fix it?

render my ply file to rgb image,but not the color in my ply file

I used your code to render my ply file, which has color information in the face element;But when rendered the color is not the color in face element ;I found the following problems:

1. I checked the function load_ply(), which does not load the color information of the face element
2. I am not sure how to render the color of face elements

I open the ply model in meshlab ,the model looks like:

ply file has color in face element:

this is the rgb file i rendered. It shows little feature of logo .

can you tell men how to change the code to render image like in meshlab

YCB-V Ground Truth Transformation Cannot Achieve Full Score?

I am doing a sanity check by feeding ground truth transformation of the YCB-V dataset into the evaluation script, but it cannot achieve full bop19_average_recall_mspd and bop19_average_recall_mssd on both Github code and BOP online system (unpublished entry name: "Ground-Truth-Test_Only/YCB-V/test"). And bop19_average_recall_vsd is 1.

More specifically, only the objects with id 13 and 18 have the problem. Object with id 13 keeps getting score 0 in all threshold settings and the score of object id 18 varies with thresholds. Scores for all other objects are 1.

The csv file I used is attached, and I manually compared some of it with scene_gt.json and it should be good. Could you please help see where I mess it up? Thanks!
gt_ycbv-test.zip

vis_object_poses Error

Hi , @thodan
I met an error when running the vis_estimated_poses.py, after reading the CSV file, it says"

NoneType' object has no attribute 'stride' , when visualizing, I use glumpy.

Wish you can help.

camera intrinsic for HOPE dataset

Hi @thodan,

I am currently trying to create a HOPE dataset with a BOP format, and I found that your cal_model_info.py providing a HOPE option. However, I still need to know the camera intrinsic if I want to use BlenderProc to create the HOPE dataset. Do you by any chance know about it?

Thank you very much!

Store coco RLE format masks into json files instead of binary mask images to reduce the bop dataset size?

Like coco annotations, the masks can be stored in RLE format in scene_gt_info.json files, so that the dataset size can be largely reduced.

how to generate real dataset

Hi, I just wonder how can I generate a real dataset in BOP format.

meshlab in ubuntu

Hi,when i try remesh_models_for_eval.py, i wonder how to set path of Meshlab in Ubuntu.

I set the path into '/usr/bin/meshlab ', but it shows:

Why the objects in testing are much smaller than that in training set

Hi , @thodan @bertid
I am not familiar with the rendering of OpenGL, and I just combine the separated given training images onto a background image, however, the objects in combined image are much larger than that in your test image. (How to put them in different depth? ) . What's wrong?

Thanks so much!

Create models_info.json for own models

Hi there,

I want to evaluate pose errors with my own obj. models. Therefore I need to create the models_info.json.
Have you any guide on how to create this file? It looks like you were using MeshLab for this task. But I am not sure.

Best wishes
Timon

How to train?

If I has a 3D model about ply file, how can I train my dataset?
There are no application doc about using train rendering code.

Thanks!

I have an visual error in mask images

For example in YCBV--0091video--899picture, some of the hidden points of banana are shown in the mask image.
I download YCBV dataset from the website of boptookit and wonder which block did I deal with that went wrong.

No module named 'bop_toolkit_lib ?

Hi
I am trying to follow the steps here

when I run python scripts/eval_bop19.py --result_filenames=gt-equivalent_ycbv-test.csv

I get this error

    from bop_toolkit_lib import config
ModuleNotFoundError: No module named 'bop_toolkit_lib'

Multiprocess rendering

Hi, I want to know that how can I render image multiprocessing.
I create one renderer(python) instance, and add objects to this renderer. I want to call it's render_object method multiprocessly. I implement this in my dataset loader. However it do not worked or dump any error.
Can you give me some advices?
Thanks!

All scene ground truths are set to True if using visib_gt_min = 0.1

In the recent commit 975fcf2 where 'visib_gt_min' was changed to -1, the code in eval_calc_scores.py was changed where the valid ground truth scenes are assigned True/False based on scene_targets.

In eval_calc_scores.py.

      216. scene_gt_valid[im_id] = [True] * len(im_gt)
      217. if p['visib_gt_min'] >= 0:
      218.  # All GT poses visible from at least 100 * p['visib_gt_min'] percent
      219.  # are considered valid.
      220.  for gt_id, gt in enumerate(im_gt):
      221.    is_target = gt['obj_id'] in im_targets.keys()
      222.    is_visib = im_gt_info[gt_id]['visib_fract'] >= p['visib_gt_min']
      223.    scene_gt_valid[im_id].append(is_target and is_visib)

Now, all the ground truths are considered valid as the append doesn't affect when considering scene_gt_valid[:im_gt] for further evaluation.

Because of the addition of line 216, line 223 might have to be changed to

scene_gt_valid[im_id][gt_id] = is_target and is_visib

why some datasets has no train images?

Hi, I download core datasets from the BOP2019 website. But after unzip this files, I found that hb_bop19 and itodd_full have no train sub-directory. Do these two datasets have no official training data?

Where to buy real objects for bop datasets ?

Hello.
I think this question is totally unrelated, but does anyone know where to buy or how to make those objects in bop datasets?
We want to use models trained by bop on a real-robot with gripper for bin picking and other stuff.

Thank you.

How to get the new targets files?

Hello, thodan！

I've trained my own dataset. But I found that evaluating datasets requires targets files.
I downloaded the test_targets_bop19.json file, but I'm sorry it doesn't meet my needs.
So can you tell me how to get my own targets files? Or is there a script to generate this file?

It would be great if you could tell me the method！Anyway, thank you for such a good project！

Add yaml-cpp to 3rd party libraries?

Where can I download the file 'test_targets_bop19.json' ?

Hi, Hodan:
the test targets file 'test_targets_bop19.json' needs to be saved under the bop dataset folder, but I didn't find this file in https://bop.felk.cvut.cz/datasets/, where can I download it?（especially for tless）
if it's not public, could you tell me the data formate of this file so I can evaluate my method on your metric?

Can I use the code to generate train dataset on a server without screen?

How to generate train image?

Hello.
I run the scripts/render_train_imgs.py. But unfortunately, the output is not as similar as training image in lm/train_pbr dataset.

Output of the script:

Training file inside lm/train_pbr

Is there any script that needed to merge output of render_train_imgs.py into above file?

Why to convert depth to distance in evaluation?

Hi, I am curious about the conversion from depth to distance when evaluation. I think with the depth map, we can do the same operation when we calculate Evsd. Nor in the original paper or this challenge explain why using distance to calculate difference is better.
Thanks！

ImportError: cannot import name 'app'

I have install glumpy, I also install the freetype and glfw as the page 'https://glumpy.readthedocs.io/en/latest/installation.html#step-by-step-install-for-x64-bit-windows-7-8-and-10'

but when I run the file "eval_calc_errors.py", a bug appears as follows:

File "e:/3DData/BOP/bop_toolkit-master/scripts/eval_calc_errors.py", line 193, in
width, height, p['renderer_type'], mode='depth')
File "E:/3DData/BOP/bop_toolkit-master\bop_toolkit_lib\renderer.py", line 93, in create_renderer
from . import renderer_py
File "E:/3DData/BOP/bop_toolkit-master\bop_toolkit_lib\renderer_py.py", line 8, in
from glumpy.app import app
ImportError: cannot import name 'app'

Translate images during rendering process ?

HI @thodan , Thank you for such a wonderful toolbox.

I wanted to know, if there is a way to translate the images during rendering process?

Because, I am working on creating my own dataset using a 3 distinct models and all the images are placed at center of the image after rendering. For my application, I would like to place all objects in same image. If there is a translation, then it would be easier to concatenate all objects to single image.

Any suggestions ?

If you have a different idea, I would gladly look into it.

Thank you for your suggestions.

Support ADD-(S) AUC metric?

Some methods like PoseCNN evaluate the Area Under Curve of ADD-(S), could bop_toolkit also support this feature?

When would the HB (HomebrewedDB) dataset be completely released?

The bug is too much!

Traceback (most recent call last):
File "/home/znb/research/bop_toolkit/scripts/render_train_imgs.py", line 115, in
ren_rgb.set_light_ambient_weight(ambient_weight)
File "/home/znb/research/bop_toolkit/bop_toolkit_lib/renderer_cpp.py", line 34, in set_light_ambient_weight
self.renderer.set_light_ambient_weight(light_ambient_weight)
AttributeError: 'bop_renderer.Renderer' object has no attribute 'set_light_ambient_weight'

Pix3D into bop_toolkit format

Do you think I can convert Pix3D dataset in bop_toolkit format ?

Are there multiple ground truth annotations for each instance of one obj in a image?

In some image which contains multiple instances of one object, it seems that only one annotation is available in scene_gt.json file. Do I miss something?

Error in opening the .ply files in the "models_eval" directory.

It seems that the .ply files in the "models_eval" directory are broken. The content is wrong no matter during loading them in python or opening them in editors.

Is the SiSo task suitable for Bop challenge?

Hello,

My method focuses on pose registration of object pose estimation, object segmentation is not yet developed. Now we can solve the SiSo problem, Is the SiSo task suitable for Bop challenge? Thanks a lot!

Can this tool make lineMod like dataset with my own data?

I can not understand how the tool works, the docs contains few information.

I want to get the Ground-truth annotations as blew:
cam_R_m2c - 3x3 rotation matrix R_m2c (saved row-wise).
cam_t_m2c - 3x1 translation vector t_m2c.

add versions to the packages in requirements.txt

Saving estimated poses - link of format description is not working

Hi there,
the link given in:

"""
Usage
2. Run your method.
"format description"
"""

throws "Server Error (500)".

Could you fix that, or give a short explanation on how the format of the output should look like to be evaluated with the bop_toolkit?

Best regards
Timon

vis_est_poses.py runs into various errors

Ahoj,
I was trying to run the vis_est_poses.py script on the created .csv file by Pix2Pose on the T-Less dataset, but ran into different errors after running the script several times.

• corrupted size vs. prev_size
• munmap_chunk() invalid pointer
• free(): invalid size
  all resulting in Aborted (core dumped)

Google search suggests that they're C/C++ related, so I'm a little lost rn.
Any help is greatly appreciated.
Cheers,
Jan

Incorrect ground truth in the PBR dataset?

I'm using the PBR dataset to train an pose estimation network. I use the depth map and gt pose to calculate each object's coordinate map, however it leads to incorrect result. Then, I plot the object model recovered from the depth map and gt pose and the gt object model (.ply) in point cloud format. The plot shows that the recovered model and the gt model is not aligned.

The following is the code I use to plot the point cloud:

import numpy as np
import open3d as o3d
from bop_toolkit import inout
from numpy.linalg import inv

obj_id = ...  # label of one object in the image (the object to plot)
gt_R_m2cs = ... # gt_R_m2cs for objects in one image  - (n_obj, 3, 3)
gt_t_m2cs = ... # (n_obj, 3, 1)
gt_labels = ...  # labels for objects in one image (n_obj, )  --  start from 1
cam_K = ... # (3, 3) - the camera's intrinsic matrix
rgb_path = "path of the rgb image"
sem_path = "semantic segmentation path"
sem_map =inout.load_im(sem_path)
depth_path = "depth map path"
depth_map = inout.load_depth(depth_path)
d_map = depth_map  # (H, W)
H, W = d_map.shape
xy = np.array(np.meshgrid(range(W), range(H), indexing='xy'), dtype=np.float)  # (2, H, W)
xyz = np.concatenate((xy * d_map[None, ...], d_map[None, ...]), axis=0)  # (3, H, W)
XYZ = (inv(cam_K) @ xyz.reshape(3, -1)).T.reshape(H, W, 3)
obj_mask = sem_map==obj_id
XYZ_obj = XYZ[obj_mask, :]  # (n, 3)
R_m2c = gt_R_m2cs[gt_labels==obj_id, :][0]
t_m2c = gt_t_m2cs[gt_labels==obj_id, :][0]
# recovered_obj = (inv(R_m2c) @ (XYZ_obj.T - t_m2c)).T  # (n, 3)
recovered_obj = np.linalg.solve(R_m2c, XYZ_obj.T - t_m2c).T

ptcd = o3d.geometry.PointCloud()
ptcd.points = o3d.utility.Vector3dVector(recovered_obj)
ptcd.paint_uniform_color([255, 0, 0])   # paint recovered model with red
# original model
base_model_path = ".../BOP/lm/models/obj_{cls_id:06d}.ply"
model_path = base_model_path.format(cls_id=obj_id)
model = ObjectModel(model_path)   # read the point cloud from .ply file

o3d.visualization.draw_geometries([ptcd, model.ptcd])

Take a cat in one image as an example:

If I use the same code to plot the recovered model using datas in BOP's LM test set, the recovered model is aligned with the gt model.

---

Transform 3d model to the camera coordinate with gt poses, plot together with scene recovered by depth map: