f2wang/objectdatasettools

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera. This project prepares training and testing data for various deep learning projects such as 6D object pose estimation projects singleshotpose, as well as object detection and instance segmentation projects.

objectdatasettools's Issues

Installation issue

Hey,
first I would like to thank you for building such an amazing tool.
But I've got some problems with the correct installation at my windows-based system.
I am currently working with python3 and already got git, cmake and pkgconfilite (via chocolately). But if I understand it right libssl-dev and libgl1-mesa-glx packages are just for Ubuntu Users. Is there an alternative for windows?
Furthermore I can't install an older version of opencv, so I got the last release opencv-python 4.2.0.34 .

I would be very thankful if the developers (or anybody else here) could help me with that problem.

Parameter or aruco makers adjustment on big Objects

Hi
I want to get big object datasets' masks and labels, such as luggage, but after registering scene computation, the point clouds look like this:

I also tried to scan longer time, or use 9*4 aruco makers but get worse result.
Is there any parameter or the size of aruco makers need to adjust?

If I manually rotate the object in Meshlab, it seems the masks are not match with the jpg files. Is it a correct way to do that?

Thanks.

a little error in readme.txt

{"fx": 614.4744262695312, "fy": 614.4745483398438,(IT SEEMS CONFLICT WITH THE FOLLOWING EXPALNATION!) "height": 480, "width": 640, "ppy": 233.29214477539062, "ppx": 308.8282470703125, "ID": "620201000292"}

If you don't know your camera's intrinsic, you can put a rough estimation in. All parameters required are fx, fy, cx, cy, where commonly fx = fy and equals to the width of the image and cx and cy is the center of the image. For example, for a 640 x 480 resolution image, fx, fy = 480(HERE IS THE ERROR, COZ IT CONFLICTS WITH ABOVE EXPLAINATION), cx = 320, cy = 240.

Can I generate the new dataset with other kind camera?

Thank you very much for the great job for providing this job~
I only have two kinds of camera,one is kinectv2 , and another one is zed.
I follow the instruction to construct the environment for this project.
I wonder about what can I do at step 5?

#### Step 5:

Install librealsense and its python wrapper.

Error occurs in step 6

Thanks a lot for this Repo!

However, have I completed steps 1 to 5, an error occurred when I try to create label files. Here is the error:

$ python3 create_label_files.py LINEMOD/lays
lays is assigned class label 0.
Traceback (most recent call last):
  File "create_label_files.py", line 96, in <module>
    mesh = trimesh.load(folder + "registeredScene.ply")
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/load.py", line 135, in load
    **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/constants.py", line 137, in timed
    result = method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/load.py", line 205, in load_mesh
    **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/ply.py", line 76, in load_ply
    ply_ascii(elements, file_obj)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/ply.py", line 485, in ply_ascii
    row = array[position]
IndexError: index 76862 is out of bounds for axis 0 with size 76862

I have also tried with python2.7, but the error appeared to be the same.

If the .ply is saved in ASCII, error would change into:

lays is assigned class label 0.
Traceback (most recent call last):
  File "create_label_files.py", line 96, in <module>
    mesh = trimesh.load(folder + "registeredScene.ply")
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/load.py", line 135, in load
    **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/constants.py", line 137, in timed
    result = method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/load.py", line 205, in load_mesh
    **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/ply.py", line 78, in load_ply
    ply_binary(elements, file_obj)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/ply.py", line 586, in ply_binary
    populate_listsize(file_obj, elements)
  File "/usr/local/lib/python3.5/dist-packages/trimesh/exchange/ply.py", line 551, in populate_listsize
    dtype=field_dtype)[0]
IndexError: index 0 is out of bounds for axis 0 with size 0

I really need some help, cause I have a deadline on July 31st💀

another？

In my opinion, the multi obj' pose label whether it can come true or not from the object detection algorithms. we found the position of multi-objects from an image.

error occured while read png with python3

It seems to be incompatible with python3.

I choose to read .png with PIL and seemingly repaired it.

reader = png.Reader(depth_file)
pngdata = reader.read()
depth = np.array(map(np.uint16, pngdata[2]))

swap all the fragments above with this:

depth = np.array(Image.open(depth_file))

Meanwhile, you need to typecast npcds to int:

int(npcds)

where is the origin of word coordinates of point clouds produced by register_scene.py ?Is it related to the markers?

Hi, first thanks for your work.
For now I want to use RGBD Camera to take RGB and depth images around single or multiple still objects ,and I want to get a full point clouds of the secene or the only object.
BUT i NEED TO KNOW the origin of word coordinates of these point clouds ,because i need this information to do manipulation with UR5 robots.
Can u give me some instructions about that ?
Thanks.

Mask vs RGB image

Below is a generated mask with the respective RGB image, the mask does not seem to be a pixel perfect of the jpeg. Is there something I am doing wrong while recording with the Real Sense camera, should I be holding the camera in a certain way or moving it in a specific way?

Tform = mesh.apply_obb()

When i reading the code in create_label_files.py , i have some question. what this code means? there already have a transforms.npy. why need this operate?

About the marker

Hi, you said "Color print the pdf with the correctly sized aruco markers in the arucomarkers folder.", is that mean print the pdf in a A4 paper? I print the marker in A4 and cut the marker to make the marker around the object, like

But the object I reconstructed is not right in scale.

Object coordinate system?

How do you define the object coordinate system ?

How to draw 3D Bounding Boxes for predicted and ground-truth pose

Hello!

I am having trouble with figuring out how to draw 3D bounding boxes to compare the predicted and ground-truth 6D pose for my object of interest. I can't understand in which frame those 8 corners are given (image frame, since 2D), but why those values are so small (corner coordinates range between 0.1 to 1.3)?

Any suggestions or hints would be helpful. Thank you

I can't generate the correct mesh for 'timer'

I follow the instruction to construct the model for timer.
But I get a ply file like this.

depth image in xtion pro

I use Xtion Pro to get the depth image from scene, and i wonder if i need to do something on the png data to fit the 8m param? I am not sure the radio here.

Issue with 2D and 3D corner coordinates

Hello!

Sorry for bothering you again, but I can't understand some moments. I have 2 questions:

1. I am confused that 2D corner coordinates are not between 0-1, but bigger than 1.
   Don't you think this is because of the camera I am using? I am using Intel RealSense D435.

2. In corners = compute_projection(transformed,K), transformed are 3D corner coordinates in world frame or in my model frame?

Some doubts about the code from create_label_files.py

I have some questions about "Tform = mesh.apply_obb()". What does it 'Tform' mean in this step. And why the final variable 'points_original' should convert by trimesh.transformations.transform_points?

Maybe my problem should be solved by query AABB and OBB means. But when I google its means still don't know what is means.

AttributeError: 'NoneType' object has no attribute 'shape' while creating Image Masks

Hello! I am Akmaral.
First of all, thank you for this great job and sharing with us.

I am getting this error: " AttributeError: 'NoneType' object has no attribute 'shape'
" while running 'python3 create_label_files.py LINEMOD/xxx' (Step 6). I have successfulIy completed all previous steps, and have registeredScene.ply inside my folder. Even updating my trimesh version doesn't help. Please can you give me a hint to solve this problem.

How to have just one mesh for the same object at different pose

Hello,
First of all congrats for the tool, I have been used it extensively. You can checkout my branch "tommaso" where I have created few scripts for bypassing the meshlab step using open3d library.

Anyway I have a question regarding the following problem: I have recorded two video of the same object lying in different positions (once on one side, and the other time on the other side).
I hence I have created two different meshes.
And currently my naive implementation is to consider these two meshes as two separate entities. Do you know how could I merge them together (maybe through ICP?) and so that I have just one mesh?

Questions about step 3

In step 3: Obtain frame transforms
Compute transforms for frames at the specified interval

Why do we need frame transforms ?
The "frames" means the frame of the object ?
What does the specified interval mean ? time interval ?

Thank you

to get highter precision dataset

@F2Wang Thank you for your excellent work.
I got unsatisfactory preformance when training singlehotpose with my own dataset. I guess it may be related to data annotation, I want you to give me some advice

1. is there any skills for data annotation? Such as trim the mesh file
2. except record.py and record2.py, have you tried to create datasets with higher precision parameters using realsense, such as with higher resolution?

example sequence - timer result

The result is wrong. What did I do wrong?
thanks!

---

Step:

1. install env (Ubuntu 18.04 Python3.6)
2. download example sequence (timer)
3. python3 order

python compute_gt_poses.py LINEMOD/timer
python register_segmented.py LINEMOD/timer
python create_label_files.py LINEMOD/timer

4. result: timer.ply
   timer.ply.zip

Changing of RGBD data

I would like to change the code where I can keep the real sense camera stationary and take a photo where each photo consists of a different pose of the object.

Would the rest of the steps still work with the above changes?

long RECORD_LENGTH generate invalid .ply file

I want to make a big dataset, so I set the RECORD_LENGTH a big value(300), the step 4 generate a invalid ply file like picture below. may be some parameter need to be adjust?

Correctness of labels

Dear Fan,

How can check the correctness of created labels (2D projections of the 3D BB from "labels" folder)?

I tried putting points on coordinates from .txt files (9 points for the center and 8 vertices) directly in the corresponding RGB image. Please have a look at the attached image.

Also, I noticed that some 2D vertices are with "-" (minus) sign.
I checked dimensions of my .ply 3D model, they look right.

Thank you.

ArUco Markers

I came across this generator for generating the markers - is there a specific format I need to follow in terms of spacing, IDs, to correctly use along with this repository?

Cannot run create_label_files.py

At line 98 of create_label_files I am getting the following error:

Traceback (most recent call last):
File "create_label_files.py", line 98, in
Tform = mesh.apply_obb()
File "/usr/local/lib/python3.6/dist-packages/trimesh/base.py", line 2066, in apply_obb
matrix = self.bounding_box_oriented.primitive.transform
File "/usr/local/lib/python3.6/dist-packages/trimesh/caching.py", line 99, in get_cached
value = function(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/trimesh/parent.py", line 124, in bounding_box_oriented
to_origin, extents = bounds.oriented_bounds(self)
File "/usr/local/lib/python3.6/dist-packages/trimesh/bounds.py", line 132, in oriented_bounds
if hasattr(obj, 'convex_hull'):
File "/usr/local/lib/python3.6/dist-packages/trimesh/caching.py", line 99, in get_cached
value = function(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/trimesh/base.py", line 1995, in convex_hull
hull = convex.convex_hull(self)
File "/usr/local/lib/python3.6/dist-packages/trimesh/convex.py", line 53, in convex_hull
qhull_options=qhull_options)
File "qhull.pyx", line 2428, in scipy.spatial.qhull.ConvexHull.init
File "qhull.pyx", line 357, in scipy.spatial.qhull._Qhull.init
scipy.spatial.qhull.QhullError: QH7023 qhull option warning: unknown 'Q' qhull option 'Qn', skip to next space
QH6035 qhull option error: see previous warnings, use 'Qw' to override: 'qhull i QJn Qt Pp QbB' (last offset 9)

While executing: | qhull i QJn Qt Pp QbB
Options selected for Qhull 2019.1.r 2019/06/21:
run-id 428483682 incidence Qtriangulate Pprecision-ignore
QbBound-unit-box 0.5 _maxoutside 0

Runtime

compatibility with python3

In order to run get_BBs.py succesfully with python3 the following is needed:
1)writer = csv.writer(open("annotations.csv", "wb"), delimiter=";") to writer = csv.writer(open("annotations.csv", "w"), delimiter=";") (at line 10)

1. at line 16 in get_BBS change xrange to range.

Confirmation of Extraction of R,t Matrix

    pose = np.load(folder)
    print(pose.shape)  # returns a 4 x 4 matrix

    pose = pose[:3]  # this drops the  [0,0,0,1] row

    R = pose.T[:3]
    t = pose.T[3]

I just want to confirm, if the above would be the correct way to extract the R,t matrx from each of the numpy transform folders. I am basing it off this transform matrix found in match_ransac:

transform = [[R[0][0],R[0][1],R[0][2],t[0]],
[R[1][0],R[1][1],R[1][2],t[1]],
[R[2][0],R[2][1],R[2][2],t[2]],
[0,0,0,1]]

'PointCloud' object has no attribute 'apply_obb'

When i run “python create_label_files.py all”，it prompts AttributeError: 'PointCloud' object has no attribute 'apply_obb'，what should i do? Thanks.

how do you get the initial pose?

I am curious about it, how do you make the mesh align to the image.

About 3D model from others way.

Hi， I want to know if I use a visualSFM to build an object's 3D model. Is this model can replace the file 'registeredScene.ply'? And I use this model to create the pose label.

Citation

Hi,

first of, not really an issue. I found your code really helpfull for understanding the singleshotpose and it allowed me to test their recognition on my own dataset. I want to cite this toolbox in the project I'm working on and wanted to ask if you had any preferences on that.

Question about step 5

1. The registered pointcloud (manually) mentioned in step 5 is it the same thing generated by the register_scene.py ? (But it seems that register_scene.py already will generate the mesh )

2. If the answer is yes in previous question, then can we just use the result from register_segmented.py without the need to do step 5 ?

3. If we really need to use meshlab, is it ok to just use the same setting as what you did in your youtube video ?

Thank you

make my own datasets for training another projects

Have you ever paid attention to DenseFusion and Deep_Object_Pose？

This two projects also realize 6D pose estimation. If I want to use the above two projects to train my data, can I use your tools to create my own datasets?

RunTimeError in Step6

@F2Wang
when I try "python create_label_files.py all", I got an error below

timer is assigned class label 0.
0%| | 0/55 [00:00<?, ?it/s]
Traceback (most recent call last):
File "create_label_files.py", line 128, in
sample_points = mesh_copy.sample(10000)
AttributeError: 'PointCloud' object has no attribute 'sample'

Hope someone gives some hints

A question about 3d reconstruction

Hi~
i wonder whether your code has used bundle adjustment to optimize the 3D reconstruction parts?
And why not use something, such as Elesticfution, to reconstruct the whole scene?

To check ground truth and mesh generated

@F2Wang I was trying to project back the ground truth generated but somehow it is not correct. Can you please tell me how I can project the ground truth back onto the RGB image and check it is correctly generated or not?
Or if you have any other idea to cross check can you please tell me?

I am trying to generate real data set and use it to train Dense fusion.

Trimesh Version, AttributeError: 'PointCloud' object has no attribute 'visual'

I believe I am not using the same version of Trimesh as the authors, as I am using their sample provided dataset of a timer. Would this issue be because of the current version of Trimesh I do have, if so which would be the correct one to install?

(boom) bash-3.2$ python create_label_files.py LINEMOD/timer
timer is assigned class label 0.
Traceback (most recent call last):
File "create_label_files.py", line 100, in
mesh.export(file_obj = folder + folder[8:-1] +".ply")
File "/Users/ishangupta/.pyenv/versions/boom/lib/python3.6/site-packages/trimesh/points.py", line 620, in export
**kwargs)
File "/Users/ishangupta/.pyenv/versions/boom/lib/python3.6/site-packages/trimesh/exchange/export.py", line 68, in export_mesh
export = _mesh_exporters[file_type](mesh, **kwargs)
File "/Users/ishangupta/.pyenv/versions/boom/lib/python3.6/site-packages/trimesh/exchange/ply.py", line 150, in export_ply
if mesh.visual.kind == 'vertex' and encoding != 'ascii':
AttributeError: 'PointCloud' object has no attribute 'visual'

Problems when generating mesh

We are met with a problem after running compose_gt_poses.py. 'Mesh saved' is printed but when we open the .ply in meshlab, we get this result.

The environment is like this:

An example of color pictures from JPENImages folder is

Our camera is realsense D435
Camera depth intrinsic is:
{'fx':640.292, 'fy': 640.292, 'height':720, 'width':1280, 'coeffs':[0,0,0,0,0], 'ppy':357.747, 'ppx':647.852, 'model':2}
The resolution of color and depth are both 1280*720.

We think the reason is that markers are not Identified. But we don't know the ids of the markers. Can you provide them?
Thanks.

how to create the 3D Location field for an object?

Hi,
recently, a lot methods try to regress the 3D Location field and get pretty results, and do you know how to create the 3D Location field from the pose and ply ?

e.g.Pose-guided Auto-Encoder and Feature-Based Refinement for 6-DoF Object Pose Regression

singleshotpose not accept binary mesh file

hi I'm sorry to bother you again.
as ObjectDatasetTools is created for singleshotpose， I notice that the mesh file generation at step 6 is binary，but the singleshotpose seems to txt format compatible only， how to deal this problem?
It is feasible to manually convert binary files to txt format. ，but I not sure whether it takes performance loss

Can I use it for multi obj?

I tried to directly record pictures with four items in it, but the .ply ,especially the occluded sides of the items, is quite bad.

pose optimization

hello,i get a result that have big pose error between the images,the error in the back is greater than first few image,i think that is problem of optimization,but i don't know how to adjust,can you help me ?and,i want to know must to be stable when recording images?

About the register_segmented.py

Hi, I tried to use the script register_segmented.py and adjust the MAX_RADIUS as you asked. But error happend:
Load and segment frames
0%| | 0/76 [00:00<?, ?it/s]/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
/usr/local/lib/python2.7/dist-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)

Traceback (most recent call last):
File "register_segmented.py", line 216, in
originals = load_pcds(path, downsample = False, interval = RECONSTRUCTION_INTERVAL)
File "register_segmented.py", line 126, in load_pcds
distance = point_to_plane(depth,sol)
File "/home/ghoshaw/usr/Detect3D/ObjectDatasetTools/utils/plane.py", line 55, in point_to_plane
plane_xyz = p[0:3]
TypeError: 'NoneType' object has no attribute 'getitem'

If I run register_scene.py, it was ok.
Is there any other parameter need to change or something? Thanks!

what's ppx and ppy mean in intrinsics.json ？

How to compute matrix transformation from two frames of object ?

How to compute matrix transformation (4x4) from two frames of object as an annotation .npy ?

about register_scene.py

when i run register_scene.py ,The reconstruction result is relatively bad,I think it's the reason for the video，but i don't get a good result after several attempts , so about make a viedo,Do you have any guidance？or another reason?

OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor

Hello Fan,

while running compute_gt_pose.py I am getting 'OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor' error, which I didn't get before. I thought this error is because of opencv version which I changed recently, but not. However I can run this code on my laptop with the same camera (RealSense). Currently I am using opencv -3.3.0.

What may cause such error? thank you.