xyjbaal/fpcc

FPCC: Fast Point Cloud Clustering-based Instance Segmentation for Industrial Bin-picking [Arxiv]

Other Implementation

• pytorch-fpcc

Citation

If you find our work useful in your research, please consider citing:

@article{XU2022255,
title = {FPCC: Fast point cloud clustering-based instance segmentation for industrial bin-picking},
journal = {Neurocomputing},
volume = {494},
pages = {255-268},
year = {2022},
issn = {0925-2312},
doi = {https://doi.org/10.1016/j.neucom.2022.04.023},
url = {https://www.sciencedirect.com/science/article/pii/S0925231222003915},
author = {Yajun Xu and Shogo Arai and Diyi Liu and Fangzhou Lin and Kazuhiro Kosuge},
keywords = {Bin-picking, 3D Point Cloud, Instance segmentaion, Deep Learning},
}

Dependencies

• tensorflow (1.13.1)
• h5py

Data generation

Thanks for [waiyc] for providing a [script] to generate synthetic data by Pybullet. The dataset is generate and recorded base on the steps mentioned in IPA Dataset.

Training & Testing

python fpcc_train.py 

python fpcc_test.py

Evaluation metric

We use the code provided by [ASIS] to calculate precision and recall.

XA Dataset

XA dataset can be downloaded [here]. Please cite this paper or "FPCC" if you want to use XA dataset in your work,

@ARTICLE{9025047,
 author={Xu, Yajun and Arai, Shogo and Tokuda, Fuyuki and Kosuge, Kazuhiro},
 journal={IEEE Access},
 title={A Convolutional Neural Network for Point Cloud Instance Segmentation in Cluttered Scene Trained by Synthetic Data Without Color},
 year={2020},
 volume={8},
 number={},
 pages={70262-70269},
 doi={10.1109/ACCESS.2020.2978506}
}

IPA Dataset

The information about IPA dataset can be found [here]. The paper about IPA dataset can be downloaded [here]. The author of IPA did not provide a public link to download the data set, so maybe you need to register first.

We only uploaded part of the IPA dataset in the "datas" folder. Use the following scripts for generating h5 files for traing.

python convert_csv2json_annotation.py
python IPA_image2pc.py
python generate_ipa_center_h5.py
python generate_file_list.py

Acknowledgemets

This project is built upon [PointNet], [PointNet++], [SGPN] and [DGCNN]

Others

The program is not very beautiful. If you have any questions, please feel free to contact me at the address below. [email protected]

fpcc's Issues

Where should generated datasets be located?

I generated my own custom dataset with the help of this script. Where should the files be placed for the training?

json file in IPA dataset

thanks for your great work and sharing it
I want to train your code with the IPA dataset but the dataset doesn't have .json file inside each folder.
How can I get these files?

output format

thanks for your great work and sharing it.
could you describe about output files? especially the numbers after (x,y,z) in the predict file or seg file that were generated by the test code

Instance segmentation to 6D pose

I see that you output the instance segmentation but is there a way to extract 6D pose too from your results?

Error in fpcc_test.py

When executing fpcc_test.py on my custom trained model I get the following error:

/home/ceres/git/FPCC/utils/test_utils.py:268: RuntimeWarning: invalid value encountered in sqrt
  dis_mask = np.sqrt(dis_mask)
/home/ceres/git/FPCC/utils/test_utils.py:269: RuntimeWarning: invalid value encountered in greater
  pts_corr[np.where(dis_mask > use_3d_mask)] = 999

What could be the cause of that?

parameter.json and d_max of Obeject ABC

Hello, I think your article is great and I want to learn more deeply. Now I have encountered several problems:

1.I wanted to use IPA Dataset but needed parameter.json at runtime, #4 The webpage mentioned in is only Bunny's parameter.json. Can I use this instead, because I don't know the parameters required for this json file? Other .json file in the webpage don't make code work properly.

2.What is the appropriate "d_max" value for Object A, B, and C? I tried to train with 0.18, but the effect was not very good. Does "d_max" refer to the maximum width of the part?

3.Does the order in the datas/*_train.txt file have any effect on the training effect？

thanks

best match between the center_score and the r_nms

what is the best match between the center_score and the r_nms when test the gear.

[Possible Idea] Use features instead of geometric center for simpler and faster position recognition

@xyjbaal
I'm a layman when it comes to neural networks but as far as I understood your paper and code, the process is basically this:

1. Generate or obtain a pointcloud dataset, where each point is assigned a score depending on it's distance to the center of the part that it belongs to.
2. Train the model.
3. Use the model to infer a new pointcloud, which tries to assign a center score to every point. This creates clusters of points with high center scores.
4. Apply a score threshold to obtain those clusters.
5. Apply a radius threshold to collect points around the clusters and assign them a segment (I believe that more is happening at this step but I don't really understand it).
6. The result is a pointcloud with two additional columns that represent the center score and segment number of each point.

My problem is, that I have a part that is shaped like a crate (not exactly but the principle is the same). Take a look at this sectional view:

+                               +
L                               L
L                               L
H             Center            H
H             +---+             H
H             | X |             H
H             +---+             H
H                               H
L                               L
L                               L
L                               L
+LLLLLLLLLLHHHHHHHHHHHLLLLLLLLLL+

H: high center score
L: low center score
X: center point

When trying to segment a bin full of those parts, this will pose a problem. For example: a part is positioned in such a way, that the bottom and two sides are visible. After the inference there will be 3 point clusters for the same part. Depending on how I assign the score and distance thresholds, I will either get 2 to 3 different segments for the same part or one segment but with additional points from other parts.

All of this led me to an idea that could possibly simplify and even speed up the matching process that follows after the segmentation:

What if we use multiple geometric features instead of the center score when training the model in Step 1 from the list above? Lets take for example every corner of the part and depending on its distance to it, each point gets a "feature score" assigned to it. After the inference, we would ideally get multiple point clusters for each part, that we can extract with a score threshold and then determine the centroid. In theory the result would be a really lightweight pointcloud consisting of those centroids, where one could then perform ICP matching against the known corners of the part.

Here is the process again step by step:

1. Generate or obtain a pointcloud dataset, where each point is assigned a score depending on it's distance to a geometric feature (eg. corner, edge, surface center, etc.) of the part that it belongs to.
2. Train the model.
3. Use the model to infer a new pointcloud, which tries to assign a feature score to every point. This creates clusters of points around those geometric features.
4. Apply a score threshold to obtain those clusters.
5. Calculate the centroids of those clusters.
6. The result is a pointcloud consisting only of those centroids that represent the position of a geometric feature (ideally).
7. Perform point matching via ICP (or something similar) against the known ideal positions of the geometric features.

I wanted to ask you if this idea is even feasible and if so, what would need to be done to achieve it. Maybe this could be a topic for your new paper :)

How to process my own data?

Hello, your work is outstanding, and I want to use your network to process the data I have collected.
Currently, I have captured some point clouds and performed segmentation and instance labeling using CloudCompare. My question is, how should I preprocess my data for training? Looking forward to your response.

r_nms in fpcc_test.py

Hi, Firstly thank for your great work.

Can you explain what r_nms param means and how to choose it?
Or maybe you can prompt some methodology to adopt r_nms and conf_th parameters, because when I trained neural networks prediction of object centers is satisfactory, but final prediction is quite weak and depend on these parameters and I cannot choose that parameters to have good results on pointcloud with few and many(20-30) elements?

How to place datasets?

I am very sorry to disturb you, I would like to ask you how to place the dataset when training the FPCC network?

How to get the segmented points？

I run the fpcc_test.py，and get the results.I only see the center score prediction，but i want to get the instance prediction.Can you tell me how to get the instance predition?Thanks!

about input point cloud size

Hi,

In your paper, In each batch in the training process, input points (N = 4, 096) are randomly sampled from each scene and each point can be sampled only once.
Just wanted to check with you, does input point cloud size matter in this case (in terms of prediction accuracy for the segmentation results) ?

For example, if we control the input point cloud size ~ 4096 points and there will be no point cloud loss in the sampling step, can it get higher prediction accuracy in this case?

Result Evaluation

Hi,

I saw that there is a function eval_3d_perclass() in test_utils.py for evaluation.
Can you let me know on how to use this function to evaluate the performance ?

Consulting Opportunity

Hi @xyjbaal,

I'm reaching out to you after reading your FPCC paper.
Are you open to new opportunities?

At smartR AI we are working on a novel end-effector for bin picking.
The challenge now involves 6D pose estimation in a cluttered scene.

Would you be interested in knowing more about our project and joining the team?
Anything from 2h/month consulting to a full-time remote job is possible!

Let me know if you are interested.

Best,
Theodoro
Project Manager - smartR AI

Collect simulation dataset

Hi,

In the paper, you have mentioned that the training data from XA dataset is collected in simulation.
Is it possible that you can provide more information on how you collect the point cloud data with segmented index? (is the data generation source code available on github?)

Thanks

Training on GPU is failing

I managed to start the training but it was running on the CPU because tensorflow-gpu was missing. After installing tensorflow-gpu==1.13.1 and Cuda 10.0 along with the corresponding cudnn 7.6.4 the training fails with the following error message:

Training for the epoch 0/100 ...
2022-02-24 19:27:57.240773: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2022-02-24 19:29:30.267115: E tensorflow/stream_executor/cuda/cuda_blas.cc:698] failed to run cuBLAS routine cublasGemmBatchedEx: CUBLAS_STATUS_EXECUTION_FAILED
2022-02-24 19:29:30.267153: E tensorflow/stream_executor/cuda/cuda_blas.cc:2620] Internal: failed BLAS call, see log for details
2022-02-24 19:29:30.267182: E tensorflow/stream_executor/cuda/cuda_blas.cc:698] failed to run cuBLAS routine cublasGemmBatchedEx: CUBLAS_STATUS_EXECUTION_FAILED
2022-02-24 19:29:30.267214: E tensorflow/stream_executor/cuda/cuda_blas.cc:2620] Internal: failed BLAS call, see log for details

Traceback (most recent call last):
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMMBatched launch failed : a.shape=[4,4096,50], b.shape=[4,50,4096], m=4096, n=4096, k=50, batch_size=4
[[{{node MatMul_4}}]]
[[{{node Mean_2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "fpcc_train.py", line 282, in
train()
File "fpcc_train.py", line 271, in train
train_one_epoch(epoch)
File "fpcc_train.py", line 236, in train_one_epoch
_, loss_val,score_loss_val, grouperr_val = sess.run([train_op, loss, score_loss, grouperr], feed_dict=feed_dict)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMMBatched launch failed : a.shape=[4,4096,50], b.shape=[4,50,4096], m=4096, n=4096, k=50, batch_size=4
[[node MatMul_4 (defined at /home/ceres/git/FPCC/models/model.py:199) ]]
[[node Mean_2 (defined at /home/ceres/git/FPCC/models/model.py:287) ]]

Caused by op 'MatMul_4', defined at:
File "fpcc_train.py", line 282, in
train()
File "fpcc_train.py", line 129, in train
loss, score_loss, grouperr = model.get_loss(net_output, labels,vdm, asm, D_MAX, MARGINS)
File "/home/ceres/git/FPCC/models/model.py", line 199, in get_loss
group_mat_label = tf.matmul(pts_group_label,tf.transpose(pts_group_label, perm=[0, 2, 1])) #BxNxN: (i,j) = 1 if i and j in the same group
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2417, in matmul
a, b, adj_x=adjoint_a, adj_y=adjoint_b, name=name)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1423, in batch_mat_mul
"BatchMatMul", x=x, y=y, adj_x=adj_x, adj_y=adj_y, name=name)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/ceres/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): Blas xGEMMBatched launch failed : a.shape=[4,4096,50], b.shape=[4,50,4096], m=4096, n=4096, k=50, batch_size=4
[[node MatMul_4 (defined at /home/ceres/git/FPCC/models/model.py:199) ]]
[[node Mean_2 (defined at /home/ceres/git/FPCC/models/model.py:287) ]]

Am I using the correct dev environment? Your paper says that you used a GTX1080 but I have an RTX3070. Is my card not compatible with the older tensorflow or CUDA versions or did I setup the wrong enviroment?