pqhieu/jsis3d

[CVPR'19] JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds

JSIS3D

This is the official Pytorch implementation of the following publication.

JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with
Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields
Quang-Hieu Pham, Duc Thanh Nguyen, Binh-Son Hua, Gemma Roig, Sai-Kit Yeung
Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Oral)
Paper | Homepage

Citation

If you find our work useful for your research, please consider citing:

@inproceedings{pham-jsis3d-cvpr19,
  title = {{JSIS3D}: Joint semantic-instance segmentation of 3d point clouds with multi-task pointwise networks and multi-value conditional random fields},
  author = {Pham, Quang-Hieu and Nguyen, Duc Thanh and Hua, Binh-Son and Roig, Gemma and Yeung, Sai-Kit},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2019}
}

Usage

Prerequisites

This code is tested in Manjaro Linux with CUDA 10.0 and Pytorch 1.0.

• Python 3.5+
• Pytorch 0.4.0+

Installation

To use MV-CRF (optional), you first need to compile the code:

cd external/densecrf
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=Release ..
make
cd ../../.. # You should be at the root folder here
make

Dataset

We have preprocessed the S3DIS dataset (2.5GB) in HDF5 format. After downloading the files, put them into the corresponding data/s3dis/h5 folder.

Training & Evaluation

To train a model on S3DIS dataset:

python train.py --config configs/s3dis.json --logdir logs/s3dis

Log files and network parameters will be saved to the logs/s3dis folder.

After training, we can use the model to predict semantic-instance segmentation labels as follows:

python pred.py --logdir logs/s3dis --mvcrf

To evaluate the results, run the following command:

python eval.py --logdir logs/s3dis

For more details, you can use the --help option for every scripts.

Note: The results on S3DIS in our paper are tested on Area 6 instead of Area 5. To reproduce the results, please change the split in train.txt and test.txt accordingly. Here I chose to keep the test set on Area 5 to make it easier to compare with other methods.

Prepare your own dataset

Check out the scripts folder to see how we prepare the dataset for training.

License

Our code is released under MIT license (see LICENSE for more details).

Contact: Quang-Hieu Pham ([email protected])

jsis3d's Issues

If I don't consider the relationship between each data, how can I modify it

Thank you for your contribution. I'm currently training with my own dataset. Unlike S3DIS , my data object is every plant, not every room. There is a certain relative relationship between each house in S3DIS, but there is no relationship between each plant. In order to achieve this, what part of your code do I need to modify? My data are as follows.
Looking forward to your reply!

libgfortran.so.4

:~/jsis3d$python pred.py --logdir logs/s3dis --mvcrf
/home/pf/anaconda3/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "pred.py", line 10, in
from sklearn.cluster import MeanShift
File "/home/pf/anaconda3/lib/python3.6/site-packages/sklearn/init.py", line 134, in
from .base import clone
File "/home/pf/anaconda3/lib/python3.6/site-packages/sklearn/base.py", line 13, in
from .utils.fixes import signature
File "/home/pf/anaconda3/lib/python3.6/site-packages/sklearn/utils/init.py", line 11, in
from .validation import (as_float_array,
File "/home/pf/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 18, in
from ..utils.fixes import signature
File "/home/pf/anaconda3/lib/python3.6/site-packages/sklearn/utils/fixes.py", line 144, in
from scipy.sparse.linalg import lsqr as sparse_lsqr # noqa
File "/home/pf/anaconda3/lib/python3.6/site-packages/scipy/sparse/linalg/init.py", line 117, in
from .eigen import *
File "/home/pf/anaconda3/lib/python3.6/site-packages/scipy/sparse/linalg/eigen/init.py", line 11, in
from .arpack import *
File "/home/pf/anaconda3/lib/python3.6/site-packages/scipy/sparse/linalg/eigen/arpack/init.py", line 22, in
from .arpack import *
File "/home/pf/anaconda3/lib/python3.6/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 45, in
from . import _arpack
ImportError: libgfortran.so.4: cannot open shared object file: No such file or directory

Did L_var term in discriminative loss forget to divide the batch_size?

jsis3d/losses/discriminative.py

Line 61 in d4c27e6

loss += torch.sum(var[i,:,:n]) / torch.sum(masks[i,:,:n])

      for i in range(batch_size):
            n = size[i]
            loss += torch.sum(var[i,:,:n]) / torch.sum(masks[i,:,:n])
      loss /= batch_size

visualize

How can I visualize the prediction results after using pred.py，What does visualization mean in eval.py? After I use it, I use meshlab to display the ply file, but some files show that the time point cloud has no color？

about plot.py

Hello, I want to execute plot.py to check the result, and the parameter "metric" has reported an error. I would like to ask you what the parameter "metric" means and what the default value you set is.
Thank you.

How to separate each instance of point cloud instance segmentation？

Hello, I want to ask how to separate each instance. For example, I just want to visualize each chair. What should I do?

On the visualization of prediction results

Hello, I predicted two files, semantics.npy and instances.npy when I reproduced your code.What is their viewing mode?3d point cloud or picture?

Instance label

Hi,
Could you please tell me, how to get the instance label, Are you manually tagged?
Thanks !

Nan Loss & NormBackward1 in discriminative.py

Hi Quang-Hieu, Thanks for providing the code.

I have managed to adopt your code for training on S3DIS dataset successfully. However, when I shifted it to my own dataset, some Nan loss appears at the first few iterations. Sometimes, it produced nan loss after the correct loss value for 1~3 iterations (in 1st epoch), while sometimes the nan loss is produced directly at the first iteration (in 1st epoch). Different learning rates & num_points values are tested, with no luck. Then I used "torch.autograd.set_detect_anomaly(True)" to detect the abnormal gradient values in network, and received an error message below. I have spent some time working on this, but still not have a clue yet. Could you please give me some instructions/comments if possible? Thanks in advance!

Error Message

• train.py: loss += criterion['discriminative'](embedded, masks, size)
• discriminative.py: L_r = self._regularization(centroids, size)
• discriminative.py: norm = torch.norm(mu, 2, dim=1)
  RuntimeError: Function 'NormBackward1' returned nan values in its 0th output.

Differences between my dataset and S3DIS:
maximum num_instances per sample: less than 10
num_classes: less than 30
point sampling: random sample 4096 or 8192 points (out of 10e4-10e6 points) for each sample (rather than splitting into blocks at first)

How to prepare my own data set

In your code, you say follow the scripts folder to learn how to prepare the data set.But there are four files in the folder, what steps should be followed to prepare.
Also, how do I set the num_points parameter when I do the prepare_h5.py file:
batch = room_to_blocks(fname, **num_points,** size=1.0, stride=0.5)
I would also like to ask which of the following data sets is appropriate for processing
Stanford3dDataset_v1.2_Aligned_Version or Stanford3dDataset_v1.2

Cannot reproduce paper results

I run your code according to the current Readme file with MV-CRF. I use the same configuration present in s3dis.json but I cannot obtain the same results evidenced in the paper.
The platform: python 3.5.2, torch 1.0, cuda 9.0 cudnn 7.3.0
After 100 epoch,

Overall accuracy: 0.829
Mean accuracy: 0.507
Mean IoU: 0.433
Mean precision: 0.389
Mean recall: 0.287

Can you help me? Thank you very much.

did anyone try this network with a custom dataset ?

Hi there ,
thanks for the great open source contribution. I was wondering how is the network performing in a say custom dataset, is there anyone out there tried this ?

thanks

Cannot reproduce paper results

I've gone through the paper and run the code for MT-PNet with the commands specified in the Readme file and the same configuration present in s3dis.json but I cannot obtain the same results evidenced in the paper.

After 100 epochs of training, evaluation gives me

"mean_accuracy": 0.4461
"mean_IoU": 0.3441
"mean_precision": 0.2504
"mean_recall": 0.2118

and going on to 200 epochs results in

"mean_accuracy": 0.4617
"mean_IoU": 0.3600
"mean_precision": 0.2795
"mean_recall": 0.2330

Am I doing something wrong? Do I need to change some parts or adjust parameters?
Let me know if you need more information.
Thank you!

[email protected]

What are the evaluation indicators in the paper [email protected] It is np.mean(p) listed in the code ? If not, how should we calculate to get the information in the paper [email protected] ？

Can't reproduce the result for MT-PNet

I ran the original code in this repo with the data downloaded from the according url. According to the config.json file, I trained for 40 epochs and the loss value reached 1.0892. But when I test on the Area 5 data, it reported as follows:

> Overall accuracy: 0.812
> Mean accuracy: 0.490
> Mean IoU: 0.397
> Mean precision: 0.281
> Mean recall: 0.238
> Writing report to logs/s3dis/eval.json...

I ran the prediction code without --mvcrf and I don't think the performance reaches the result reported in the paper, which has 86.7% oAcc for MT-PNet.
Could you please help me about this issue?

mvcrf issue

Hi, can I directly use your mvcrf code just for segmentation, without instance? Many thanks.

Best Regards
Frank

training error

Hi, @pqhieu ,

After running python train.py --config configs/s3dis.json --logdir logs/s3dis, I got the following error. I had downloaded S3DIS dataset (h5 format) and have it extracted in JSIS3D/data/s3dis/h5 as JSIS3D/data/s3dis/h5/s3dis_h5_07042019/*.h5

python train.py --config configs/s3dis.json --logdir logs/s3dis
/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "train.py", line 31, in <module>
    dataset = S3DIS(args['root'], training=True)
  File "/data/code9/JSIS3D/loaders/s3dis.py", line 19, in __init__
    self.coords.append(fin['coords'][:])
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/h5py/_hl/group.py", line 167, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'coords' doesn't exist)"

It seems some error in h5py reading. Could you suggest me how to fix such issue?

THX!

Here are my package configurations:
h5py == 2.7.1
hdf5==1.10.1
pytorch == 0.4.0

About preparing my own dataset

hello，i am confued about the blocks in room_to_blocks in prepare_h5.py , what should i set up the threshold , and how dose cells work?

KeyError: "Unable to open object (object 'coords' doesn't exist)"

Hello,I meet a problem,
KeyError: "Unable to open object (object 'coords' doesn't exist)"

my dataset path is that: JSIS3D/data/s3dis/h5/*.h5. can you tell me how to solve this problem?looking forward to your reply.

Segmentation fault(core dumped)

Every time I try to run python train.py --config configs/s3dis.json --logdir logs/s3dis I get the error "Segmentation fault(core dumped)" . What can be the problem for that? I am using Ubuntu 18.04. And below these are my environment packages. Any help would be highly appreciated. Thank you.

• # Name Version Build Channel
• _libgcc_mutex 0.1 main
• _openmp_mutex 4.5 1_gnu
• blas 1.0 mkl
• ca-certificates 2021.10.26 h06a4308_2
• certifi 2021.10.8 py37h06a4308_0
• cffi 1.15.0 py37h7f8727e_0
• common_cmplr_lib_rt 2021.4.0 intel_3561 intel
• common_cmplr_lic_rt 2021.4.0 intel_3561 intel
• cuda100 1.0 0 pytorch
• dpcpp_cpp_rt 2021.2.0 intel_610 intel
• freetype 2.11.0 h70c0345_0
• giflib 5.2.1 h7b6447c_0
• icc_rt 2021.2.0 intel_610 intel
• intel-cmplr-lib-rt 2021.4.0 intel_3561 intel
• intel-cmplr-lic-rt 2021.4.0 intel_3561 intel
• intel-opencl-rt 2021.4.0 intel_3561 intel
• intel-openmp 2021.4.0 h06a4308_3561
• intelpython 2021.4.0 0 intel
• jpeg 9d h7f8727e_0
• lcms2 2.12 h3be6417_0
• libedit 3.1.20210910 h7f8727e_0
• libffi 3.2.1 hf484d3e_1007
• libgcc-ng 9.3.0 h5101ec6_17
• libgomp 9.3.0 h5101ec6_17
• libpng 1.6.37 hbc83047_0
• libstdcxx-ng 9.3.0 hd4cf53a_17
• libtiff 4.2.0 h85742a9_0
• libwebp 1.2.0 h89dd481_0
• libwebp-base 1.2.0 h27cfd23_0
• lz4-c 1.9.3 h295c915_1
• mkl 2021.4.0 h06a4308_640
• mkl-service 2.4.0 py37h5c78031_2 intel
• mkl_fft 1.3.1 py37hbd9d9a5_0 intel
• mkl_random 1.2.2 py37hd708486_1 intel
• mkl_umath 0.1.1 py37h244f2d4_9 intel
• ncurses 6.3 h7f8727e_2
• ninja 1.10.2 py37hd09550d_3
• numpy 1.20.3 py37h2742bb4_2 intel
• numpy-base 1.20.3 py37hf707ed8_2 intel
• olefile 0.46 py37_0
• opencl_rt 2021.4.0 intel_3561 intel
• openmp 2018.0.3 intel_0 intel
• openssl 1.0.2u h7b6447c_0
• pillow 8.4.0 py37h5aabda8_0
• pip 21.2.2 py37h06a4308_0
• pycparser 2.21 pyhd3eb1b0_0
• python 3.7.0 h6e4f718_3
• pytorch 1.0.0 py3.7_cuda10.0.130_cudnn7.4.1_1 [cuda100] pytorch
• readline 7.0 h7b6447c_5
• setuptools 58.0.4 py37h06a4308_0
• six 1.16.0 pyhd3eb1b0_0
• sqlite 3.33.0 h62c20be_0
• tbb 2021.4.0 intel_643 intel
• tbb4py 2021.4.0 py37_intel_643 intel
• tk 8.6.11 h1ccaba5_0
• torchvision 0.2.1 py_2 pytorch
• wheel 0.37.0 pyhd3eb1b0_1
• xz 5.2.5 h7b6447c_0
• zlib 1.2.11 h7b6447c_3
• zstd 1.4.9 haebb681_0

step size when training

Hi.

Firstly, I appreciate for your great work and sharing your code.

Currently, I'm using your model for my project and I have a question.

On your thesis, it says that you set your decay rate to 0.5 and it applies for every 50 epochs.

However, in the config.json file, the step size is set to 20, not 50.

Which one is the right value? thank you

About the input channel of data

The costom data of pointcloud is 3 dimensions， why is this data 9 dimensions， thank you very much for checking my issue

make failed

Hi @pqhieu ,

When I compile CRF, all is good, but when I run make, it shows :

Scanning dependencies of target lbfgs
[  3%] Building C object external/CMakeFiles/lbfgs.dir/liblbfgs/lib/lbfgs.c.o
[  7%] Linking C static library liblbfgs.a
[  7%] Built target lbfgs
Scanning dependencies of target optimization
[ 11%] Building CXX object src/CMakeFiles/optimization.dir/optimization.cpp.o
[ 14%] Linking CXX static library liboptimization.a
[ 14%] Built target optimization
Scanning dependencies of target densecrf
[ 18%] Building CXX object src/CMakeFiles/densecrf.dir/util.cpp.o
[ 22%] Building CXX object src/CMakeFiles/densecrf.dir/permutohedral.cpp.o
[ 25%] Building CXX object src/CMakeFiles/densecrf.dir/unary.cpp.o
[ 29%] Building CXX object src/CMakeFiles/densecrf.dir/pairwise.cpp.o
[ 33%] Building CXX object src/CMakeFiles/densecrf.dir/objective.cpp.o
[ 37%] Building CXX object src/CMakeFiles/densecrf.dir/densecrf.cpp.o
[ 40%] Building CXX object src/CMakeFiles/densecrf.dir/labelcompatibility.cpp.o
[ 44%] Building CXX object src/CMakeFiles/densecrf.dir/higherorder.cpp.o
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp: In member function 'void HigherOrderPotential::apply(Eigen::MatrixXf&, const MatrixX
f&, const VectorXi&)':
/data/lirong/jsis3d/external/densecrf/src/higherorder.cpp:39:17: error: ISO C++ forbids declaration of 's' with no type [-fpermissive]
     for ( auto& s : indices_ ) {
                 ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:39:21: warning: range-based 'for' loops only available with -std=c++11 or -std=gnu++1
1
     for ( auto& s : indices_ ) {
                     ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:42:25: error: ISO C++ forbids declaration of 'v' with no type [-fpermissive]
             for ( auto& v : s.second ) {
                         ^
/data/lirong/jsis3d/external/densecrf/src/higherorder.cpp:42:29: warning: range-based 'for' loops only available with -std=c++11 or -std=gnu++1
1
             for ( auto& v : s.second ) {
                             ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:42:31: error: request for member 'second' in 's', which is of non-class type 'int'
             for ( auto& v : s.second ) {
                               ^
/data/lirong/jsis3d/external/densecrf/src/higherorder.cpp:52:17: error: ISO C++ forbids declaration of 's' with no type [-fpermissive]
     for ( auto& s : indices_ ) {
                 ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:52:21: warning: range-based 'for' loops only available with -std=c++11 or -std=gnu++1
1
     for ( auto& s : indices_ ) {
                     ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:54:21: error: ISO C++ forbids declaration of 'v' with no type [-fpermissive]
         for ( auto& v : s.second ) {
                     ^
/data/lirong/jsis3d/external/densecrf/src/higherorder.cpp:42:29: warning: range-based 'for' loops only available with -std=c++11 or -std=gnu++1
1
             for ( auto& v : s.second ) {
                             ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:42:31: error: request for member 'second' in 's', which is of non-class type 'int'
             for ( auto& v : s.second ) {
                               ^
/data/lirong/jsis3d/external/densecrf/src/higherorder.cpp:52:17: error: ISO C++ forbids declaration of 's' with no type [-fpermissive]
     for ( auto& s : indices_ ) {
                 ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:52:21: warning: range-based 'for' loops only available with -std=c++11 or -std=gnu++1
1
     for ( auto& s : indices_ ) {
                     ^
/data/lirong/jsis3d/external/densecrf/src/higherorder.cpp:54:21: error: ISO C++ forbids declaration of 'v' with no type [-fpermissive]
         for ( auto& v : s.second ) {
                     ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:54:25: warning: range-based 'for' loops only available with -std=c++11 or -std=gnu++1
1
         for ( auto& v : s.second ) {
                         ^
/data/lr/jsis3d/external/densecrf/src/higherorder.cpp:54:27: error: request for member 'second' in 's', which is of non-class type 'int'
         for ( auto& v : s.second ) {
                           ^
src/CMakeFiles/densecrf.dir/build.make:230: recipe for target 'src/CMakeFiles/densecrf.dir/higherorder.cpp.o' failed
make[2]: *** [src/CMakeFiles/densecrf.dir/higherorder.cpp.o] Error 1
CMakeFiles/Makefile2:124: recipe for target 'src/CMakeFiles/densecrf.dir/all' failed
make[1]: *** [src/CMakeFiles/densecrf.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Can you give me some advice?
Without CRF, I could run the whole project successfully, but I am really interested in CRF part.

I am using python=3.7, torch=1.1.0

Thanks for your help !

Code about MV-CRF

Would you open source code about "MVCRF"?
I hope that open complete source code.
Thank you very much!

pre_trained model

Hi @pqhieu ,
Thanks for your sharing this work,
Could you provide the pre-trained model?
Thanks !

how to visualize the experiment results of sematic segmentation on s3dis dataset?

Hello, how to visualize the experiment results of semantic segmentation on s3dis dataset?Do you ｕｐload ｔｈｅ code about visualization of semantic segmentation on s3dis dataset?looking forward your reply.

ｃａｎ you send me ｙour files have compiled?

I can not compile your code　ｃｏｒrectly by myself, can you send me your files only about have compiled?

coords and points

Hello,
I am confused about the 'coords' and 'points'. I thought 'points' contain the XYZ, RGB and Normal. But I don't konw what does 'coords' represent. Looking forward your reply.

TypeError: __init__() takes 1 positional argument from MeanShift

solved by

y = MeanShift(bandwidth=args['bandwidth'], n_jobs=8).fit_predict(embedded[b])

and it is really slow:).

can you provide code about color mapping?

Hello,do you implement color mapping?can you provide code about color mapping?I can not make it by myself.looking forward your reply.

How to calculate the mAP?

I find there is no score and only one pair of RP, so how to calculate the mAP?

Very slow predicition

Thanks for the great work!

While the training is fast and takes less than a second/batch, the prediction is very slow (takes 100 sec/batch.)
Has anybody experienced this?
I tried this on V100 instance on AWS so it shouldn't be a hardware problem..
CUDA=10.1, pytorch=1.0

It seems that the model is grabbing the GPU when I look at nvidia-smi.

sceneNN

Thank you for your contribution!I have a question.
If I use a dataset like sence NN, Do I need to turn room into block in prepareh5. py？

Training and Inference

@pqhieu thanks for open sourcing the wonderfull work , i had few queries
Q1 have you trained the architecture on the available other dataset like semanttic Kitti and 3D dataset
Q2 If not trained can we follow the same training pipeline , if trained can you please share the pre-trained model
Q3 can we use the currently pre-trained model to test on custom dataset which less number of point cloud density

Thanks in advance

Could you tell me how to generate instance label for SceneNN dataset

@pqhieu There is only semantic label in SceneNN dataset, how to generate instance label? Thanks