aim-uofa/dyco3d

dyco3d's Issues

Sometimes proposal_idx was empty

Hi, when I tried to run the inference code, I met a problem that the variable proposal_idx was empty. Could you give me any suggestion to debug the code? Thank you very much! The location of code and error information are listed below.

DyCo3D/model/pointgroup/pointgroup.py

Line 454 in 17d22c2

proposal_idx = torch.cat(proposal_idx, dim=0)

Traceback (most recent call last):
File "test_s3dis.py", line 276, in
test(model, model_fn, data_name, cfg.test_epoch)
File "test_s3dis.py", line 70, in test
preds = model_fn(batch, model, epoch)
File "/home/xiaodchen/Generalization/DyCo3D/model/pointgroup/pointgroup.py", line 613, in test_model_fn
ret = model(input_, p2v_map, coords_float, coords[:, 0].int(), batch_offsets, epoch, ins_sample_num=-1, training=False)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/xiaodchen/Generalization/DyCo3D/model/pointgroup/pointgroup.py", line 574, in forward
proposal_idx, proposal_len, scores = self.generate_proposal(mask_logits.squeeze(dim=0).sigmoid(), object_idxs,
File "/home/xiaodchen/Generalization/DyCo3D/model/pointgroup/pointgroup.py", line 454, in generate_proposal
proposal_idx = torch.cat(proposal_idx, dim=0)
RuntimeError: There were no tensor arguments to this function (e.g., you passed an empty list of Tensors), but no fallback function is registered for schema aten::_cat. This usually means that this function requires a non-empty list of Tensors. Available functions are [CPU, CUDA, QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradNestedTensor, UNKNOWN_TENSOR_TYPE_ID, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

supplementary materials not found~

Thank you for your outstanding work!
Where can I find your supplementary materials? I want to learn about the difference between your clustering algorithm and pointgroup~

Problem when reproducing the result

Thank you for sharing the code. When I re-trained the model on the training set of ScanNetv2 (masking head size of 16) with your default config then tested it on the validation set, the result is lower than the result of your provided pretrained model (see image below).

So I have some concerns about the implementation details:

1. In the paper, the cluster radius is set to 0.03 but in your code is 0.02.
2. In the paper, you used the dice loss to train the semantic segmentation but I have not seen it in your code.
3. I trained the model with batch_size = 6 on a single V100 GPU. Does the batch_size strongly affect the performance?

Could you give me some advices？ Thanks.

Error: python setup.py bdist_wheel

I met several problems when I tried to compile spconv.
List them below for someone encounters the same problem.

Environment:

• Python 3.7.0
• Pytorch 1.1.0
• CUDA 10.1

1. pybind11 does not contain a CMakeLists.txt file.-- Configuring incomplete, errors occurred!

• solution:
  Clone the file folder from here to lib/spconv/third_party/pybind11

---

2. fatal error: boost/geometry.hpp: No such file or directory

• solution:
  Install the boost with following command:

sudo apt-get install libboost-filesystem-dev
sudo apt-get install libboost-dev

In this case, it seems that the $INCLUDE_PATH$ is no need to specify manually.
Note: I reference from here and here

---

3. nms.h:140:11: error: cout is not a member of std

• solution:
  Find the nms.h and follow here

---

Other issue when compiling pointgroup_ops :

4. error: google/dense_hash_map: No such file or directory

• solution:
  Check here

apt-get install libsparsehash-dev

---

Update:

5. After compiling finished, I met the following error:
   RuntimeError: CUDA error: invalid argument

• Solution:
  uninstall pytorch and torchvision(install by pip)

conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0 -c pytorch

Then, recompile the spconv and pointgroup_ops.

---

Other issues solving reference 1
Other issues solving reference 2

THC dependency issue pointgroup_ops

pointgroup_ops/src/bfs_cluster.h includes <THC/THC.h> that leads to the failure of pointgroup_ops installation.

proposal_idx empty in validation

When I was validating on ScanNetv2, I encounterd the problem in the function generate_proposal().

[2021-07-27 11:48:44,183  INFO  test.py  line 44  41515]  >>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>                                                                                                                [585/15021]
[2021-07-27 11:48:53,470  INFO  scannetv2_inst.py  line 108  41515]  Testing samples (val): 312
/home/sysadmin/.fei/DyCo3D/model/pointgroup/pointgroup.py:432: UserWarning: This overload of nonzero is deprecated:
        nonzero()
Consider using one of the following signatures instead:
        nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729047590/work/torch/csrc/utils/python_arg_parser.cpp:882.)
  proposal_id_n = ((mask_logits[n] > threshold) + (seg_pred == inst_pred_seg_label[n].item()) > 1).nonzero().squeeze(dim=1)
Traceback (most recent call last):
  File "test.py", line 265, in <module>
    test(model, model_fn, data_name, cfg.test_epoch)
  File "test.py", line 64, in test
    preds = model_fn(batch, model, epoch)
  File "/home/sysadmin/.fei/DyCo3D/model/pointgroup/pointgroup.py", line 605, in test_model_fn
    ret = model(input_, p2v_map, coords_float, coords[:, 0].int(), batch_offsets, epoch, ins_sample_num=-1, training=False)
  File "/home/sysadmin/.conda/envs/mm3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/sysadmin/.fei/DyCo3D/model/pointgroup/pointgroup.py", line 568, in forward
    min_pts_num=50)
  File "/home/sysadmin/.fei/DyCo3D/model/pointgroup/pointgroup.py", line 445, in generate_proposal
    proposal_idx = torch.cat(proposal_idx, dim=0)
RuntimeError: There were no tensor arguments to this function (e.g., you passed an empty list of Tensors), but no fallback function is registered for schema aten::_cat.  This usually means that this function requires a non-empty li
st of Tensors.  Available functions are [CPU, CUDA, QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched
, VmapMode].

The problem is in the for loop that all proposal_id_n is a tensor of zero. However I think my dataset is well processed.

        for n in range(n_inst):
            proposal_id_n = ((mask_logits[n] > threshold) + (seg_pred == inst_pred_seg_label[n].item()) > 1).nonzero().squeeze(dim=1)
            score = mask_logits[n][proposal_id_n].mean()
            seg_label = inst_pred_seg_label[n]
            if proposal_id_n.size(0) < min_pts_num:
                continue
            proposal_id_n = batch_id[proposal_id_n.long()].unsqueeze(dim=1)
            id_proposal_id_n = torch.cat([torch.ones_like(proposal_id_n)*num, proposal_id_n], dim=1)
            num += 1
            tmp = proposal_len[-1]
            proposal_len.append(proposal_id_n.size(0)+tmp)
            proposal_idx.append(id_proposal_id_n)
            scores.append(score)

Could you give me some advice？ Thanks.

How to calculate the single AP

I want to ask how to calculate the single AP as there is no dirct score. Is that use classification and the mask quality as SOLOv2? (I haven't read the code of CondIns)

Implementation detail on S3DIS dataset

Hello, in the article it said the model has been tested in the S3DIS dataset, could you add the S3DIS preprocessing and testing code?

Error: python setup.py bdist_wheel

The some error occurs when I execute the command
python setup.py bdist_wheel

I see README.md that
Add the $INCLUDE_PATH$ that contains boost in lib/spconv/CMakeLists.txt. (Not necessary if it could be found.)

But I don't understand boost?

running bdist_wheel
running build
running build_py
running build_ext
/home/ubuntu/de/dyco3d/lib/spconv/build/lib.linux-x86_64-3.7
Release
-- The CXX compiler identification is GNU 5.4.0
-- The CUDA compiler identification is unknown
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Check for working CXX compiler: /home/ubuntu/Downloads/yes/envs/dyco3d/bin/c++
-- Check for working CXX compiler: /home/ubuntu/Downloads/yes/envs/dyco3d/bin/c++ - broken
CMake Error at /home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/cmake/data/share/cmake-3.22/Modules/CMakeTestCXXCompiler.cmake:62 (message):
The C++ compiler

`/home/ubuntu/Downloads/yes/envs/dyco3d/bin/c++`

is not able to compile a simple test program.

It fails with the following output:

Change Dir: /home/ubuntu/de/dyco3d/lib/spconv/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make -f Makefile cmTC_0ead2/fast && /usr/bin/make  -f CMakeFiles/cmTC_0ead2.dir/build.make CMakeFiles/cmTC_0ead2.dir/build
make[1]: Entering directory '/home/ubuntu/de/dyco3d/lib/spconv/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_0ead2.dir/testCXXCompiler.cxx.o
/home/ubuntu/Downloads/yes/envs/dyco3d/bin/c++   -DVERSION_INFO=\"1.0\"  -o CMakeFiles/cmTC_0ead2.dir/testCXXCompiler.cxx.o -c /home/ubuntu/de/dyco3d/lib/spconv/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
Linking CXX executable cmTC_0ead2
/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/cmTC_0ead2.dir/link.txt --verbose=1
/home/ubuntu/Downloads/yes/envs/dyco3d/bin/c++ -DVERSION_INFO=\"1.0\"  CMakeFiles/cmTC_0ead2.dir/testCXXCompiler.cxx.o -o cmTC_0ead2
/usr/bin/ld: cannot find crt1.o: No such file or directory
/usr/bin/ld: cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status
CMakeFiles/cmTC_0ead2.dir/build.make:98: recipe for target 'cmTC_0ead2' failed
make[1]: *** [cmTC_0ead2] Error 1
make[1]: Leaving directory '/home/ubuntu/de/dyco3d/lib/spconv/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeTmp'
Makefile:127: recipe for target 'cmTC_0ead2/fast' failed
make: *** [cmTC_0ead2/fast] Error 2

CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:2 (project)

-- Configuring incomplete, errors occurred!
See also "/home/ubuntu/de/dyco3d/lib/spconv/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeOutput.log".
See also "/home/ubuntu/de/dyco3d/lib/spconv/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
File "setup.py", line 88, in
zip_safe=False,
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/init.py", line 155, in setup
return distutils.core.setup(**attrs)
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 148, in setup
return run_commands(dist)
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
dist.run_commands()
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
self.run_command(cmd)
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "setup.py", line 41, in run
self.build_extension(ext)
File "setup.py", line 71, in build_extension
subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
File "/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/subprocess.py", line 328, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/ubuntu/de/dyco3d/lib/spconv', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/ubuntu/de/dyco3d/lib/spconv/build/lib.linux-x86_64-3.7/spconv', '-DCMAKE_PREFIX_PATH=/home/ubuntu/Downloads/yes/envs/dyco3d/lib/python3.7/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.7', '-DSPCONV_BuildTests=OFF', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr"', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

No learning rate decay in the present training code.

Dear author:
Thanks for your marvelous job and the code shared in github. I just run your code, and the result is lower than that stated in the paper, and the validation result is shown in the image below:

I find out that in the present train.py, the code that realize lr decay is missing:

I wonder if the code is not right, or you put the lr decay in other place. Looking forward to your reply.

Sincere.

conda install -c daleydeng gcc-5

ERROR conda.core.link:_execute(700): An error occurred while installing package 'daleydeng::gcc-5-5.4.0-2'

when the code will release?

Problems in training

Thank you for sharing the code. I would like to ask about some of the problems I encountered when running your code. What is the upper limit for the number of iterations of the code? For data of different batch sizes, it seems that the upper limit of iteration is 30k times. Does it mean that the larger batch size will lead to more training times? I would like to ask what your specific settings are during training.

Some details in this paper

I am sorry to disturb you again because I am confused about two sentences below in this paper.
(1) During training, the ground truth for $C^z$ if it has the largest number of points in $C^z$.
What is the meaning of largets number of points? For a predicted mask, doesn’t it only contain one specific category of mask?

(2) $\mathcal 1$ is an indicator function, showing the loss is only computed on the points that have identical semantic labels with group $C^z$

From the paper, I think a cluster obtains the semantic prediction from Seg Head and contains the points which belong to same semantic label. So it does not need an indicator function to determine whether a point have the identical semantic labels with group? But it seems I misunderstood. Could you give me some advice？ Thanks.

[Bug Report] Hardcoded variables

Hi!

Thanks for open sourcing this amazing work!

I might be wrong, but I think several hardcoded variables should have been passed in through the config file. e.g. in the following example, the min_pts_num=50 should have been passed in via the cluster_npoint_thre: 50 in the config file.

DyCo3D/model/pointgroup/pointgroup.py

Lines 574 to 576 in 17d22c2

	proposal_idx, proposal_len, scores = self.generate_proposal(mask_logits.squeeze(dim=0).sigmoid(), object_idxs,
	threshold=0.5, seg_pred=semantic_preds_, inst_pred_seg_label=inst_pred_seg_label,
	min_pts_num=50)

This took me quite a moment, as small objects with fewer than 50 points were missing in the outputs, even if I turned cluster_npoint_thre in the config file to 5 : )

subprocess.CalledProcessError

subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2

How can I solve this error? Anyhelp would be greatly appreciated please.

Loss implementation different in code & paper

Hi, similar to #9 , I noticed the types of loss mentioned in the paper (L_ctr, L_mask, L_dice) are not implemented/used in the released code. Instead, the code is using exactly the same loss as PointGroup (semantic loss, offset loss, score loss).

When I checked the value in mask_logits, I found it's almost random values that are much smaller than the threshold=0.5. Therefore, I think the current code is NOT the right version that we can see the effect of the mask head design.

Could you please take a look and update the correct version? Thanks

Prepare the ScanNet dataset

Code to prepare the ScanNet dataset can not be found. Any code to prepare the train/val/test set?

prepare env guide

1. conda create -n dyco3d python==3.7.0
2. conda activate dyco3d
3. git clone https://github.com/aim-uofa/DyCo3D.git
4. git clone pybind11
5. copy pybind11 to lib/spconv/third_party/pybind11
6. cd DyCo3D
7. conda install pytorch==1.1.0 cudatoolkit=10.0
8. pip install cmake==3.14.4
9. pip install -r requirements.txt # comment cmake
10. conda install -c bioconda google-sparsehash -y
11. conda install libboost -y
12. conda install -c daleydeng gcc-5 -y
13. cd lib/spconv
14. python setup.py bdist_wheel

• if you got some libstdc++.so.6 error, try this
  • cd ~/anaconda3/envs/dyco3d/lib
  • rm libstdc++.so.6
  • ln -s libstdc++.so.6.0.xx libstdc++.so.6 # xx mean like 26/29
• if something wrong, try delete build dir in spconv
• or reinstall cmake use pip uninstall cmake && pip install cmake==3.14.4
• try export LD_LIBRARY_PATH=/home/ubuntu/anaconda3/envs/dyco3d/lib/python3.7/site-packages/torch/lib/:LD_LIBRARY_PATH to solve error I forgot about, reference
• otherwise i dont know how to solve

17. Run cd dist and pip install the generated .whl file.
18. cd lib/pointgroup_ops
19. python setup.py develop
20. if everthing is fine, you can use scannet to check
    • CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 --master_port=$((RANDOM + 10000)) train.py --config config/dyco3d_multigpu_scannet.yaml --output_path work_dirs --use_backbone_transformer

---

ImportError: /home/ubuntu/3D_MODEL/DyCo3D/lib/pointgroup_ops/PG_OP.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

This error should not be important.

---

Here is the own data guide:

1. create .pth file, the content should be x y z r g b semantic_label instance_label
2. just imitate scannet's pattern
3. then change the yaml

---

if you got some error like CUDA kernel failed : invalid configuration argument
maybe your data is empty, try to debug this line to see if args is empty

idx_shift, start_len_shift = pointgroup_ops.ballquery_batch_p(coords_ + pt_offsets_, batch_idxs_, batch_offsets_, self.cluster_radius, self.cluster_shift_meanActive) # in pointgroup.py

---

below is my env.

Problem when reproducing the result

Thank you for your code. When I trained the model using Scannet v2 on my server (batch size 16, four 3090 GPUs), I found a reduction on the peformance, which is shown below. When I tried to load your trained model for evaluation, I realized that your model is trained using 50k iterations. So, I changed the iter to 5k in the config file. However, the results is still unsatisfactory. Could you give more details about your experiments to produce the results, like the number of prepare iterations using PointGroup (i.e., the training itr for semantic segmentation), batch size and corresponding learning rate? Thank you!

How to test on my own point cloud data using DyCo3D?

My point cloud data is in pcd format, which contains many workpiece. Can I use DyCo3D to segment each workpiece with pretrained model and how to do that? I‘m a new guy in this field and looking forward to your reply.

Would you release the PointNet++ backbone model?

As title

Test on s3dis dataset

Hi,

First of all, I highly apreciate your work. It has great influence on the subject and have multiple sue posibilities.

Though, I would like to ask you regarding possible instructions on how to prepare s3dis dataset to allow your model train on it and evaluate. Could you share such instruction? From what I saw in article, this method was already tested on s3dis dataset.

Hope for your reply.

best regards