Question about seg in S3DIS

[wangyueft]

Question about seg in S3DIS

[jediofgever]

okay so briefly;

Data generation

• I am using Gazebo simulator + simulated depth camera to generate point cloud of my object that I want to semantic segment
• I down sample pointcloud with PCL function, I adjust leafsize parameter for down sampling point cloud so that I can get a point cloud size close to 4096 points
• I crop number of points to be exactly 4096
• Since I use Gazebo simulator, I have instant access to pose and dimension of my object, so according to that information I label each of these 4096 points
• I save this labeled points as .ply file
• This cycle goes on for the number of frames that you want in your dataset in my case 8192

After I get approximately 8192 .ply files, I squeeze these .ply files to 4 .h5 files
for example the train0.h5 will have following keys and properties;

data (2048,4096,9) 
label (2048,4096) 

the script to generate .h5 training files is as follows

import h5py
import os, os.path
import numpy as np
from plyfile import PlyData, PlyElement

 
# directory that includes .ply files .ply, 1.ply ......... 8192.ply
DIR = '/home/atas/pcl_dataset/'

 
for m  in range(0,4):

    data = np.zeros((2048, 4096, 9), dtype = np.float32)
    label = np.zeros((2048,  4096),dtype = np.uint8)
 
	
    f = h5py.File('data'+str(m)+'.h5', 'w')

    for i in range(0, 2048):
	    plydata = PlyData.read(DIR + str(m*2048+i) + ".ply")
            


	    for j in range(0, 4096):
                    ## NORMALIZE POINT  CLOUD 
                    MAX_X, MAX_, MAX_Z = 4,4,4
                    MIN_X, MIN_Y, MIN_Z = -4,-4,-4

		    data[i, j] = [plydata['vertex']['x'][j], plydata['vertex']['y'][j], plydata['vertex']['z'][j], plydata['vertex']['red'][j]/255.0, plydata['vertex']['green'][j]/255.0, 
                                  plydata['vertex']['blue'][j]/255.0,(plydata['vertex']['x'][j]-MIN_X)/8.0,(plydata['vertex']['y'][j]-MIN_Y)/8.0,(plydata['vertex']['z'][j]-MIN_Z)/8.0]

                      # The label information is embedded into rgb values, so if the rgb values are all 0 that mens the point is NONobject
                    if(plydata['vertex']['blue'][j] > 0 or plydata['vertex']['red'][j] > 0 or plydata['vertex']['green'][j] > 0 ):
                        label[i,j] = 1
                        #print("found object point")
                    else:
                        label[i,j] = 0
                        



    f.create_dataset('data', data = data)
    f.create_dataset('label', data = label)

After you generate this training .h5 files, put them under dgcnn/ tensorflow/sem_seg/data and add a .txt(for example all_files.txt) file under this directory which includes path to your .h5 file
like this;

data/data_train0.h5

Training

Simply I use the following to start training under sem_seg direcotry

python3 train.py --log_dir log1

in train.py , make sure ALL_FILES points to your .txt file which includes path to .h5 files
you can see my branch which only have sem_seg
https://github.com/jediofgever/dgcnn
you can see last two commits that i changed to adapt code for myself

[MrCrazyCrab]

@jediofgever i guess ops['labels_phs'][1] is set for the training with two GPUs, so if you intent to use only one GPU, you can remove the code. Because i don't have a chance to train, so i'm not sure about if is right.
As for the .h5 files, you can refer to the article "PointNet Deep Learning on Point Sets for 3D Classification and Segmentation", the first 3 are xys, the next 3 are RGB and the last is xys which are normolized.

[MrCrazyCrab]

@jediofgever The first way is you can make up your data with gray value and the information of normalized xyz . Another way is chage the code just as you do, but i think you should pay attention to the construction of the cnn net(which you can see the code in model.py) rather than just change the 9 => 3. i incline to the first method for i don't think you are familiar with the entire code now.

[jediofgever]

I met same problem, I just removed

ops['labels_phs'][1]: current_label[start_idx_1:end_idx_1],
ops['is_training_phs'][1]: is_training}

Bu I am not sure if it is right way to solve this.
@ChenXie93 do you know what are the 9 values in .h5 files ?
I assume the first 3 are x y z but I have no information about last 6

[jediofgever]

Thanks for your info, I also don't have 2 gpus, are you trying to train network in your own data ?

[MrCrazyCrab]

@jediofgever , yeah . I change the code for 1 gpu training, and it didn't work. Anaconda does not report code errors， but the data cann't be feeded into the gpu. i cann't figure out till now. if you make it, may you share your experience?

[jediofgever]

@ChenXie93 , in my case I am trying to use my own dataset, when I removed those 2 lines I don't encounter that error anymore related to gpu, but I have problems to make placeholders to be compatible with my data type;
for example the original input_place holder is as follows ;

pointcloud_pl= (batch_size, num_points, 9)
label_pl= (batch_size, num_points)

but my point cloud does not have 9 values , only got x y z, so changed 9 => 3, but I still have error in label_pl part.

I think if you are trying to use your own data your problem also could be related to data compatibility to input placeholder

[jediofgever]

@ChenXie93, in fact I don't have much experience in CNN and not familiar to codebase to modify architecture... thanks for your suggestions, I probably will do a training just with x, y, z values and labels. I will share results here for your reference, and after that I will also try adding up gray values and normalized xyz,
and lastly what does normalized means here ? and how did you get these values ?

[MrCrazyCrab]

@jediofgever

> "Each point is represented by a 9-dim vector of XYZ, RGB and normalized location as to the room (from 0 to 1)." Cited from "PointNet Deep Learning on Point Sets for 3D Classification and Segmentation"

and you can refer some normalization methods to choose the bets one for your own data. As for the gray value, you are free to make up or replace RGB with other features . It would be ok if you process the data with the same approach, and you should know that all the time.

[jediofgever]

I understand the meaning of normalization now, it is just to compress xyz values between (0,1) with respect to some reference, appreciate your help much @ChenXie93

[jediofgever]

Hi @ChenXie93, I was able to train an object and network can perform quite ok despite my dataset being small size. Have you got it working yet I can probably share my experience so far, if you are still trying

[MrCrazyCrab]

@jediofgever , no, i still cannot train the dataset, could you share you experince on how to set the environment?

[jediofgever]

@ChenXie93 I am not sure at what point you are having problem, is that an code error ?

[MrCrazyCrab]

Thanks for your share, i have no problem in pre-processing my own data. In fact, the code itself dosen't has a error. However, when i run the code, the spyder stoped with the kernel restarted, and I found that the data were not feeded into the net. May be i need pay more attention to the code. Thank you so much for tell me the details, i 'm curious about one thing that whether you process your data through PCL? i don't know much about PCL function and i just random sample my data into 4096, so why do you need the PCL funtion to sample the data, is there any benifits to do it, more accuracy?
Sorry to ask your so many questions, i'm just eager to find a way to solve my seg problem.