train with self-made dataset

[confusezius]

train with self-made dataset

[Confusezius]

You should be able to keep mosts of the pipeline intact, but you will need to adjust the Dataloader for you .mat-format, i.e. editing the __getitem()__-function in Utilities/PyTorch_Datasets.py. Regarding the weightmaps, you need to adjust the repective loading in __getitem()__ as well.

[jayZ1995]

Thanks very much for your reply. I'm still trying to understand your code so I convert mat file to numpy files instead since it's easier for me, is it acceptable for the network? I also wonder where could I define the route to input the self-made dataset? Many thx

[Confusezius]

Yes, numpy files work well, since the default dataloader already excepts numpy formatting (see L120 in Utilities/PyTorch_Datasets.py).

The network generally only accepts Torch.Tensors, but the conversion from numpy to torch is done automatically when calling the dataloader in the main script.

Depending on the layout of your dataset, you will need to either:
[1] create a CSV-file similar to what Data_Preparation/Convert_NibVolumes_To_Slices.py does, i.e. with headers "Volume" and "Slice Path", and entries for the volume name and the path to the respective volume slice, respectively, and load it in L17-18 in Utilities/Pytorch_Datasets.py.
OR
[2] In Utilities/Pytorch_Datasets.py -> Basic_Image_Dataset_2D directly create a self.volume_details - dictionary (L50) with volume names as keys and a list of paths to all respective slices as items.

[jayZ1995]

Thanks again for the detailed response. I create three csv files: Assign_2D_volume, Assign_2D_livermasks, Assign_2D_lesionmasks each with one column of file name and one column of path, I wonder if I am following you correctly and do I need to create a separate file to align volume, liver mask and lesion mask? Also could you please tell me how to create weight map for the masks I made ie which program should I run and maybe re-program?

[Confusezius]

The csv-files look correct - and depending on whether you do liver or lesion segmentation, you will also need csv-files doing the same for the respective weightmaps if you decide not to use e.g. Dice Loss but weighted crossentropy loss.

If you look at the detailed explanation in the README or [3.]-(1.) it tells you that running python Data_Preparation/Convert_NibVolumes_to_Slices3D.py will generate weightmaps as well. Look at L176-185 in the file to see the flags you can pass to the script (e.g. by running python Data_Preparation/Convert_NibVolumes_to_Slices3D.py --path_2_training_volumes <your path to your 3D volumes in .nii-format> --save_path_4_training_slices <where to save everything>).

If you have everything else and only want to generate weightmaps, just look at the find_borders()-function in the same script which computes the utilized weightmaps.

[Confusezius]

I will close this for now, feel free to reopen it if you encounter other issues :).

[jayZ1995]

I have some trouble creating weight map using Convert_NibVolumes_to_Slices.py (not the 3D version since my data are already slices). The program outputs:
Reloaded modules: network_zoo, Network_Utilities, General_Utilities, PyTorch_Datasets, Function_Library
0%| | 0/1 [00:00<?, ?it/s]
Without weightmap or csv file. I noticed that on L178 of that file explaining that txt file are read, I wonder which txt file should I adjust? Thanks :)