josedolz/livianet

This repository contains the code of LiviaNET, a 3D fully convolutional neural network that was employed in our work: "3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study"

livianet's Issues

About the Config

Hi @josedolz, The training for my dataset works smoothly, but the results need some improvement. So I'm thinking about changing some hyper-parameters. But I am confused about some parts of the config file:

For "learning rate change Type", there seems no choices to choose from, same for the "Cost function values", "First Epoch Change LR", "Frequency Change LR".

TODO. To define some changes in the learning rate

learning Rate change Type = 0

Subvolumes (i.e. samples) sizes.

Validation equal to testing samples

sampleSize_Train = [27,27,27]
sampleSize_Test = [45,45,45]

Cost function values

0:

1:

costFunction = 0
SoftMax temperature = 1.0

========= Learning rate ==========

L1 Regularization Constant = 1e-6
L2 Regularization Constant = 1e-4

TO check

The array size has to be equal to the total number of layers (i.e. CNNs + FCs + Classification layer)

#Leraning Rate = [0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001 ]
Leraning Rate = [0.001]

First epoch to change learning rate

First Epoch Change LR = 1

Each how many epochs change learning rate

Frequency Change LR = 2

TODO. Add learning rate for each layer

Thanks!

Code for visualization of the feature map activation

Hi Jose,

I am quite curious about the figures you showed about the feature map activation. I want to implement that too. Do you have that part of code available or any idea how I can add on it. Thanks.

Directory structure for multiple modalities.

Hello,

I have been trying to run your algorithm on some nifti data that I have, but have been getting
ValueError: could not broadcast input array from shape (0,0,0) into shape (1,25,25,25)
Is this maybe because of the format of my image files? I have multiple modalities, and am not sure how to handle that (your image files appear to only have one image per subject). This is my current file setup:

train/
----| MR/
--------| flair_0.nii.gz
--------| flair_1.nii.gz
--------| flair_2.nii.gz
--------| t1_0.nii.gz
--------| t1_1.nii.gz
--------| t1_2.nii.gz
----| Label/
----| ROI/
with Label and ROI having the same structure, but prefixed with "label" and "ROI" respectively. Is this how I should be handling modalities?

About the number of samples at each subEpoch

Hi @josedolz , I'm wondering, would 1000 be too much for the number of samples? E.g. for a dataset of 5 subjects, 200 samples would be extracted from each subject in a subEpoch, but there would be many redundant samples among these 200 ones. previously I got pretty good results on LiviaNET with my own dataset, and now I'm trying to replicate it on another framework using the exact same architecture and hyperparameters but with no success. I found that this parameter would critically influence the training (loss fluctuates greatly all the time and is hard to get decreased) but am wondering why this was not an issue when using the LiviaNET.

Thanks.

A problem about NAN

Hi josedolz,
So sorry to bother you.
I am trying to use your network to do some experiment.
But I find that when I train my own model. The cost begin to become NAN after one train step.
I really don't know why. Did you meet the same problem before?
The parameter of the network is set as your paper's first architecture.(kernel 777, input 272727)
And I change the code to fit the Python3.

version of LIvianet trained for cerebellar lobules parcellsation

Dear,

It is possible to get the version of LiviaNet that was specifically trained for cerebellar lobules parcellation ?

Thanks in advance,
Best,

Maxime dieudonné

how did you got the GT?

Hi Jose,
The GT are available only on .mat without header, Could you please give me some information about the GT ? how did you made them for exemple ?

Best regards,

Maxime dieudonné

TypeError with theano

Hello,

I'm trying to run your training code with
python ./networkTraining.py ./LiviaNET_Config.ini 0

and I'm getting TypeError with theano 1.0.1 and python 2.7 and numpy 1.14.3

TypeError: ('TensorType(float32, scalar) cannot store accurately value 0.001, it would be represented as 0.001. If you do not mind this precision loss, you can: 1) explicitly convert your data to a numpy array of dtype float32, or 2) set "allow_input_downcast=True" when calling "function".', 0.001, 'Container name "None"')

that one was coming when callingmyLiviaNet3D.initTrainingParameters with myParserConfigIni.learning_rate

which I fixed with
np.cast["float32"](myParserConfigIni.learning_rate)

the same occurred with myParserConfigIni.momentumValue. Fixed likewise.

Unfortunately I'm now hitting the following error:

File "/usr/local/lib/python2.7/dist-packages/theano/gradient.py", line 1326, in access_grad_cache
term = access_term_cache(node)[idx]
File "/usr/local/lib/python2.7/dist-packages/theano/gradient.py", line 1162, in access_term_cache
new_output_grads)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 711, in L_op
return self.grad(inputs, output_grads)
File "/usr/local/lib/python2.7/dist-packages/theano/tensor/nnet/abstract_conv.py", line 2495, in grad
d_bottom = bottom.type.filter_variable(d_bottom)
File "/usr/local/lib/python2.7/dist-packages/theano/tensor/type.py", line 234, in filter_variable
self=self))
TypeError: Cannot convert Type TensorType(float32, 4D) (of Variable AbstractConv2d_gradInputs{convdim=2, border_mode='valid', subsample=(1, 1), filter_flip=True, imshp=(190, 100, 19, 19), kshp=(9, 100, 1, 1), filter_dilation=(1, 1), num_groups=1, unshared=False}.0) into Type TensorType(float64, 4D). You can try to manually convert AbstractConv2d_gradInputs{convdim=2, border_mode='valid', subsample=(1, 1), filter_flip=True, imshp=(190, 100, 19, 19), kshp=(9, 100, 1, 1), filter_dilation=(1, 1), num_groups=1, unshared=False}.0 into a TensorType(float64, 4D).

which is getting more complex and it seems to point at a general casting issue, maybe due to the newer version of theano i'm using?

Could you take a look and help me run your code?

Regards,

Visualization

Hi Jose,

Thanks you so much for your repo and it is really good. May I ask what visualization tool you used for the 3D image in the main page, it is quite fancy.

Deploy the model without ground truth

Hi @josedolz , thanks for the nice work.
I don't understand why the ground truth has to also be provided when testing (not validating) a model (in LiviaNET_Segmentation.ini). It looks like that they are used for obtaining coordinates for sampling but is that necessary (in sampling.py)? How can I use the trained model to segment an unseen image without ground truth?

Error occured while training

type error when training

Pre-/Post- processing steps

Hi, thanks for great performance. I read your papers and interested in pre-/post- processing steps. In your paper, you mentioned that

1. we used a simple preprocessing step that includes volume-wise intensity normalization.
2. As the post-processing step, we remove these small regions by keeping only the largest connected
   component from each class.

So, given an image img (depthxwidthxheight) and ROI region-separates brain and background- the volume-wise intensity normalization means img=(img-img[ROI].mean())/img[ROI].std() (PYTHON). Am I right?

For the second point, could you provide some script to implement it? I did not find it in your code.

Thanks

How to concatenate x1 and x2 in HyperDenseNet

Hi.

Studying your HyperDenseNet, I am curious about how to concatenate x1( size of x1 is 27x27x27 ) and x2( size of x2 is 25x25x25 ) in HyperDenseNet. I think that to concatenate x1 and x2, we extract 25x25x25 volumes around the center of x1. Is that right? Thanks!

About the MHD

Hi,Jose.
Recently, I counted the Dice and HD, and Dices can reach 0.9, but HD is around 3 or 4, which is so different with your result. What are the differences between HD and MHD, and is there anything I need to do when I count HD?

how to see the segmentation results after training?

Hi, this work is great! Do we have to write function ourselves to convert the segmentation results (.mat) to something we can actually see (.nifti)? Thanks.

Requirements for input size?

Hi @josedolz, I found that I got errors when my training data vary in size, e.i., (256, 256, 261), (256, 256, 277)... but shouldn't FCN allow inputs with arbitrary size? Also, do we have to extract patches for training? Thanks!

The error:

ValueError: ('could not broadcast input array from shape (19,19,19) into shape (19,19)', 'Container name "None"')

ZeroDivisionError: integer division or modulo by zero

============== EPOCH: 1/3 ================= --- SubEPOCH: 1/2 ... Get samples for subEpoch... Traceback (most recent call last): File "./networkTraining.py", line 82, in <module> networkTraining(sys.argv[1:]) File "./networkTraining.py", line 77, in networkTraining startTraining(networkModelName,configIniName) File "/Users/toni/Git/LiviaNET/src/LiviaNet/startTraining.py", line 139, in startTraining applyPadding File "/Users/toni/Git/LiviaNET/src/LiviaNet/Modules/IO/sampling.py", line 50, in getSamplesSubepoch samplesPerSubject = numSamples/len(randIdx) ZeroDivisionError: integer division or modulo by zero

It seems I have encountered a GPU-related error

EDGE DETECTION

Hello,

I am Natalia Cantavella and I was reading your code to try to identify how you segmented the edges. I am involved in a project about segmenting the putamen, caudate, and thalamus in Multiple Sclerosis. We have good dice results but our problem is that the edges are not well segmented. I was wondering if you could explain to me how could I do to segment them right.

Thanks for all.

Natalia

Pre-processing MRI images

Hi @josedolz, you mentioned in your paper that for pre-processing you applied volume-wise intensity normalization, bias-field correction and skull striping. From my understanding, this is not included in this repository as part of LiviaNet training pipeline. Is that right? If so, which methods did you used for this pre-processing?

Best,
Francisco

Probabilities contain NaN while using my nifti input

Hi, thank you so much for sharing your work and helping us out!

I have been able to run your model successfully on the initial set of images. However, I am not able to use my .nii data successfully. I get the following error:

*********************  STARTING NETWORK TRAINING *******************
*********************  STARTING TRAINING ************************
*************  Starting training model (Reading parameters) **************
--- Training model (Reading parameters...)
--- Do training in 3 epochs with 2 subEpochs each...
-------- Reading Images names used in training/validation -------------

================== Images for training ================
Image(0): MR_Img1.nii  |  GT: GT_Img1.nii  
Image(1): MR_Img2.nii  |  GT: GT_Img2.nii  
Image(2): MR_Img3.nii  |  GT: GT_Img3.nii  
Image(3): MR_Img4.nii  |  GT: GT_Img4.nii  
Image(4): MR_Img5.nii  |  GT: GT_Img5.nii  

================== Images for validation ================
Image(0): MR_Img6.nii  |  GT: GT_Img6.nii  

... Loading model from outputFiles/LiviaNet_Test/Networks/liviaTest_Epoch15
... Network architecture successfully loaded....

============== EPOCH: 16/3 =================
--- SubEPOCH: 1/2
... Get samples for subEpoch...
('len(randIdx) is ', 5)
... getting 200 samples per subject...
...Processing subject: 1. 20.0 % of the whole training set...
...Processing subject: 2. 40.0 % of the whole training set...
...Processing subject: 3. 60.0 % of the whole training set...
...Processing subject: 4. 80.0 % of the whole training set...

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/mnt/d/JupyterNotebooks/LiviaNET/src/networkTrainingLarynx.py in <module>()
    80 
    81 if __name__ == '__main__':
---> 82    networkTraining(sys.argv[1:])

/mnt/d/JupyterNotebooks/LiviaNET/src/networkTrainingLarynx.py in networkTraining(argv)
    75     # Training the network in model name
    76     print " ******************************************  STARTING NETWORK TRAINING ******************************************"
---> 77     startTraining(networkModelName,configIniName)
    78     print " ******************************************  DONE  ******************************************"
    79 

/mnt/d/JupyterNotebooks/LiviaNET/src/LiviaNet/startTraining.pyc in startTraining(networkModelName, configIniName)
   137                                                 sampleSize_Train,
   138                                                 receptiveField,
--> 139                                                 applyPadding
   140                                                 )
   141 

/mnt/d/JupyterNotebooks/LiviaNET/src/LiviaNet/Modules/IO/sampling.pyc in getSamplesSubepoch(numSamples, imageNames, groundTruthNames, roiNames, imageType, sampleSizes, receptiveField, applyPadding)
    87                                                      samplingDistribution,
    88                                                      sampleSizes,
---> 89                                                      receptiveField
    90                                                      )
    91 

/mnt/d/JupyterNotebooks/LiviaNET/src/LiviaNet/Modules/IO/sampling.pyc in getSamplesSubject(imageIdx, imgSubject, gtLabelsImage, roiMask, samplingDistribution, sampleSizes, receptiveField)
   165                                                 size = numOfSamplesToGet,
   166                                                 replace=True,
--> 167                                                 p=maskCoordsFlattened)
   168 
   169         centralVoxelsCoord = np.asarray(np.unravel_index(centralVoxelsIndexes, maskCoords.shape))

mtrand.pyx in mtrand.RandomState.choice()

ValueError: probabilities contain NaN

Dataset
|--MR (7 CT nii images MR_Img1 to MR_Img7)
|--Labels (7 CT nii images GT_Img1 to GT_Img7)
|-ROI (Empty)

I have nothing in the ROI folder.

My configuration file is :

############################################################################################################################################
#################################################        CREATION OF THE NETWORK       #####################################################
############################################################################################################################################


############## =================== General Options  =================  ################
[General]
networkName = liviaTest
# Saving Options
folderName = LiviaNet_Test


############## =================== CNN_Architecture  =================  ################
[CNN_Architecture]
numkernelsperlayer = [10,20,30,100]

# Kernels shapes:  (Note, if kernel size is equal to 1 on one layer means that this layer is fully connected)
# In this example there will be 3 conv layers and 1 fully connected layer (+ classification layer)
kernelshapes = [[3, 3, 3], [3, 3, 3], [3, 3, 3], [1]]

# Intermediate layers to connect to the last conv layer (just before the first fully connected layer)
intermediateConnectedLayers = []

# In the current implementation it does not support pooling (To be done...)
pooling_scales = [[1,1,1],[1,1,1],[1,1,1]]

# Array size should be equal to number of fully connected (FC) layers + classification layer
dropout_Rates = [0.25,0.5] 

# Non-linear activations
# Type: 0: Linear
#       1: ReLU
#       2: PReLU
#       3: LeakyReLU
activationType = 2

# TODO. Include activation type for Softmax layer
# Number of classes: background + classes to segment
n_classes = 2

# ------- Weights initialization ----------- #
# There are some ways to initialize the weights. This is defined by the variable "weight_Initialization"
# Here, there is a list of supported methods
# 0, Classic
# 1: Delving (He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." ICCV'15)
# 2: Load Pre-trained
#      ----------
# There is also the choice of which layers will be initialized with pre-trained weights. This is specified in the variable
# "load weight layers". This can be either empty (i.e. all layers will be initialized with pre-trained weights in case 
# "weight_Initialization" is 1)
weight_Initialization_CNN = 1 #was 1
weight_Initialization_FCN = 1
#load weight layers = [] # Next release
# If using pre-trained models, specify the folder that contains the weights and the indexes of those weights to use
# To ease the transfer between different softwares (i.e matlab for instance), and between different architectures,
# the weights for each layer should be stored as a single file.
# Right now weights have to be in .npy format
weights folderName = /mnt/d/JupyterNotebooks/LiviaNET/trainedWeights
# Same length as conv layers
weights trained indexes = [0,1,2] 
#weight_Initialization_Sec = 1

############## =================== Training Options  =================  ################
[Training Parameters]
#n_epochs=20
batch_size=10
number Of Epochs = 3 
number Of SubEpochs = 2 
number of samples at each SubEpoch Train = 1000
# TODO. To define some changes in the learning rate
learning Rate change Type = 0
# Subvolumes (i.e. samples) sizes.
# Validation equal to testing samples
sampleSize_Train = [25,25,25]
sampleSize_Test  = [45,45,45]

# Cost function values
# 0:
# 1:
costFunction = 0 
SoftMax temperature = 1.0
#### ========= Learning rate ========== #####
L1 Regularization Constant = 1e-6
L2 Regularization Constant = 1e-4

# TO check
# The array size has to be equal to the total number of layers (i.e. CNNs + FCs + Classification layer)
#Leraning Rate = [0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001 ]
Leraning Rate = [0.001]
# First epoch to change learning rate
First Epoch Change LR = 1
# Each how many epochs change learning rate
Frequency Change LR = 2
# TODO. Add learning rate for each layer

#### ========= Momentum ========== #####
# Type of momentum
# 0: Classic
# 1: Nesterov
Momentum Type = 1
Momentum Value = 0.6
# Use momentum normalized?
momentumNormalized = 1

#### ======== Optimizer ===== ######
# Type: 0-> SGD
#       1-> RMSProp (TODO. Check why RMSProp complains....)
Optimizer Type = 1

#In case we chose RMSProp
Rho RMSProp = 0.9
Epsilon RMSProp = 1e-4

# Apply Batch normalization
# 0: False, 1: True
applyBatchNormalization = 1
BatchNormEpochs = 20

# Apply padding to images
# 0: False, 1: True
applyPadding = 1

############################################################################################################################################
#################################################            TRAINING VALUES           #####################################################
############################################################################################################################################

[Training Images]
#here
imagesFolder = /mnt/d/JupyterNotebooks/LiviaNET/Dataset2/MR/ 
GroundTruthFolder = /mnt/d/JupyterNotebooks/LiviaNET/Dataset2/Label/
# ROI folder will contain the ROI where to extract the pacthes and where to perform the segmentation.
# Values of the ROI should be 0 (non interest) and 1 (region of interest)
#ROIFolder = /mnt/d/JupyterNotebooks/LiviaNET2/Dataset/ROI/
# If you have no ROIs
ROIFolder = []
# Type of images in the dataset
#     0: nifti format
#     1: matlab format
# IMPORTANT: All the volumes should have been saved as 'vol'
imageTypes = 0

# Indexes for training/validation images. Note that indexes should correspond to the position = inex + 1 in the folder,
# since python starts indexing at 0
indexesForTraining = [0,1,2,3,4]
indexesForValidation = [5]

Can you please help me out with what is happening and how I could go about fixing this? Thanks.

Error when applying custom dataset

Hi, your work is fantastic! I want to apply it on the MICCAI dataset but got this error:

============== EPOCH: 1/5 =================
--- SubEPOCH: 1/2
... Get samples for subEpoch...
... getting 200 samples per subject...
...Processing subject: 1. 20.0 % of the whole training set...
...Processing subject: 2. 40.0 % of the whole training set...
...Processing subject: 3. 60.0 % of the whole training set...
...Processing subject: 4. 80.0 % of the whole training set...
...Processing subject: 5. 100.0 % of the whole training set...
Traceback (most recent call last):
File "./networkTraining.py", line 82, in
networkTraining(sys.argv[1:])
File "./networkTraining.py", line 77, in networkTraining
startTraining(networkModelName,configIniName)
File "/Users/elaine/Desktop/LiviaNET-master/src/LiviaNet/startTraining.py", line 147, in startTraining
myLiviaNet3D.trainingData_y.set_value(gt_samplesAll, borrow=True)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/compile/sharedvalue.py", line 146, in set_value
self.container.value = new_value
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/gof/link.py", line 479, in set
self.storage[0] = self.type.filter(value, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/tensor/type.py", line 150, in filter
converted_data = theano._asarray(data, self.dtype)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/misc/safe_asarray.py", line 34, in _asarray
rval = np.asarray(a, dtype=dtype, order=order)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py", line 474, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: ('could not broadcast input array from shape (19,19,19) into shape (19,19)', 'Container name "None"')

The files are in Nifti. Thanks in advance!

About the perfermance of results and sampling

Hi, Jose. Thanks for your guide and I had run the example code successfully using the given data, but the result was not satisfactory. Here is the Testing result I got:

And this is the result using my own data:

Should the results be like these above using the example code ? Or there is something wrong in my training?

Another question is about sampling. In your paper, it said that the segment size is 272727, and at each subepoch, a total of 500 samples were randomly selected from the training image segments. Is the segment size equals to sample size? (I didn't find anything about segment in your code, just sampling[252525] ). And the 500 samples should be selected from several MR/GT images, it's that right?

Questions about HyperDenseNet configuration

Hi.

Continuously studying your HyperDenseNet, I have a few questions.

1. Are T1 and Flair concatenated before the input of conv_1 in HyperDenseNet?

2. Also are T1 and Flair used for the input of all subsequent layers?

3. Is order important when concatenating outputs of all preceding layers from both paths?

4. The initial learning rate to 0.001 is reduced by a factor of 2 after every 5 epochs ( starting from epoch 10 ). So, when epoch is 10, the learning rate is 0.001*0.5. Next when epoch is 15, the learning rate is 0.001 * 0.5 * 0.5. Is that way right? also Is that epoch subepoch?

Thank you!

Dropout value fix to 0.0 and 0.5 in fully connected and softmax layers

Hi guys, I just discovered there is a bug in the public version of the code. Dropout rates in the fully connected and softmax layers are fix and set to 0.0 and 0.5, respectively. (LiviaNET.py file). As soon as I can I will correct this so that dropout rates can be read from the config file, and commit the new files with these modifications.

Transfer code to py3

py2 stopped upgrading, and have you try to transfer code to py3?

How do you make a whole prediction with patches?

Hi, your work is great. I'm curious to know do you stitch patches at the end? Otherwise you should only get the probability map for that single patch, right?