bigmb/unet-segmentation-pytorch-nest-of-unets

Implementation of different kinds of Unet Models for Image Segmentation - Unet , RCNN-Unet, Attention Unet, RCNN-Attention Unet, Nested Unet

Unet-Segmentation-Pytorch-Nest-of-Unets

Implementation of different kinds of Unet Models for Image Segmentation

1. UNet - U-Net: Convolutional Networks for Biomedical Image Segmentation https://arxiv.org/abs/1505.04597

2. RCNN-UNet - Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation https://arxiv.org/abs/1802.06955

3. Attention Unet - Attention U-Net: Learning Where to Look for the Pancreas https://arxiv.org/abs/1804.03999

4. RCNN-Attention Unet - Attention R2U-Net : Just integration of two recent advanced works (R2U-Net + Attention U-Net)

5. Nested UNet - UNet++: A Nested U-Net Architecture for Medical Image Segmentation https://arxiv.org/abs/1807.10165

With Layer Visualization

1. Getting Started

Clone the repo:

git clone https://github.com/bigmb/Unet-Segmentation-Pytorch-Nest-of-Unets.git

2. Requirements

python>=3.6
torch>=0.4.0
torchvision
torchsummary
tensorboardx
natsort
numpy
pillow
scipy
scikit-image
sklearn

Install all dependent libraries:

pip install -r requirements.txt

3. Run the file

Add all your folders to this line 106-113

t_data = '' # Input data
l_data = '' #Input Label
test_image = '' #Image to be predicted while training
test_label = '' #Label of the prediction Image
test_folderP = '' #Test folder Image
test_folderL = '' #Test folder Label for calculating the Dice score

4. Types of Unet

Unet

RCNN Unet

Attention Unet

Attention-RCNN Unet

Nested Unet

5. Visualization

To plot the loss , Visdom would be required. The code is already written, just uncomment the required part. Gradient flow can be used too. Taken from (https://discuss.pytorch.org/t/check-gradient-flow-in-network/15063/10)

A model folder is created and all the data is stored inside that. Last layer will be saved in the model folder. If any particular layer is required , mention it in the line 361.

Layer Visulization

Filter Visulization

TensorboardX Still have to tweak some parameters to get visualization. Have messed up this trying to make pytorch 1.1.0 working with tensorboard directly (and then came to know Currently it doesn't support anything apart from linear graphs)

Input Image Visulization for checking

a) Original Image

b) CenterCrop Image

6. Results

Dice Score for hippocampus segmentation ADNI-LONI Dataset

7. Citation

If you find it usefull for your work.

@article{DBLP:journals/corr/abs-1906-07160,
  author    = {Malav Bateriwala and
               Pierrick Bourgeat},
  title     = {Enforcing temporal consistency in Deep Learning segmentation of brain
               {MR} images},
  journal   = {CoRR},
  volume    = {abs/1906.07160},
  year      = {2019},
  url       = {http://arxiv.org/abs/1906.07160},
  archivePrefix = {arXiv},
  eprint    = {1906.07160},
  timestamp = {Mon, 24 Jun 2019 17:28:45 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1906-07160},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

8. Blog about different Unets

In progress

unet-segmentation-pytorch-nest-of-unets's Issues

torchsummary.summary attention unet

HI!
very nice work!

when I use the attention unet, torchsummary.summary(model_test, input_size=(3, 128, 128)) can get the error:

Traceback (most recent call last):
File "pytorch_run.py", line 122, in
torchsummary.summary(model_test, input_size=(3, 128, 128))
File "/home/lix/miniconda3/envs/ptseg/lib/python3.6/site-packages/torchsummary/torchsummary.py", line 74, in summary
model(*x)
File "/home/lix/miniconda3/envs/ptseg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/lix/Documents/code/Unet-Segmentation-Pytorch-Nest-of-Unets/Models.py", line 360, in forward
x4 = self.Att5(g=d5, x=e4)
File "/home/lix/miniconda3/envs/ptseg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 495, in call
hook_result = hook(self, input, result)
File "/home/lix/miniconda3/envs/ptseg/lib/python3.6/site-packages/torchsummary/torchsummary.py", line 20, in hook
summary[m_key]["input_shape"] = list(input[0].size())
IndexError: tuple index out of range

Could you please give me a solution? I'll very appreciate about it!

RuntimeError: output with shape [1, 96, 96] doesn't match the broadcast shape [3, 96, 96]

Dice Score and the different and connection with MIou?

Hello ,thanks for you code.I have confuse about evaluation metrics,and what is the dice score ? It different and connection with MIouU

In pytorch_run.py 259 line, Why are you multiplying by x.size(0)

Question about output channel

Does output channel means number of class?
if i use output channel = 4 that means number of class = 4?

my dataloader and baseline model is different but i want to use models from this repository,any other demo of this repository work in a single notebook would help a lot,thanks in advance

question about learning_rate

In line 157th, MAX_STEP = int(1e10).
In CosineAnnealingLR(), T_Max is the number of epochs since the last restart in SGDR.
if you setup T_max = 1e10, the changes of Learning rate is very slow. It needs to change it?
Also, why did you choose this CosineAnnealingLR()？
thanks!

I am a beginner,can you give me your dataset and label file that you used in this project,thank you very much

Runtime error?

I trasnform the 1-channel images to the 3-channel ，why did this happen on my laptop?
The error prompt is as follows：

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "", line 1, in
File "", line 1, in
File "", line 1, in
Traceback (most recent call last):
File "", line 1, in
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 105, in spawn_main
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 105, in spawn_main
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 105, in spawn_main
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 114, in _main
exitcode = _main(fd)
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 114, in _main
exitcode = _main(fd) exitcode = _main(fd)
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 225, in prepare
prepare(preparation_data)
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 225, in prepare
prepare(preparation_data)
_fixup_main_from_path(data['init_main_from_path'])

_fixup_main_from_path(data['init_main_from_path'])
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 114, in _main
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 225, in prepare
prepare(preparation_data)
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 225, in prepare
run_name="mp_main")
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 263, in run_path
run_name="mp_main")
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 263, in run_path
_fixup_main_from_path(data['init_main_from_path'])
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
_fixup_main_from_path(data['init_main_from_path'])
run_name="mp_main")
File "D:\Anaconda3\envs\pytorchs\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 263, in run_path
run_name="mp_main")
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname) pkg_name=pkg_name, script_name=fname)

File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 96, in _run_module_code
pkg_name=pkg_name, script_name=fname)pkg_name=pkg_name, script_name=fname)
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 96, in _run_module_code
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 96, in _run_module_code

File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 85, in _run_code
mod_name, mod_spec, pkg_name, script_name)
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 85, in _run_code
mod_name, mod_spec, pkg_name, script_name)
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 85, in _run_code
mod_name, mod_spec, pkg_name, script_name)
File "D:\Anaconda3\envs\pytorchs\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)exec(code, run_globals)

File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\pytorch_run.py", line 105, in
File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\pytorch_run.py", line 105, in
exec(code, run_globals)
exec(code, run_globals)
File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\pytorch_run.py", line 105, in
File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\pytorch_run.py", line 105, in
torchsummary.summary(model_test, input_size=(3, 128, 128)) #三通道torchsummary.summary(model_test, input_size=(3, 128, 128)) #三通道torchsummary.summary(model_test, input_size=(3, 128, 128)) #三通道

torchsummary.summary(model_test, input_size=(3, 128, 128)) #三通道
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torchsummary\torchsummary.py", line 72, in summary
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torchsummary\torchsummary.py", line 72, in summary
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torchsummary\torchsummary.py", line 72, in summary
model(*x)
model(*x)model(*x)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call

File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
result = self.forward(*input, **kwargs)
result = self.forward(*input, **kwargs) File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 116, in forward

File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 116, in forward
File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 116, in forward
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torchsummary\torchsummary.py", line 72, in summary
model(*x)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 116, in forward
d2 = self.Up_conv2(d2)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
d2 = self.Up_conv2(d2)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
d2 = self.Up_conv2(d2)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
d2 = self.Up_conv2(d2)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 25, in forward
x = self.conv(x)
result = self.forward(*input, **kwargs) File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call

File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 25, in forward
result = self.forward(*input, **kwargs)result = self.forward(*input, **kwargs)

File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 25, in forward
File "G:\pytorchfenge\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Unet-Segmentation-Pytorch-Nest-of-Unets-master\Models.py", line 25, in forward
result = self.forward(*input, **kwargs)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
x = self.conv(x)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
x = self.conv(x)x = self.conv(x)

File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
result = self.forward(*input, **kwargs)
result = self.forward(*input, **kwargs) File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\container.py", line 92, in forward

File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
input = module(input)
input = module(input)input = module(input)input = module(input)

File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\conv.py", line 338, in forward
result = self.forward(*input, **kwargs) result = self.forward(*input, **kwargs)
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\batchnorm.py", line 83, in forward
result = self.forward(*input, **kwargs)
self.padding, self.dilation, self.groups)

File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\conv.py", line 338, in forward
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\modules\conv.py", line 338, in forward
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 326.08 MiB already allocated; 4.47 MiB free; 7.92 MiB cached)
self.padding, self.dilation, self.groups)
self.padding, self.dilation, self.groups)
RuntimeErrorRuntimeError: : CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 326.08 MiB already allocated; 4.47 MiB free; 7.92 MiB cached)CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 326.08 MiB already allocated; 4.47 MiB free; 7.92 MiB cached)

exponential_average_factor, self.eps)

File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\nn\functional.py", line 1697, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 350.08 MiB already allocated; 0 bytes free; 11.92 MiB cached)
Traceback (most recent call last):
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\utils\data\dataloader.py", line 511, in _try_get_batch
data = self.data_queue.get(timeout=timeout)
File "D:\Anaconda3\envs\pytorchs\lib\queue.py", line 172, in get
raise Empty
queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "G:/pytorchfenge/Unet-Segmentation-Pytorch-Nest-of-Unets-master/Unet-Segmentation-Pytorch-Nest-of-Unets-master/pytorch_run.py", line 256, in
for x, y in train_loader:
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\utils\data\dataloader.py", line 576, in next
idx, batch = self._get_batch()
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\utils\data\dataloader.py", line 543, in _get_batch
success, data = self._try_get_batch()
File "D:\Anaconda3\envs\pytorchs\lib\site-packages\torch\utils\data\dataloader.py", line 519, in _try_get_batch
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 14356, 9092, 1384, 1152) exited unexpectedly

The color of label and a suggestion

Hello, the author, first of all, thank you very much for the code. Excuse me, the label of the picture I input is binary. Why is the predicted picture not binary? Where do I need to change the color of the forecast picture? There is also a small suggestion, can you make the code more readable?

question with 'data_transform'

Hi, I have some doubts with 'data_transform' function. As you suggested , input image should be 3 channel image and input label should be 1 channel image , but I find you use the same data_transform function

data_transform = torchvision.transforms.Compose([
# torchvision.transforms.Resize((128,128)),
# torchvision.transforms.CenterCrop(96),
torchvision.transforms.ToTensor(),
torchvision.transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
])

for input image and input label during training. And use another function

data_transform = torchvision.transforms.Compose([
# torchvision.transforms.Resize((128,128)),
# torchvision.transforms.CenterCrop(96),
torchvision.transforms.Grayscale(),
#torchvision.transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]) ])
for input image and input label for calculating the Dice Score.

and the codes output error with shape error with those functions when i run them . So I am wondering is there a misktake with the defination and using for data_transform function ?
Thanks in advance ! Have a nice day!

where is the data?

BrokenPipeError: [Errno 32] Broken pipe

excellent work！
but i got this error during training.
environment:
win7
cuda:10.0
pytorch:1.1.0
Traceback (most recent call last):
File "", line 1, in
Traceback (most recent call last):
File "F:/chenzihao/Unet-Segmentation-Pytorch-Nest-of-Unets/pytorch_run.py", line 244, in
File "F:\chenzihao\miniconda\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "F:\chenzihao\miniconda\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "F:\chenzihao\miniconda\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "F:\chenzihao\miniconda\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
for x, y in train_loader:
File "F:\chenzihao\miniconda\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter
run_name="mp_main")
File "F:\chenzihao\miniconda\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "F:\chenzihao\miniconda\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "F:\chenzihao\miniconda\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)return _MultiProcessingDataLoaderIter(self)
File "F:\chenzihao\Unet-Segmentation-Pytorch-Nest-of-Unets\pytorch_run.py", line 244, in

File "F:\chenzihao\miniconda\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init
for x, y in train_loader:
File "F:\chenzihao\miniconda\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter
return _MultiProcessingDataLoaderIter(self)
File "F:\chenzihao\miniconda\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init
w.start()
File "F:\chenzihao\miniconda\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "F:\chenzihao\miniconda\lib\multiprocessing\context.py", line 223, in _Popen
w.start()
File "F:\chenzihao\miniconda\lib\multiprocessing\process.py", line 112, in start
return _default_context.get_context().Process._Popen(process_obj)
File "F:\chenzihao\miniconda\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "F:\chenzihao\miniconda\lib\multiprocessing\popen_spawn_win32.py", line 46, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "F:\chenzihao\miniconda\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
self._popen = self._Popen(self)
File "F:\chenzihao\miniconda\lib\multiprocessing\context.py", line 223, in _Popen
_check_not_importing_main()
File "F:\chenzihao\miniconda\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
return _default_context.get_context().Process._Popen(process_obj)

File "F:\chenzihao\miniconda\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "F:\chenzihao\miniconda\lib\multiprocessing\popen_spawn_win32.py", line 89, in init
reduction.dump(process_obj, to_child)
File "F:\chenzihao\miniconda\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe

question about attention unet

When I chose unet, I got the following error. How can I fix it?
Traceback (most recent call last):
File "/home/pharos/Pharos/Program/Att_U-Net/Unet-Segmentation-Pytorch-Nest-of-Unets-master/pytorch_run.py", line 100, in
torchsummary.summary(model_test, input_size=(3, 224, 224))
File "/home/pharos/anaconda3/lib/python3.7/site-packages/torchsummary/torchsummary.py", line 72, in summary
model(*x)
File "/home/pharos/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/pharos/Pharos/Program/Att_U-Net/Unet-Segmentation-Pytorch-Nest-of-Unets-master/Models.py", line 349, in forward
x4 = self.Att5(g=d5, x=e4)
File "/home/pharos/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 543, in call
hook_result = hook(self, input, result)
File "/home/pharos/anaconda3/lib/python3.7/site-packages/torchsummary/torchsummary.py", line 19, in hook
summary[m_key]["input_shape"] = list(input[0].size())
IndexError: tuple index out of range

hello,is label a 3 channels image?

i got this mistake ,it the label a 3 channels image?not a single channle image

Bad performance in single-channel input data

Hi! Firstly, appreciate for your great work. However, I ran into some problems while using this project.

My data is NCI ISBI 2013, which is a medical segmentation dataset and the data is 3D MRI images(in .dicom and .nrrd format). Following your instructions in the readme, I tried to split the 3D data to 2D png images, but the thing is that every slice of the 3D data is a 1-channel 2D grey image. I failed to transform the data to 3-channel images(such as using cv2.COLOR_GRAY2RGB, or stack the array to 3hw). So I changed the code from
'''
torchvision.transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
'''
to
'''
torchvision.transforms.Normalize(mean=(0.5), std=(0.5))
'''
and it works finally. But the loss is always around 0.45, and the final dice in test data after 500 epochs is about 0.06, which is very low. So can you give me some help? Appreciate it a lot!

May have future issues in input size in basic Unet

Hi, when I was implementing the pytorch model in the class U_net, I found that with some values of input size (for example 700x700), there could be a problem inside

d5 = torch.cat((e4, d5), dim=1)

where e4 and d5 may not be the same size, instead they will be different only with one. I guess is with the up conv that in some covs the calculate of size is using int(), some are larger than .5 so it increase 1.

• For people who want to solve this quick, I recommend to change the size to (400,400) or (800,800) which will work

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

I use my images to train, then show this error . I'm a beginner, could you give me some suggestion?

pytorch==1.1.0
cuda==9.0
cudnn==7.6.0
python==3.6.8

================================================================
Total params: 34,527,041
Trainable params: 34,527,041
Non-trainable params: 0

Input size (MB): 0.19
Forward/backward pass size (MB): 307.38
Params size (MB): 131.71
Estimated Total Size (MB): 439.27

Successfully created the main directory './model'
Successfully created the prediction directory './model/pred' of dice loss
Successfully created the model directory './model/Unet_D_15_4'
/home/pcsk/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:1386: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
Traceback (most recent call last):

File "", line 1, in
runfile('/home/pcsk/myfile/haima/pytorch_run.py', wdir='/home/pcsk/myfile/haima')

File "/home/pcsk/anaconda3/lib/python3.6/site-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "/home/pcsk/anaconda3/lib/python3.6/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "/home/pcsk/myfile/haima/pytorch_run.py", line 270, in
lossT.backward()

File "/home/pcsk/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)

File "/home/pcsk/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py", line 93, in backward
allow_unreachable=True) # allow_unreachable flag

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

How to prep the datasets?

How to prep the datasets? lets say the pancras mentioned on this work, originally available here https://wiki.cancerimagingarchive.net/display/Public/Pancreas-CT?

Thanks for your time.

errror while output is multiclass

I change this line into
model_test = model_unet(model_Inputs[0], 3, 5)

raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
ValueError: Target size (torch.Size([4, 1, 256, 256])) must be the same as input size (torch.Size([4, 5, 256, 256]))

my train images is 3 channels,and label images is 1 channel ,and label values is 0,1,2,3,4,could you tell me how solve this problem?

Please give dataset

About the dataset

HI!
very nice work!

I have no idea of this dataset. The link is as follows:

https://ida.loni.usc.edu/pages/access/search.jsp?tab=collection&loginKey=1568321853435385522&userEmail=949774840%40qq.com&project=ADNI&page=DOWNLOADS&subPage=IMAGE_COLLECTIONS

It just have .nii projects, I can not find .png which you are using in the test_image and test_label.
Could you please give me a very specific link? I'll very appreciate about it!

how to assure the identical transform on both image and labels

Hi, Thank for this great project. I have a question about the data augmentation. In Image_Dataset_folder class, the transforms of the image and label are separate. How to assure the same random transform of images could be used in the related labels?

Some meaningless parameters

There are some meaningless parameters in this system that I found it make no sense. For example,i_valid in pytorch_run.py file.it was used in the part of early stopping, but it makes no effect. The judge of i_valid's value was marked out.
And I don't understand the meaning of "epoch_valid " in pytorch_run.py file. Why the value of it is "epoch-2"?Could you help me understand those codes? Thank you.

question about torchsummary

hello, in line 102:torchsummary.summary(model_test, input_size=(3, 128, 128))
I get a error: Error in dlopen or dlsym: libnvrtc.so.10.0: cannot open shared object file: No such file or directory.
Is the problen about version of torchsummary?

Negative number occur in 'Loss' ...

Hi, thanks for this project first.

I train the Unet and I have found negative number occur in 'Loss' after a few epoches.
I have debug and found that negative number occur because of the bce_withlogitcs is negative...

Have you meet this problem, could you tell me how can I fix this problem.

thanks.

what is the 2d_from_3d?

How to use the code?

Test Models on Test Image Folder

I would like to test the trained models on my test image folder. It seems the code does not do that. Rather, some pre-selected images are iteratively evaluated per each epoch during training.

I would like to use the already trained model to test my Test dataset and generate segmentation maps.

Memory Errors

Nice set of models. I'm training segmentations of images of the dimension (w/ batch size) 32x3x512x512 with 4 output classes. For some reason I am getting memory errors using the nested unet. (12 Gb GPU)

Is this what you'd expect to have happen given the data size and model architecture?

About the performance on ISIC2017 dataset

Hello,
Great Work!
I have some questions about the results on ISIC2017(skin cancer segmentation dataset). I trained the model using your code, and got satisfied results on DRIVE (eye ) and Lung segmentation datasets, but I can't get normal results on ISIC 2017, the performance is very poor, the dc is only 0.34. So, Do you have any idea about the problem?

question about mutil-classes

hi! I want to train in my cus dataset! It has 12+1(background) classes . I changed the line299 to softmax and change the line of model_test = model_unet(model_Inputs[0], 3, 12) . But if has a question that :::::: ValueError: Target size (torch.Size([4, 1, 96, 96])) must be the same as input size (torch.Size([4, 12, 96, 96])) I think it's the last layer of the network that outputs a feature map for each class. But my label is single channel ，it is shape is [4,1,96,96] . How can T calculate the loss ? Thank u !!!!

Data transformation

Hello. Thanks for sharing this beautiful work.
I am using this dataset: https://www.kaggle.com/c/carvana-image-masking-challenge
When I delete cropping on transformation everything works great, but when I don't delete, predictions are being incredibly awful.
So in each iteration, I am changing my Dataloader randomly (in epoch for-loop). The batch will either will consist of cropped or not chopped pictures. Is my approach correct? What am I doing wrong do you think my images that cropped are giving me terrible results?

RuntimeError

RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

tuple index out of range

hi,bigmb
when i use AttU_Net,i got a problem:
#===================================================
Traceback (most recent call last):
File "C:/Users/lenovo/Desktop/pspnet/unet1.py", line 382, in
summary(model,(3,256,256))
File "E:\Anaconda3\envs\pytorch\lib\site-packages\torchsummary\torchsummary.py", line 72, in summary
model(*x)
File "E:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "C:/Users/lenovo/Desktop/pspnet/unet1.py", line 349, in forward
x4 = self.Att5(g=d5, x=e4)
File "E:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 549, in call
hook_result = hook(self, input, result)
File "E:\Anaconda3\envs\pytorch\lib\site-packages\torchsummary\torchsummary.py", line 19, in hook
summary[m_key]["input_shape"] = list(input[0].size())
IndexError: tuple index out of range
#===============================================================
I don't know how to solve it ,could you help me .Thank you very much! Looking forward to your reply.
Best.

how to use this with my own dataset?

i want to train it with my own dataset. Can anyone help?

about the iou of results

hello, I use my own data to train the model, however, there is little that can be done to improve the iou by tuning lr (currently the highest accuracy is 60%,lr=0.1/0.05/0.01, weight_decay=1e-8).
Is there any way to improve the performance?
looking forward to your reply！

error while training

i am using carvana dataset for training in which images are .jpg and labels are png i encountered this problem

<PIL.PngImagePlugin.PngImageFile image mode=L size=1918x1280 at 0x22C0868E7F0>
Traceback (most recent call last):
File "pytorch_run.py", line 300, in
s_label = data_transform(im_label)
File "C:\Users\vcvis\AppData\Local\Programs\Python\Python36\lib\site-packages\torchvision\transforms\transforms.py", line 61, in call
img = t(img)
File "C:\Users\vcvis\AppData\Local\Programs\Python\Python36\lib\site-packages\torchvision\transforms\transforms.py", line 164, in call
return F.normalize(tensor, self.mean, self.std, self.inplace)
File "C:\Users\vcvis\AppData\Local\Programs\Python\Python36\lib\site-packages\torchvision\transforms\functional.py", line 208, in normalize
tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
RuntimeError: output with shape [1, 1280, 1918] doesn't match the broadcast shape [3, 1280, 1918]

please guide

Single Channel Input

Which parameters do you have to change if you are working with 1-channel images instead of 3 ?

TypeError: function takes exactly 1 argument (3 given)

Hi there,

I'm relatively new to medical image segmentation and I want to use your models to experiment on some CT/MRI images and labels in NIFTI (.nii.gz) format.

I converted the 3D NIFTI images and labels to 2D images in .png format using your 2d_from_3rd.py script, but after running pytorch_run.py, I was faced with the following error:

May I know what might be the cause of this error? Thanks!

size mismatch

Hi, the code is awsome, but how can I fix this error, since I have change my image from 4 channels,to RGB

RuntimeError: output with shape [1, 530, 500] doesn't match the broadcast shape [3, 530, 500]

With reagrds, any help would be so grateful!

Request for a detailed readme so that we can train and test the models

Visualizing attention blocks

Hi,

Great job on implementing the models from Ozan Oktay. I am, unfortunately, having a bit of difficulty in interpreting how you're visualizing the attention gates (intermediate kernels/layers) in the Attention Unet model. Could you please provide some documentation on your implementation?

In particular, you lose me here:

class LayerActivations():
    """Getting the hooks on each layer"""

    features = None

    def __init__(self, layer):
        self.hook = layer.register_forward_hook(self.hook_fn)

    def hook_fn(self, module, input, output):
        self.features = output.cpu()

    def remove(self):
        self.hook.remove()

which is referenced here:

    x1 = torch.nn.ModuleList(model_test.children())
    # x2 = torch.nn.ModuleList(x1[16].children())
     #x3 = torch.nn.ModuleList(x2[0].children())

    #To get filters in the layers
     #plot_kernels(x1.weight.detach().cpu(), 7)

    #####################################
    # for images
    #####################################
    x2 = len(x1)
    dr = LayerActivations(x1[x2-1]) #Getting the last Conv Layer

    img = Image.open(test_image)
    s_tb = data_transform(img)

    pred_tb = model_test(s_tb.unsqueeze(0).to(device)).cpu()
    pred_tb = F.sigmoid(pred_tb)
    pred_tb = pred_tb.detach().numpy()

    plot_kernels(dr.features, n_iter, 7, cmap="rainbow")

How is this code grabbing the Attention-Gates in the interior of the model?

where is the dataset?

loss

hello,dice_loss may different from the “edge loss” in the U-net paper。Can you provide the same loss function as the original paper？Looking forward to your reply

Regarding Tensorboard Support Graph

Issues in Tensorboard Graph.
It works with tensorboardX with Pytorch1.0.1 , but if pytorch version is 1.1.0.
Error : " Pytorch Version is too old. "

Check line 166-170

#Working on it later.

ValueError: Shape mismatch: im1 and im2 must have the same shape.

How to save this problem?ValueError: Shape mismatch: im1 and im2 must have the same shape.

Why can't train the model on the GPU

I don't modify the code about GPU, and "CUDA is available. Training on GPU".
What's more, I have put the input and model into the cuda. So What else do I need to put in cuda?
Thanks.

Visualization issue

`
def plot_kernels(tensor, n_iter, num_cols=5, cmap="gray"):
"""Plotting the kernals and layers
Args:
Tensor :Input layer,
n_iter : number of interation,
num_cols : number of columbs required for figure
Output:
Gives the figure of the size decided with output layers activation map

Default : Last layer will be taken into consideration
    """
if not len(tensor.shape) == 4:
    raise Exception("assumes a 4D tensor")

fig = plt.figure()
i = 0
t = tensor.data.numpy()
b = 0
a = 1

for t1 in t:
    for t2 in t1:
        i += 1

        ax1 = fig.add_subplot(5, num_cols, i)
        ax1.imshow(t2, cmap=cmap)
        ax1.axis('off')
        ax1.set_xticklabels([])
        ax1.set_yticklabels([])

        if i == 1:
            a = 1
        if a == 10:
            break
        a += 1
    if i % a == 0:
        a = 0
    b += 1
    if b == 20:
        break

plt.savefig(
    './model/pred/Kernal_' + str(n_iter - 1) + '_epoch_'
    + str(i))`

Why is all these data added. It stops when I try to change something here.

how many memory does the Nest Unet require when dealing with 3D image?

I modified the network to deal with 3D medical image with a GPU(12G), but it feeds back "CUDA out of memory. Tried to allocate 1.50 GiB (GPU 0; 11.91 GiB total capacity; 9.84 GiB already allocated; 1.45 GiB free; 21.18 MiB cached)"

so how many GPU memory does the Nest Unet require when dealing with 3D image?

How to test single image

I want to know the code of how to test single image.
Thanks

module 'torch.nn.functional' has no attribute 'Sigmoid'

Hi Malav,

While I was using AttU_Net and R2ATTU_Net, it throws some error, could you help regarding this issue...

I think you might have noticed this issue, but have missed somehow in this file.

#passsing this string so that if it's AttU_Net or R2ATTU_Net it doesn't throw an error at torchSummary

>> Might be something missing here..

model_test = model_unet(model_Inputs[0], 3, 1)

model_test.to(device)

Best,
-Jimut