shijianjian/efficientnet-pytorch-3d

A PyTorch implementation of EfficientNet

efficientnet-pytorch-3d's Issues

pre-trained weights for 3D models

Hello, really nice package, just interested in the 3D application, will you also provide/implement loading some pre-trained models from 3D? The common use-case would be some medical MRI or CT scans... 🐰

Feature request: "include_top=True/False" to allow customization of final layers

In the original tensorflow implementation an option called include_top can be passed as an argument to the model constructor.
When set to False, the final layers of the model are ommitted, which would allow easy implementation of one's own classification layers on top of the feature extractor.

My suggestion would be to add a single line here https://github.com/shijianjian/EfficientNet-PyTorch-3D/blob/master/efficientnet_pytorch_3d/model.py#L192 such that it becomes;

if self._global_params.include_top:
        # Pooling and final linear layer
        x = self._avg_pooling(x)
        x = x.view(bs, -1)
        x = self._dropout(x)
        x = self._fc(x)

return x

The include_top argument can default to True and then can be set to False using the overide_params option in the from_name function if one would like to just get the EfficientNet as a feature extractor.

This can be added easily by changing these line https://github.com/shijianjian/EfficientNet-PyTorch-3D/blob/master/efficientnet_pytorch_3d/utils.py#L20-L24 to

# Parameters for the entire model (stem, all blocks, and head)
GlobalParams = collections.namedtuple('GlobalParams', [
    'batch_norm_momentum', 'batch_norm_epsilon', 'dropout_rate',
    'num_classes', 'width_coefficient', 'depth_coefficient',
    'depth_divisor', 'min_depth', 'drop_connect_rate', 'image_size', 'include_top'])

and these lines https://github.com/shijianjian/EfficientNet-PyTorch-3D/blob/master/efficientnet_pytorch_3d/utils.py#L256-L282 to:

def efficientnet3d(width_coefficient=None, depth_coefficient=None, dropout_rate=0.2,
                 drop_connect_rate=0.2, image_size=None, num_classes=1000, include_top=True):
    """ Creates a efficientnet model. """

    blocks_args = [
        'r1_k3_s222_e1_i32_o16_se0.25', 'r2_k3_s222_e6_i16_o24_se0.25',
        'r2_k5_s222_e6_i24_o40_se0.25', 'r3_k3_s222_e6_i40_o80_se0.25',
        'r3_k5_s111_e6_i80_o112_se0.25', 'r4_k5_s222_e6_i112_o192_se0.25',
        'r1_k3_s111_e6_i192_o320_se0.25',
    ]
    blocks_args = BlockDecoder.decode(blocks_args)

    global_params = GlobalParams(
        batch_norm_momentum=0.99,
        batch_norm_epsilon=1e-3,
        dropout_rate=dropout_rate,
        drop_connect_rate=drop_connect_rate,
        # data_format='channels_last',  # removed, this is always true in PyTorch
        num_classes=num_classes,
        width_coefficient=width_coefficient,
        depth_coefficient=depth_coefficient,
        depth_divisor=8,
        min_depth=None,
        image_size=image_size,
        include_top=include_top
    )

    return blocks_args, global_params

How to freeze layer in this model!!

Can this model be modified with Unet3D?

Hello, Thanks for sharing you code.

I'm trying to implement Unet3D with efficientnet3D backbone.
Can you give me a tip to write codes?

Thank you.

Unbound `oz` in Conv3dDynamicSamePadding

Do you mean iz / sz instead of iz / oz?

EfficientNet-PyTorch-3D/efficientnet_pytorch_3d/utils.py

Line 114 in 3e79bcd

oh, ow, oz = math.ceil(ih / sh), math.ceil(iw / sw), math.ceil(iz / oz)

.from_pretrained for 3d inputs

Cant we use pretrained models for 3d images? I had 3d clinical images they are not too much and I wanted to use them for image classification. so I needed a kind of pretrained model. I saw repository but I couldnt understand can I use .from_pretrained for 3d or not. If it is possible can you give example by code?

Is the .from_name() method loads the pretrained weights?

Hi. I'm trying to use it on the CT scan images.
But I'm not sure about whether the function .from_name() loads some weights like imagenet or not.
Thanks in advance!

UnboundLocalError: local variable 'oz' referenced before assignment

Hi,

I'm not 100% sure but I think I found a little Issue. In line 114 of utils.py the variable 'oz' has to be changed into the variable 'sz', otherwise I get an error.

EfficientNet-PyTorch-3D/efficientnet_pytorch_3d/utils.py

Line 114 in c33efbb

oh, ow, oz = math.ceil(ih / sh), math.ceil(iw / sw), math.ceil(iz / oz)

Input shape dimensions = C x T x H x W ?

Channel x Time(or NumFrames) x Height x Width

I am attempting to load my model in the following format

Question 1:

How do input 16 frames of size 456x456 into the EfficientNet model?

I am trying to classify 16 frame snippets from video clips.

    #load model
    from efficientnet_pytorch_3d import EfficientNet3D

    model_EfficientNet3D = EfficientNet3D.from_name("efficientnet-b7", in_channels=3)
    summary(model_EfficientNet3D, input_size=(3, 16, 456, 456))

I have 16 images I want to send into the EfficientNet3D, is this possible?

A similar comment was made by @shijianjian here : #11 (comment)

"Say, change from Conv3D(kernel_size=(3, 3, 3)) to Conv3D(kernel_size=(1, 3, 3))will probably work for your case."

I am very lost here because I dont understand where to actually change this code.

I cant even find this specific code in the model file: Conv3D(kernel_size=(3, 3, 3))

Also since I am using the pip install efficientnet-pytorch for this EfficientNet3D, I am having trouble understanding how to manipulate the actual model code since its a pip install.

If I was to manually load the efficientnet-pytorch model with PyTorch, where and how would I be able to load the model weights?

Please help me in any way, this is a wonderful project and I am grateful for the contribution. Just need a bit of support on loading the model.

Question 2.

How can I use this 3D-EfficientNet model as a backbone feature extractor? I would need to export features at a certain layer instead of getting a final classification.

Thanks so much !!

How to train from scratch used myself dataset?

It's difficult for me to train the net using my 3d dataset, for example, how to load data、preprocess.
Can anyone share train script?
Thanks all.

How to set the network input size

Hello, is the network input size only 1x64x64x64?My data shape is (batchsize,1,8,112,112). How should I set it?Can anyone answer that? thank you.

Input size different to 1x64x64x64 throws Exception

Hey,
I fed input tensors of e.g. the shape 1x48x48x48 into this network package. Unfortunately, I get the error message in the image.

Can anybody tell me, how the network can be adapted to also use it with custom sized squared input tensors?
Thanks a lot in advance!

RuntimeError: expected scalar type Double but found Float

RuntimeError Traceback (most recent call last)
/tmp/ipykernel_18/944444650.py in
----> 1 train_model(model, train_loader, val_loader, criterion, optimizer, 5)

/tmp/ipykernel_18/2880739147.py in train_model(model, train_loader, val_loader, criterion, optimizer, num_epochs)
10 images, classes = item
11 optimizer.zero_grad()
---> 12 output = model(images.double())
13 loss = criterion(output, classes)
14 accelerator.backward(loss)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

/tmp/ipykernel_18/791916627.py in forward(self, x)
6 def forward(self, x):
7 # x = nn.functional.interpolate(x, size=(224,224,224), mode='trilinear')
----> 8 x = self.effnet(x)
9 return x

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/efficientnet_pytorch_3d/model.py in forward(self, inputs)
189 bs = inputs.size(0)
190 # Convolution layers
--> 191 x = self.extract_features(inputs)
192
193 if self._global_params.include_top:

/opt/conda/lib/python3.7/site-packages/efficientnet_pytorch_3d/model.py in extract_features(self, inputs)
171
172 # Stem
--> 173 x = self._swish(self._bn0(self._conv_stem(inputs)))
174
175 # Blocks

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/efficientnet_pytorch_3d/utils.py in forward(self, x)
144 def forward(self, x):
145 x = self.static_padding(x)
--> 146 x = F.conv3d(x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups)
147 return x
148

RuntimeError: expected scalar type Double but found Float

Same outputs after model.eval()

Hi, I have some issues when using model on validation data, model.eval() gives the same output for any validation data, Can you please help me with this?

how to use batch size 8 ,as I think its fixed 32 in this model...