Giter Site home page Giter Site logo

Comments (82)

Wilbur529 avatar Wilbur529 commented on May 13, 2024 15

@abhigoku10 @dbolya EfficientNet is really efficient!!! I used the implementation and pretrained weights from EfficientNet-PyTorch. To validate the availability, i did two experiments:

  1. EfficientNet-B0 has similar Top-5 Acc.(93.2%) with ResNet-50(93.0%) on ImageNet. When i change the backbone of YOLACT to EfficientNet-B0, the convergence rate and mAP between them almost is same. But the total model sizes are YOLACT550-EfficientNet-B0(47MB) and YOLACT550-ResNet-50(120MB).

  2. EfficientNet-B4 has similar Top-5 Acc.(96.3%) with SENet(96.2%) on ImageNet, and its parameters size(19M) is less than ResNet-50(26M). When i change the backbone of YOLACT to EfficientNet-B4, i found it may achieve a better performance beyond the previous network. After 55 epochs training, now it have achieved 30.95 mAP on COCO test-dev, and it seems not converge completely. It is important to note that the YOLACT550-EfficientNet-B4 model size is only 101MB.

from yolact.

dbolya avatar dbolya commented on May 13, 2024 11

Sounds good! If it ends up working well, I might just make an official MobileNet-v2 version for YOLACTv1.1. That seems like it would be nice to have.

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024 4

@dbolya Hi, I have done a preliminary test on my own simple dataset. And the experiment result made me feel excited~! With the help of MobileNet-v2, I successfully reduced the model size from over 100 MB to 31 MB, but without a significant performance loss on my own dataset. Next, i will try it on MS-COCO dataset, and i hope for the fantastic performance^_^

from yolact.

IMFinethankyou avatar IMFinethankyou commented on May 13, 2024 3

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?

Thanx!!

26.8 mAP for B0, 29.6mAP for B3.
GTX 1080Ti, base YOLACT 37 FPS
EfficientNet-B0 41FPS

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024 2

THX~!
I used the implementation and the pretrained weights from this repo:https://github.com/tonylins/pytorch-mobilenet-v2

I did the experiment follow these steps bellow:

  1. Modify the network architecture by removing the final global average pooling layer and the final 1x1 conv layer;
  2. Make a custom init_backbone() method in which i translated the node name and discarded the unuseful tensor;
  3. Add a new config-dict to the 'config.py', and set some helpful parameters.

Here is my MobileNet-v2 implementation:

class MobileNetV2(nn.Module):
    def __init__(self):
        super(MobileNetV2, self).__init__()
        block = InvertedResidual
        input_channel = 32
        interverted_residual_setting = [
            # t, c, n, s
            [1, 16, 1, 1],
            [6, 24, 2, 2], # useful
            [6, 32, 3, 2], # useful
            [6, 64, 4, 2],
            [6, 96, 3, 1], # useful
            [6, 160, 3, 2],
            [6, 320, 1, 1], # useful
        ]

        # Select the output layers
        # idx-1 => 56 * 56 * 24
        # idx-2 => 28 * 28 * 32
        # idx-4 => 14 * 14 * 96
        # idx-6 => 7 * 7 * 320
        select_idxs = [1, 2, 4, 6]


        # building first layer
        input_channel = int(input_channel)
        self.conv2d = conv_bn(3, input_channel, 2)
        self.layers = nn.ModuleList()
        self.channels = []

        # building inverted residual blocks
        features = []
        for idx, (t, c, n, s) in enumerate(interverted_residual_setting):
            output_channel = int(c)

            for i in range(n):
                if i == 0:
                    features.append(block(input_channel, output_channel, s, expand_ratio=t))
                else:
                    features.append(block(input_channel, output_channel, 1, expand_ratio=t))
                input_channel = output_channel

            if idx in select_idxs:
                self.layers.append(nn.Sequential(*features))
                self.channels.append(output_channel)
                features = []

        self.backbone_modules = [m for m in self.modules() if isinstance(m, nn.Conv2d)]

        # random initial
        self._initialize_weights()

    def forward(self, x):
        x = self.conv2d(x)

        outs = []
        for layer in self.layers:
            x = layer(x)
            outs.append(x)

        return tuple(outs)

    def _initialize_weights(self):
        modules = self.modules()
        for m in modules:
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()

    def init_backbone(self, path):
        """ Initializes the backbone weights for training. """
        state_dict = torch.load(path, map_location='cpu')

        # Replace featuresXXX -> layers.x.xx(/conv2d.x) etc.
        keys = list(state_dict)
        for key in keys:
            if key.startswith('features'):
                idx = int(key.split('.')[1])
                if idx <= 17:
                    if idx == 0:
                        new_key = 'conv2d.' + key[11:]
                    elif (idx >= 1 and idx <= 3):
                        new_key = "layers.0." + str(idx - 1) + key[10:]
                    elif (idx >= 4 and idx <=6):
                        new_key = "layers.1." + str(idx - 4) + key[10:]
                    elif (idx >= 7 and idx <= 13):
                        if idx <= 9:
                            new_key = "layers.2." + str(idx - 7) + key[10:]
                        else:
                            new_key = "layers.2." + str(idx - 7) + key[11:]
                    else:
                        new_key = "layers.3." + str(idx - 14) + key[11:]

                    state_dict[new_key] = state_dict.pop(key)
                else:
                    state_dict.pop(key)
            else:
                state_dict.pop(key)


        # Note: Using strict=False is berry scary. Triple check this.nn.ModuleList()
        self.load_state_dict(state_dict, strict=False)

And there is the corresponding config:

mobilenetv2_backbone = resnet50_backbone.copy({
    'name': 'MobileNetV2',
    'path': 'xxx.pth',
    'type': MobileNetV2,
    'args': (),
    # 'transform': resnet_transform,

    'selected_layers': [1, 2, 3],
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,
    'use_pixel_scales': False,
})
yolact_mobilenetv2_coco_config = yolact_resnet50_config.copy({
    'name': 'yolact_mobilenetv2_coco',

    'backbone': mobilenetv2_backbone.copy(),

    'use_semantic_segmentation_loss': True
})

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024 2

@abhigoku10 EfficientNets and MobileNet-v3 is coming~~~

from yolact.

lkj1114889770 avatar lkj1114889770 commented on May 13, 2024 2

@abhigoku10 can you share your yolact-efficient code?

from yolact.

linshaodan avatar linshaodan commented on May 13, 2024 2

@abhigoku10 @dbolya EfficientNet is really efficient!!! I used the implementation and pretrained weights from EfficientNet-PyTorch. To validate the availability, i did two experiments:

  1. EfficientNet-B0 has similar Top-5 Acc.(93.2%) with ResNet-50(93.0%) on ImageNet. When i change the backbone of YOLACT to EfficientNet-B0, the convergence rate and mAP between them almost is same. But the total model sizes are YOLACT550-EfficientNet-B0(47MB) and YOLACT550-ResNet-50(120MB).
  2. EfficientNet-B4 has similar Top-5 Acc.(96.3%) with SENet(96.2%) on ImageNet, and its parameters size(19M) is less than ResNet-50(26M). When i change the backbone of YOLACT to EfficientNet-B4, i found it may achieve a better performance beyond the previous network. After 55 epochs training, now it have achieved 30.95 mAP on COCO test-dev, and it seems not converge completely. It is important to note that the YOLACT550-EfficientNet-B4 model size is only 101MB.

Can you please share your EfficientNet backbone?

from yolact.

dbolya avatar dbolya commented on May 13, 2024 1

Hmm, can you still do a sanity check and change self.load_state_dict(state_dict, strict=False) to self.load_state_dict(state_dict, strict=True) and see if there's anything unexpected is missing? I think even with one anchor per layer, you shouldn't get < 1 mAP on the first validation epoch.

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024 1

@Wilbur529 i was able to train the model using mobilenet architecture, thanks for sharing the reference implementation , when i tested it on images its able to detect with low accuracy but when i run it for a video the detection are not happening at all . Did u face the similar issue , should we make any changes with the code

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024 1

@abhigoku10 I let it ran on only one single GPU, so it needed long time for training. At 200 epochs, the mAP are 14.36(box) and 15.02(mask).

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024 1

I am reading both of their paper, and i haven't found a suitable pytorch implementation of them~ Once I succeed in my experiment, i will show the modified parts.

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024 1

@Wilbur529 thanks for the update eagers waiting for this

from yolact.

zhangbaoj avatar zhangbaoj commented on May 13, 2024 1

What does 'selected_layers': [1, 2, 3] mean?

from yolact.

dbolya avatar dbolya commented on May 13, 2024 1

@zhangbaoj I replied in your new issue.

from yolact.

dbolya avatar dbolya commented on May 13, 2024 1

@abhigoku10 Can you make a pull request for it? Then we can all review it.

from yolact.

xscjun avatar xscjun commented on May 13, 2024 1

@Wilbur529
I meet the warning " Warning: Moving average ignored a value of inf " During the training , when I change the backbone of resnet to mobilenetv2 . Did you met this warning ?

from yolact.

dbolya avatar dbolya commented on May 13, 2024 1

@xscjun I have a patch coming as a result of #222 that might fix that issue for you.

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024 1

Thanks!
And what about the anchors? How can i know the best settings?

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024 1

@Auth0rM0rgan i was able to implement efficient net as backbone the performance was lower compared to resnet for objects like pedistrains and the fps was higher than resent 50 , i just used the same pred_scales, ascpect ratios similar to mobile net not sure is that the right approach , i could not share the implementation due to firm polices , @Wilbur529 @dbolya can you suggest 'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,
'use_pixel_scales': False, is this the rigthvalues selected

from yolact.

dbolya avatar dbolya commented on May 13, 2024

That would be a good test, though I will warn that we've tried VGG16+FPN before and got sub-par performance (though maybe that's to be expected). If you want to change out the backbone head over to backbone.py and check out how I implemented Resnet and VGG. If you need any help, let me know.

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

@dbolya Hi, could you show me your train log file of resnet&coco?

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

This is my recently train log with mobilenetv2&coco...

[ 16] 59810 || B: 2.489 | C: 3.295 | M: 2.861 | S: 0.897 | T: 9.542 || ETA: 19 days, 23:21:26 || timer: 2.437

[ 16] 59820 || B: 2.505 | C: 3.321 | M: 2.840 | S: 0.898 | T: 9.565 || ETA: 19 days, 22:54:13 || timer: 2.125
[ 16] 59830 || B: 2.500 | C: 3.299 | M: 2.802 | S: 0.910 | T: 9.511 || ETA: 19 days, 22:34:59 || timer: 1.966
[ 16] 59840 || B: 2.504 | C: 3.271 | M: 2.787 | S: 0.900 | T: 9.461 || ETA: 19 days, 22:51:36 || timer: 2.071
[ 16] 59850 || B: 2.480 | C: 3.282 | M: 2.751 | S: 0.881 | T: 9.394 || ETA: 19 days, 22:43:14 || timer: 2.415
[ 16] 59860 || B: 2.462 | C: 3.247 | M: 2.752 | S: 0.883 | T: 9.344 || ETA: 19 days, 22:27:50 || timer: 2.144
[ 16] 59870 || B: 2.460 | C: 3.247 | M: 2.726 | S: 0.881 | T: 9.315 || ETA: 19 days, 22:22:21 || timer: 2.091
[ 16] 59880 || B: 2.489 | C: 3.290 | M: 2.696 | S: 0.891 | T: 9.366 || ETA: 19 days, 22:38:05 || timer: 2.101
[ 16] 59890 || B: 2.478 | C: 3.300 | M: 2.712 | S: 0.892 | T: 9.382 || ETA: 19 days, 22:48:17 || timer: 2.301
[ 16] 59900 || B: 2.485 | C: 3.299 | M: 2.711 | S: 0.889 | T: 9.384 || ETA: 19 days, 22:53:21 || timer: 2.046
[ 16] 59910 || B: 2.514 | C: 3.316 | M: 2.724 | S: 0.880 | T: 9.435 || ETA: 19 days, 23:27:14 || timer: 1.973
[ 16] 59920 || B: 2.505 | C: 3.308 | M: 2.721 | S: 0.877 | T: 9.410 || ETA: 19 days, 23:25:19 || timer: 2.257
[ 16] 59930 || B: 2.496 | C: 3.290 | M: 2.710 | S: 0.863 | T: 9.360 || ETA: 19 days, 23:15:29 || timer: 2.061
[ 16] 59940 || B: 2.467 | C: 3.269 | M: 2.694 | S: 0.864 | T: 9.293 || ETA: 19 days, 22:23:27 || timer: 1.924
[ 16] 59950 || B: 2.473 | C: 3.249 | M: 2.702 | S: 0.868 | T: 9.292 || ETA: 19 days, 22:55:27 || timer: 2.374
[ 16] 59960 || B: 2.497 | C: 3.261 | M: 2.702 | S: 0.858 | T: 9.318 || ETA: 19 days, 23:18:36 || timer: 2.055
[ 16] 59970 || B: 2.532 | C: 3.274 | M: 2.726 | S: 0.862 | T: 9.393 || ETA: 19 days, 23:51:10 || timer: 2.111
[ 16] 59980 || B: 2.493 | C: 3.262 | M: 2.723 | S: 0.861 | T: 9.339 || ETA: 19 days, 23:32:02 || timer: 1.991
[ 16] 59990 || B: 2.500 | C: 3.253 | M: 2.718 | S: 0.860 | T: 9.331 || ETA: 19 days, 23:44:44 || timer: 3.396

The loss value looks like going down. But the convergence speed looks so slowly. And i got a poor evaluated result:

box | 0.78 | 2.17 | 1.76 | 1.39 | 1.08 | 0.75 | 0.43 | 0.19 | 0.07 | 0.01 | 0.00 |
box | 1.31 | 3.44 | 2.92 | 2.37 | 1.78 | 1.31 | 0.78 | 0.33 | 0.14 | 0.04 | 0.00 |
box | 1.47 | 3.88 | 3.23 | 2.54 | 2.05 | 1.47 | 0.82 | 0.44 | 0.21 | 0.04 | 0.00 |
box | 2.17 | 5.26 | 4.63 | 3.84 | 3.12 | 2.26 | 1.42 | 0.77 | 0.27 | 0.11 | 0.00 |
box | 2.41 | 5.79 | 5.10 | 4.26 | 3.46 | 2.54 | 1.74 | 0.82 | 0.34 | 0.06 | 0.00 |
box | 2.65 | 6.06 | 5.34 | 4.67 | 3.76 | 2.90 | 1.96 | 1.05 | 0.51 | 0.20 | 0.03 |
box | 2.82 | 6.62 | 5.94 | 4.82 | 4.05 | 2.94 | 2.04 | 1.17 | 0.53 | 0.12 | 0.00 |

mask | 0.80 | 1.91 | 1.61 | 1.35 | 1.09 | 0.83 | 0.58 | 0.38 | 0.21 | 0.07 | 0.00 |
mask | 1.44 | 3.13 | 2.71 | 2.28 | 1.95 | 1.62 | 1.20 | 0.81 | 0.55 | 0.12 | 0.01 |
mask | 1.73 | 3.73 | 3.28 | 2.81 | 2.35 | 1.88 | 1.43 | 0.96 | 0.57 | 0.26 | 0.03 |
mask | 2.38 | 4.86 | 4.39 | 3.80 | 3.27 | 2.65 | 2.09 | 1.49 | 0.95 | 0.29 | 0.03 |
mask | 2.76 | 5.46 | 5.04 | 4.48 | 3.84 | 3.15 | 2.43 | 1.70 | 0.96 | 0.44 | 0.06 |
mask | 3.01 | 5.81 | 5.34 | 4.82 | 4.16 | 3.53 | 2.72 | 1.97 | 1.10 | 0.56 | 0.07 |
mask | 3.31 | 6.41 | 5.85 | 5.27 | 4.56 | 3.94 | 2.99 | 2.13 | 1.31 | 0.58 | 0.06 |

Hope to discuss more with you ^_^

from yolact.

dbolya avatar dbolya commented on May 13, 2024

Hmm with the mAP starting that low (< 1), it looks likes the behavior we get when we train from scratch. Are you using imagenet pretrained weights in your MobileNet implementation? If not, it'll take way longer to converge.

There's also a gotcha when loading pretrained weights because I had to set strict=False in the torch.load for loading backbone weights. Set that back to True if you are using pretrained weights but suspect it's not loading properly.

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

haha, i found the problem...Just after i commented...
When i do test on my own dataset, i removed some other anchors on purpose... and i forgot to recover them.

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

Thx, i'm just going to do it!

from yolact.

andeyeluguo avatar andeyeluguo commented on May 13, 2024

So,what's the accuracy and fps with mobilenet v2 now?@Wilbur529

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

@andeyeluguo
It has run almost 200 epochs now, and the mAP are 14.36(box) and 15.02(mask) respectively. But I found that the model does not converge completely, so keep waiting...

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 wonderful work on the modification on the network backbone . keep going !!!!!!!!!!

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 hi is it possible to share your mobilenet implementation py ?? how good is the accuracy and fps compared to resenet 50,101

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

@abhigoku10 You could find the model and the accuracy in my previous answers, and i will do a benchmark test later. As far as I can see from the training log, i think it would be vary fast.

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

@abhigoku10 Good job! I think it may not be a difficult problem. If you couldn't find the error code, why not write a for-loop upon your image inference function to simulate the video inference:)

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 yup that was my next step , but the video execution uses threading and mutiframe buffer functionality so just trying to check the feasibility on that , but did u face the same issues with the video

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

@abhigoku10 Sry i did not face this problem. Based on your description, i think the model has no mistake. So just pay more attention to the evalvideo method, and check the postprocess procedure, good luck for u~

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 sure i shall do it , how many iterations you had trained ur mobilenet model , since i trained with 2 lakh iterations the results on the validation set is not that great though the fps is high

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 thanks for the info i shall run it to 200 epochs and share my mAP, anyother architecture u tried besides mobilenet

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 cool any plans on sharing the reference code

from yolact.

pandamax avatar pandamax commented on May 13, 2024

I am reading both of their paper, and i haven't found a suitable pytorch implementation of them~ Once I succeed in my experiment, i will show the modified parts.

Looking forward for your update~there is a reference that may be helpful

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 can you share the reference code for the efficient net so that i can test it on different data , what is the fps and accuracy u achieved

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 wonderful experimentation and anlaysis , would you be able to share the yolact-efficient code , i would be able to perform certain more experiments , since effcient-b0 model size is an interesting one
THnaks in advance

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@Wilbur529 @dbolya i have trained the model with efficient b0 and b4 here is my following observations .

  1. Efficient b0 is of model size of 42MB its mAP is similar to Resnet 18 which is of size 72MB but the performance of b0 for multiframe=2 is 20 fps but for resent18 with multiframe=2 is 22 fps for a resolution of 1280x720, which is strange since the model size and params was less but we getting less fps compared to resnet18
  2. Similary b4 is of model size of 96MB if of 16fps for multiframe=2 but Resnet 50 which is of size 125MB is of 18fps for multiframe=2
    I am testing on 2080TI gpu system . any suggestions on this behaviour

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

@abhigoku10 I may suggest you only measure the network inference time:)

from yolact.

dbolya avatar dbolya commented on May 13, 2024

@abhigoku10 @Wilbur529 Yeah if you can, use the included --benchmark mode. Also, that might just be your implementation being slow, but having fewer parameters doesn't necessarily mean you're going to be faster if evaluation is more complex.

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@dbolya i shall check up my implementation once again , i agree its not directly proportional that fewer params higher accuracy but is it not higher fps

from yolact.

Jackerz312 avatar Jackerz312 commented on May 13, 2024

@Wilbur529 @abhigoku10 can you help me with the efficient backbone? i trained efficientnet-b3 but i only have 30.05 box and 28.13 mask (using pretrained weights). I think my problem is in the config file, can you share that with me pls?

pd: i'm using resnet50 config copy and [3,4,5] selected layers.

Thx :)

from yolact.

 avatar commented on May 13, 2024

@dbolya Any word regarding integrating a new backbone into yolact?

from yolact.

dbolya avatar dbolya commented on May 13, 2024

@deepseek I didn't intend to add any new backbones to the master branch, but I guess if enough people want mobile net or efficient net, I could try. It should be relatively simple to add new backbones yourself if you look at how they're defined in backbone.py and how their configs are set up.

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@dbolya i have an efficient net cod but i wanted to review it is it possbile to review it before merging into the branch @Jackerz312 and @deepseek once is get it reviewed i can share it

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

@Wilbur529 ,
Can you please share your EfficientNet backbone?
How can i use it in me network?

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

What the FPS did you get?
On which GPU?

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

@IMFinethankyou , can you share you'r config and backbone file?
I don't know how to do it...

from yolact.

IMFinethankyou avatar IMFinethankyou commented on May 13, 2024

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

@IMFinethankyou , i tried this one, but YOLACT need more functions/fields like:
add_layer()
layers

from yolact.

IMFinethankyou avatar IMFinethankyou commented on May 13, 2024

@IMFinethankyou , i tried this one, but YOLACT need more functions/fields like:
add_layer()
layers

@sdimantsd , add_layer() is used in the backbone construction process, not neccessary in EfficientNet, and the #layers# only used in the load_weights() function for compatability of different version of backbone weights, also not necessary for EfficientNet

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?

Thanx!!

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@IMFinethankyou i performed the same thing but i had got same accuracy for B4 , as @sdimantsd was asking did u re-generate the anchors ?? i used the mobile net anchors . Can you share what changes you have made in the config file

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

@IMFinethankyou, I don't know what im doing wrong, but i steel can't manage EfficientNet to work as backbone...
I used it as backbone the following config:
efficient_net_config = yolact_base_config.copy({
'name': 'efficient_net',
# 'fpn': None,
# Dataset stuff
'dataset': ws_datasetV1,
'num_classes': len(ws_datasetV1.class_names) + 1,

# Training params
'lr_steps': (280000, 600000, 700000, 750000, 850000, 1000000, 1100000, 1200000, 1300000, 1350000, 1400000, 1450000),
'max_iter': 800000000,


'masks_to_train': 300,
'max_size': 700,

'backbone': efficient_net_backbone.copy({
    'selected_layers': list(range(2, 5)),

    'pred_scales': yolact_base_config.backbone.pred_scales,
    'pred_aspect_ratios': yolact_base_config.backbone.pred_aspect_ratios,
    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': True,  # This is for backward compatability with a bug
}),

})

(i copyed from another backbone, im not shur if it's good)

When i try to train i get the following error:
AttributeError: 'EfficientNetBackbone' object has no attribute 'channels'
in yolact.py line 426:
src_channels = self.backbone.channels

If i set backvone.channels to []
It crash in line 430:
self.fpn = FPN([src_channels[i] for i in self.selected_layers])

Can you please help me?

from yolact.

IMFinethankyou avatar IMFinethankyou commented on May 13, 2024

@IMFinethankyou, I don't know what im doing wrong, but i steel can't manage EfficientNet to work as backbone...
I used it as backbone the following config:
efficient_net_config = yolact_base_config.copy({
'name': 'efficient_net',

'fpn': None,

Dataset stuff

'dataset': ws_datasetV1,
'num_classes': len(ws_datasetV1.class_names) + 1,

# Training params
'lr_steps': (280000, 600000, 700000, 750000, 850000, 1000000, 1100000, 1200000, 1300000, 1350000, 1400000, 1450000),
'max_iter': 800000000,


'masks_to_train': 300,
'max_size': 700,

'backbone': efficient_net_backbone.copy({
    'selected_layers': list(range(2, 5)),

    'pred_scales': yolact_base_config.backbone.pred_scales,
    'pred_aspect_ratios': yolact_base_config.backbone.pred_aspect_ratios,
    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': True,  # This is for backward compatability with a bug
}),

})

(i copyed from another backbone, im not shur if it's good)

When i try to train i get the following error:
AttributeError: 'EfficientNetBackbone' object has no attribute 'channels'
in yolact.py line 426:
src_channels = self.backbone.channels

If i set backvone.channels to []
It crash in line 430:
self.fpn = FPN([src_channels[i] for i in self.selected_layers])

Can you please help me?

backbone.channels is a list, contains the channels of the output feature map
for instance: you backbone output a tuple of 5 feature maps:
torch.Size([1, 64, 67, 67])
torch.Size([1, 128, 33, 33])
torch.Size([1, 176, 16, 16])
torch.Size([1, 304, 8, 8])
torch.Size([1, 512, 4, 4])
then the channels should be [64, 128, 176, 304, 512]
backbone.channels is used latter in the FPN module

from yolact.

IMFinethankyou avatar IMFinethankyou commented on May 13, 2024

@IMFinethankyou i performed the same thing but i had got same accuracy for B4 , as @sdimantsd was asking did u re-generate the anchors ?? i used the mobile net anchors . Can you share what changes you have made in the config file

I borrowed the config from the base-config, no special changes. the only change i made is set USE_SQUARE_ANCHORS to False.

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

@IMFinethankyou, thanks, i check it now.
How can i know the backbone output?

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

also, init_backbone() is not exist :-(

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

I don't know how you did it, Also backbone_modules is missing...

from yolact.

sdimantsd avatar sdimantsd commented on May 13, 2024

@IMFinethankyou Can you help me plz?

from yolact.

IMFinethankyou avatar IMFinethankyou commented on May 13, 2024

@IMFinethankyou Can you help me plz?

@sdimantsd, you should modify the efficientnet according to the backbone.py, so it can fit into the YOLACT. my connection to github is rediculously slow.

from yolact.

linshaodan avatar linshaodan commented on May 13, 2024

@IMFinethankyou , can you share you'r config and backbone file? thks

from yolact.

IMFinethankyou avatar IMFinethankyou commented on May 13, 2024

from yolact.

eslambakr avatar eslambakr commented on May 13, 2024

@dbolya Hello thanks for your amazing work. When I change the backbone to MobileNet by the following the above code which is provided by @Wilbur529, I got this error:

RuntimeError: copy_if failed to synchronize: device-side assert triggered

And sometimes I got Nan and inf in the loss However I am working with the latest version with has this problem was solved by #222 (Note: I got it only when changing the backbone to mobilenet)

@Wilbur529 @abhigoku10 Thank you for providing us by a reference code for MobilnetV2 backbone but unfortunately I doesn't work for me as I mentioned above so did you face the above problems and could help?

Are there any changes than the above provided code?

Thanks in advance.

from yolact.

eslambakr avatar eslambakr commented on May 13, 2024

I solved the above issue by lowering the learning rate to 1e-4 instead of 1e-3.
I mentioned the solution to help others who may face the same issue.

But I faced another problem now, the mAP on the validation dataset (I used a custom dataset) as shown in the below figure
mAP_mobilenetv2

Note that: The training is working fine when I used the base backbone, the problem appears only when I changed the backbone to mobilenet.

from yolact.

eslambakr avatar eslambakr commented on May 13, 2024

Hello,
@Jackerz312 @Wilbur529 @abhigoku10
could you provide us with your implementation to Efficient-net backbone as It seems I couldn't port it in right way I followed this implementation https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py

I have a couple of questions @dbolya which may help me in porting it in the right manner
1- How should I set this values? will they be affected by changing the backbone?

'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,

2- Regarding the output I set it like that self.select_idxs = [1, 2, 4, 6]

3- should I take care of any other configuration while porting a new backbone?

For your information currently the loss is stack at certain high value and the mAP is zero

Thanks in advance.

from yolact.

linshaodan avatar linshaodan commented on May 13, 2024

@IMFinethankyou @Jackerz312 @Wilbur529 @abhigoku10
I use the efficientnet as the backbone show this errors :
Traceback (most recent call last):
File "train.py", line 532, in
train()
File "train.py", line 322, in train
losses = net(datum)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
output = module(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "train.py", line 147, in forward
preds = self.net(images)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/datasets/yolact-master/yolact.py", line 600, in forward
proto_out = self.proto_net(proto_x)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 256 16 3 3, expected input[2, 40, 69, 69] to have 16 channels, but got 40 channels instead

the config as follow:
yolact_EfficientNet_mydataset_config = yolact_base_config.copy({
'name': 'yolact_EfficientNet',

'dataset': my_custom_dataset,
'num_classes': len(my_custom_dataset.class_names) + 1,

'fpn': None,
'max_iter': 120000,
'lr_steps': (60000, 100000),

#'mask_proto_src': 0,

'backbone': EfficientNet_backbone.copy({
    'selected_layers': list(range(1, 4)),
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,

    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': False,  # This is for backward compatability with a bug

}),

})
can you help me,please?
Can you please share your EfficientNet backbone code?
thks in advance

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

I solved the above issue by lowering the learning rate to 1e-4 instead of 1e-3.
I mentioned the solution to help others who may face the same issue.

But I faced another problem now, the mAP on the validation dataset (I used a custom dataset) as shown in the below figure
mAP_mobilenetv2

Note that: The training is working fine when I used the base backbone, the problem appears only when I changed the backbone to mobilenet.

The gap between box mAP and mask mAP is not abnormal. You may find the answer in the author'
s new papar YOLACT++.

from yolact.

Wilbur529 avatar Wilbur529 commented on May 13, 2024

@IMFinethankyou @Jackerz312 @Wilbur529 @abhigoku10
I use the efficientnet as the backbone show this errors :
Traceback (most recent call last):
File "train.py", line 532, in
train()
File "train.py", line 322, in train
losses = net(datum)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
output = module(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "train.py", line 147, in forward
preds = self.net(images)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/datasets/yolact-master/yolact.py", line 600, in forward
proto_out = self.proto_net(proto_x)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 256 16 3 3, expected input[2, 40, 69, 69] to have 16 channels, but got 40 channels instead

the config as follow:
yolact_EfficientNet_mydataset_config = yolact_base_config.copy({
'name': 'yolact_EfficientNet',

'dataset': my_custom_dataset,
'num_classes': len(my_custom_dataset.class_names) + 1,

'fpn': None,
'max_iter': 120000,
'lr_steps': (60000, 100000),

#'mask_proto_src': 0,

'backbone': EfficientNet_backbone.copy({
    'selected_layers': list(range(1, 4)),
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,

    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': False,  # This is for backward compatability with a bug

}),

})
can you help me,please?
Can you please share your EfficientNet backbone code?
thks in advance

It seems like you did not change the channels num properly. You can check the first few layers in the protonet.

from yolact.

Auth0rM0rgan avatar Auth0rM0rgan commented on May 13, 2024

Hey @Wilbur529, Have you tried EfficientNet as a backbone? if yes, what performance and FPS have you achieved compare to ResNet50? Are you planning to share your EfficientNet backbone?
Also, I have a question about your MobileNet backbone that what is the intuition behind of selecting these layers select_idxs = [1, 2, 4, 6] not other layers?

Thanks!

from yolact.

linshaodan avatar linshaodan commented on May 13, 2024

from yolact.

linshaodan avatar linshaodan commented on May 13, 2024

@abhigoku10
i was able to implement efficient net as backbone,but after training to 1000 steps the loss gradient cannot be reduced.i use efficient-b0 b3 b4, b4 of loss gradient connot be reduced,b0 and b3 of mAP is very low.
please help !
Thanks in advance!

from yolact.

abhigoku10 avatar abhigoku10 commented on May 13, 2024

@linshaodan even i had the same behaviour i had trained for 6lakh iterations , but my mAP was low compared to resnet50 but fps was high

from yolact.

linshaodan avatar linshaodan commented on May 13, 2024

@abhigoku10
Yes, I got the same results as you said.
thank you.

from yolact.

Serge-weihao avatar Serge-weihao commented on May 13, 2024

Can you tell me what was the FPS on one of the "default" YOLACT backbone?

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?
Thanx!!

26.8 mAP for B0, 29.6mAP for B3.
GTX 1080Ti, base YOLACT 37 FPS
EfficientNet-B0 41FPS

@IMFinethankyou could you tell me the mAP for val set and test-dev?

from yolact.

moddent avatar moddent commented on May 13, 2024

@IMFinethankyou Hi,
I modify EfficientDet as EfficientNet backbone,
but I met this:
image
during training with efficientnet-b0 backbone.
And the loss doesn't reduce after iteration 1000.
The config as shown below:
efficientnet_backbone = resnet101_backbone.copy({
'name': 'EfficientNet',
'path': 'xxx.pth',
'type': EfficientNetBackbone,
'transform': efficientnet_transform,
'selected_layers': list(range(1, 4)),
})

'backbone': efficientnet_backbone.copy({
'selected_layers': list(range(1, 4)),
'use_pixel_scales': True,
'preapply_sqrt': False,
'use_square_anchors': False,
'pred_aspect_ratios': [[[1.414]]] * 5,
'pred_scales': [[24], [48], [96], [192], [384]],
}),

I don't know what goes wrong.
Can you share your config and backbone file or give some suggestions?
thks.

from yolact.

SpaceView avatar SpaceView commented on May 13, 2024

I made some study and changed the backbone to EfficientNet, you can have my source code at SpaceView/Yolact_EfficientNet, Please go there and try.

The source code uses EfficientNet-b0, but you can change it to any sub-model easily.

I can say that the training speed is really much slower even than the Resnet50, and I searched that this is well noted by lots of done work. Currently I don't have enough time to fine-tune this model, not to say finish the training.


I made my above comments 3 days ago, NOW I update my comments as below,

--> EfficientNet is difficult to train, not a good choice for yolact, as far as I can say. On CPU the overall speed is almost the same as Resnet50. the main reason is that the FPN, proto_net, predictions layers, etc. are the same, the backbone doesn't take such a dominant role as we have expected.

--> VGG16 is also not a good choice, it is much slower even than Resnet101, both in training and evaluation.

from yolact.

linshaodan avatar linshaodan commented on May 13, 2024

from yolact.

Willforcv avatar Willforcv commented on May 13, 2024

haha, i found the problem...Just after i commented... When i do test on my own dataset, i removed some other anchors on purpose... and i forgot to recover them.

did you remove anchors? where to recover them?

from yolact.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.