Apply FReLU to YOLOv5

[megvii-model]

Apply FReLU to YOLOv5

[glenn-jocher]

@clw5180 I've run some experiments with YOLOv5s, which seem to show some early improvement. I have not tried larger models, and improvements from one size to another sometimes does not extrapolate well, so I can't draw any conclusions across the entire v5 lineup, but so for small models it appears to help.

One warning is that FReLU increases training requirements greatly, similar to Swish perhaps, not quite as bad as Mish, so it's probably not a viable replacement for all activation functions in a model, but might be suitable for partial replacement.

[nmaac]

Hi @glenn-jocher ,
(1) you should use torch.max(), which is different from MegEngine API.
(2) yes you could try to use FReLU only in the backbone first, then try it in head. you could also replace all the LeakyReLU activations to FReLU because there is only one activation in each BottleneckCSP which are not originally many.

[glenn-jocher]

@nmaac great, thanks for the feedback! We'll try to experiment with this and see if it helps our results!

[clw5180]

We'll try to experiment with this and see if it helps our results!

Hi glenn, is there any improment on COCO or VOC dataset about FReLU ? Or how does it perform on the verification set? Thanks !