[Question] ResNet pre actiovation blocks

[pi-tau]

[Question] ResNet pre actiovation blocks

[phlippe]

Hi @pi-tau, in general, the models presented here are based on the common 'modern' implementations of them and not necessarily 1-to-1 as in the original papers. Regarding your specific questions:

• Activation before first ResNet block: that is true, if we would apply an activation there, the first block would essentially apply a BN+ReLU on an output of just another ReLU. This is not recommended since you zero out a lot more neurons and does not have any apparent benefit (at least to me). In most current 'modern' Pre-ResNet implementations, you don't have this layer either.
• Activation after last ResNet block: you are right that sometimes an additional activation can be used before the pooling layer. However, I often see that if you do pooling, an activation function is often not needed. I had actually trained multiple versions of the ResNet with the extra final activation, and commonly got worse results. Thus, I decided to stick with the one without activation here.
• Downsample layers: in general, you only apply the self.downsample part when you need to change the feature dimension. For normal ResBlocks with same input+output features, no 1x1 conv is needed.

[pi-tau]

Hi, this is quite interesting that the extra final activation worsens results. Thanks for sharing.