About activation layer and inference:

[zeiss-microscopy]

About activation layer and inference:

[zeiss-mamthor]

Hi,

• It is correct that there is no activation after both 1x1 convolution layers. This applies to our modules as well as for almost all replacements (e.g., ResNets, WRNs, VGGs, etc.). For our MobileNet experiments, however, we retain the original structure of the inverted residual bottleneck as it was extensively analyzed in the MobileNet paper that such an arrangement of (i) placing residuals in the bottleneck path and (ii) using an intermediate activation for the distributing 1x1 conv is beneficial. But there is no activation in the bottleneck, which would affect the subspace transformation.
• When merging both 1x1 conv layers for inference, you will have the same output as for BSConvS, but at the FLOP count of BSConvU. This is because you actually reverse the matrix factorization of Eqn. 8 in the paper.
• We noticed speed-ups, but we did not perform a quantitative evaluation. Also note that it is heavily influenced by the actual implementation and used hardware, as it all depends on the support of an optimized implementation for depthwise convolutions.