There seems a bug for 1x1 convolution

[facebookresearch]

There seems a bug for 1x1 convolution

[btgraham]

Hmm. I have never actually tested sparseconvnet.Convolution with 1x1 filters.

For 1x1 convolutions, the sparsity pattern does not change, so it is better to
use 'sparseconvnet.SubmanifoldConvolution' with size 1x1
or (even better) 'sparseconvnet.NetworkInNetwork' (equivalent to the above but optimized for size 1x1).

See for example

SparseConvNet/sparseconvnet/networkArchitectures.py

Line 144 in 6698676

return scn.NetworkInNetwork(nIn, nOut, False)

[LaiSongxuan]

Thank you. : )

[LaiSongxuan]

Hello, @btgraham. I now have a ten-layer standard convolutional network, and the input is indeed sparse (only about 5% of the input is active; the input size is Nx7x200x400). However I found the pytorch implementation is several times faster than the sparseconvnet implementation. The latter has iterative SubmanifoldConvolution and BatchNormReLU layers. The GPU utilization is about 95% in both cases.

Is the performance gap normal (since pytorch uses high optimized cudnn libraries) or I have done something wrong?

[btgraham]

It will depend on the network, and how the sparsity changes through the network (due to convolutions, pooling, etc). I can take a look at the network, and a typical input, if you show me.

SparseConvNet is not as heavily optimised as PyTorch/CUDNN kernels, so for training relatively small networks running on the GPU, dense convolutions may be faster. Sparsity is more important in 3D, or when running on the CPU/low power hardware.

[LaiSongxuan]

The network is not deep, and does not change the sparsity pattern a lot. I have profiled the program, and it seems the convolution backward kernels are much slower than the pytorch kernels.

Can you please give me you email? I'll send you my code. I have cleaned and simplified the code, and it should be easy to run by "python main.py".

[btgraham]

[email protected]

[bigsheep2012]

@LaiSongxuan hello, may I know some details about your pytorch implementation that several times faster than sparseconvnet implementation?
You just use some combination of torch.nn.conv2d or torch.nn.conv3d layers in your pytorch implementation?

[LaiSongxuan]

@bigsheep2012 Right. I use iterative layers of nn.conv2d, nn.Batchnorm2d and nn.RELU.