Speed up data preparation

[facebookresearch]

Speed up data preparation

[btgraham]

Calling input.precumputeMetadata(..) can help if called in the context of multiple data preparation threads, see e.g.
https://github.com/facebookresearch/SparseConvNet/blob/master/examples/Assamese_handwriting/data.py
If you are downsampling with size-2 stride-2 operations, use input.precomputeMetadata(2).
If you are downsampling with size-3 stride-2 operations, use input.precomputeMetadata(3).

precomputeMetadata uses Convolution_InputSgsToRulesAndOutputSgs as it is assumed multiple threads will be running anyway.
If you don't call precomputeMetadata, then the Convolution_InputSgsToRulesAndOutputSgs_OMP function is called as needed.

What is your network architectures? What is the input?

[zjhthu]

Thanks for your reply, I find a mistake in my previous experiments.
precumputeMetadata speeds most time in ValidConvolution_SgsToRules computation, I replace it with OMP version, and it works. Now precumputeMetadata takes about 4.3s. Besides, setLocations takes about 2s totally.
But data preparation (CPU) still takes longer time than GPU computation, 6.2s vs 3.0s, which causes GPU wait for data.
My data preparation code is simillar to your examples. My input is 256x192x256, my network is

        self.sparseModel = scn.Sequential().add(scn.ValidConvolution(3, 1, 16, 3, False))
        self.sparseModel.add(scn.MaxPooling(3, 2, 2))
        self.sparseModel.add(scn.SparseResNet(3, 16,[['b', 64, 1, 1]]))
        self.sparseModel.add(scn.MaxPooling(3, 2, 2))
        self.sparseModel.add(scn.ResNetUNet(3, 64, 2, 4))

[btgraham]

At training time, you should be able to use threads to run the single-threaded precomputeMetaData in in parallel to keep the GPU busy.