Speed/Memory Bottlenecks

[eden-kramer-lab]

Speed/Memory Bottlenecks

[Who8MyLunch]

Hi @edeno you might consider using FFTW (https://github.com/pyFFTW/pyFFTW) if you're looking for faster performance. It makes a huge improvement. It's a drop-in replacement for Numpy and Scipy FFT implementations.

[edeno]

Thanks @Who8MyLunch. I will definitely check that out.

You don't happen to have any suggestions for something that has to take the dot product many times do you? I'm considering moving that portion to using cython, but I don't have much experience with that.

[Who8MyLunch]

@edeno there's a number of options for accelerating your dot products, the simplest being Numpy and the next being Cython. Cython is good, but it's not trivial. Numpy is trivial to use. There are many examples around the internet of using both to speed up fundamental work tasks like dot products. Good luck!

[christian-oreilly]

I had to delete any use of lru_cache in my local copy to avoid the memory exploding to unreasonable values (above 64 GB). Even deletion of the objects and explicit garbage collection would not do the trick. So, bottom line, depending on where your use case falls with respect to the speed/memory usage tradeoff, caching might not always be a good idea. A possible improvement would be to make the caching optional and control it by an argument to the function using it (set by default to True for backward compatibility).

[edeno]

@christian-oreilly That makes sense. I'll think about a better way to handle this. Thank you for that feedback.