failed in Integrating 1 million cells

[jhu99]

failed in Integrating 1 million cells

[brianhie]

Hi @screamer, is the MemoryError happening in the correct() function? You can take a look at the whole stack trace to make sure.

You might also try decreasing the batch size. I believe 256GB should be sufficient.

You can also try sketching-based integration for additional efficiency: https://github.com/brianhie/scanorama/blob/master/scanorama/scanorama.py#L147

Will close since this doesn't appear to be a bug with Scanorama, but please feel free to ask additional questions!

[jhu99]

Hello @brianhie , thanks for your fast replying. I used BATCH_SIZE=2000. It returns the same problem. It seems that the error was happening in the integrate() function, because I didn't see the results of print('Integrated panoramas in {:.3f}s'.format(time() - t0)). Could I solved it by use a smaller BATCH_SIZE like 500? or better to use the sketch option? It costs about 18 hours to test the error everytime.

[brianhie]

If your batch size is that low and you use the integrate() function I think 256 GB should be sufficient. Maybe sketch it with a very low amount (e.g., a sketch size of 1000) and see if that works?