How to run with large datasets in limited memory?

[ZeroLi-Bio]

How to run with large datasets in limited memory?

[zijianni]

Hi @ZeroLi-Bio , a dataset of 80k tissue spots is not a common input for SpotClean, as it was primarily designed for Visium data, which contains at most 5k spots.

For non-Visium data, rigorously speaking, the nature of spot swapping has not been scientifically verified (though we believe it is a common artifact across many protocols, and this is not hard to check - just check what's in the background spots, which shouldn't contain any expression), and there is no guarantee that the model assumption of SpotClean on spot swapping still holds true. Please be careful if you decide to apply SpotClean to non-Visium data.

For your idea of randomly partitioning spots into subsets (if I understand it correctly), I think that's a smart direction to explore. However, our model assumes that each spot contains spot-swapped expressions from all nearby spots. If only a subset of nearby spots are provided, the effect from the rest nearby spots are not modelled, and it could result in incorrect estimations.

[ZeroLi-Bio]

Thank you a lot!