Correct resampling of multiple swaths that do not fit into memory

[pytroll]

Correct resampling of multiple swaths that do not fit into memory

[mraspaud]

@cpaulik Sorry for the late answer.
I would say it would be a nice addition to pyresample! Do you have some examples ?

[floriankrb]

I have the same use case of resampling swaths with high memory consumption.

I saw some dependencies on dask in pyresample. Would using dask/xarray be a solution to avoid loading everything in memory ?

@cpaulik I would be very interested in knowing more about your working solution.

[mraspaud]

Yes, we're working with dask and xarray atm, and that helps with memory consumption. Feel free to join us on slack to discuss it, as it's still quite experimental (https://pytrollslackin.herokuapp.com/)

[mraspaud]

@cpaulik I'm assigning you to this issue, feel free to contact us if you need help with a PR

[cpaulik]

Thanks. Just don't expect a solution too soon. Since I first raised this issue two things have changed.

1. I switched jobs and no longer have access to the code I talked about in the original post. This is a minor problem since we would have to rewrite most of it anyway for a successful integration in pyresample.
2. I realized that we would have to extend pyresample in a way that the user can express how the points in the input datasets are related and how much overlap e.g. in time is necessary to perform a seamless resampling between parts of a swath.

Do you have any thoughts on 2.?

[mraspaud]

No rush, we can wait :)

About 2. I'm wondering how that data structure would be implemented. An adjacency graph/matrix of sorts maybe ?