DKIST dataset load memory usage is doubled

[arparaschiv]

DKIST dataset load memory usage is doubled

[Cadair]

hello @arparaschiv thanks a lot for the detailed report. I shall have a look into it, although it will probably be next week before I find the time.

Out of curiosity, and maybe as a short term fix, what are you doing that needs the whole array in memory? Could you make use of any of the dask trickery to work only on chunks of the data at a time?

[arparaschiv]

Hi @Cadair. There is no urgency for a short term fix.

I am in the process of preparing some workshop content for new Cryo-NIRSP users and was attempting to explain how to work with asdf files and how to use the compute method to get chunks that you need. I stumbled on this by chance, to be honest. We already have warnings in our materials that people should think about how to load data in a smart way, rather than reading entire datasets in memory. For smaller chunks you would not notice/care, but our datasets are quite large, and users being users, I would expect people getting either kernel resets/or system hangs in no time when they start using these toolsets.