`open_datatree` performance

[aladinor]

`open_datatree` performance

[TomNicholas]

Hi guys! Thanks for your interest!

@kmuehlbauer is right, we're merging datatree in upstream, and I believe that what you're asking for is essentially covered by the "Generalize backends to support groups." bullet point already listed on pydata/xarray#8572.

Whilst the others tagged above are working through those tasks in #8572, no-one has been assigned to that item yet, so if anyone here would like to help that would be amazing. You're welcome to come to the datatree meeting next week!

We looked into the open_datatree method and found the function loops through each node, opening/loading the Dataset at each node.

Yep, this implementation was only ever intended to be a proof-of-principle, it's not optimized at all.

[kmuehlbauer]

Thanks @aladinor for initiating this discussion.

As datatree is currently undergoing merge into xarray, spearheaded by
@TomNicholas, @flamingbear, @owenlittlejohns and @eni-awowale it would be great to hear thoughts from their side on that particular topic.

If necessary we can arrange participation in the weekly datatree meeting to discuss.

Anyway, any guidance and suggestions how to move forward with our presented data layout is greatly appreciated.

Thanks a bunch!