Code chunk meta/coords

[grantbuster]

Code chunk meta/coords

[grantbuster]

yes but I think actually you just need to add chunks['meta'] = meta_chunks based on the docstring for "chunks" in this method:

nsrdb/nsrdb/file_handlers/outputs.py

Lines 38 to 39 in b2f01d3

	chunks : dict
	Dictionary of chunk tuples corresponding to each dataset.

I didn't remember that we sub-class the reV outputs handler for the nsrdb applications. You may not need to modify any reV code, just make any changes in the class linked above.

[MRossol]

code from WTK:

from rex.rechunk_h5.chunk_size import ArrayChunkSize
chunks = ArrayChunkSize.compute(self.meta)

[grantbuster]

So that method that michael reference will get the correct chunk shapes for any array. I think we want this to happen wherever the coordinate dataset gets written in the reV outputs hander and then here for the nsrdb meta data:

nsrdb/nsrdb/nsrdb.py

Line 337 in b2f01d3

def _init_output_h5(self, f_out, dsets, time_index, meta, force=False):

[bnb32]

so compute chunks for coords/meta and then write them out in chunked form like meta = meta.create_dataset("chunked", (1000, 1000), chunks=chunks)?

[bnb32]

added chunks for meta here

nsrdb/nsrdb/file_handlers/outputs.py

Lines 56 to 57 in 0cc0a99

	meta_chunks = ArrayChunkSize.compute(meta.to_numpy())
	chunks['meta'] = meta_chunks

and for coordinates here

nsrdb/nsrdb/file_handlers/outputs.py

Lines 59 to 62 in 0cc0a99

	if add_coords:
	coords = meta[['latitude', 'longitude']].to_numpy()
	coords_chunks = ArrayChunkSize.compute(coords)
	chunks['coords'] = coords_chunks