new I/O bottleneck as a result of reformatting

[gaelforget]

new I/O bottleneck as a result of reformatting

[owang01]

@gaelforget I submitted a pull request to the gcmfaces git for an updated read_nctiles.m that should improve the performance when reading V4r4 files. It now takes about the same time to read V4r3 or V4r4 files. See the pull at MITgcm/gcmfaces#12.

[timothyas]

Yes this is something I've experienced with dask (a tool underlying xarray). It seems like dask generally performs better with larger but fewer files. It doesn't totally matter how big the files are (probably to a certain threshold, that I haven't yet experienced) because dask doesn't read things into memory when a "read" call is made, it just figures out how to point to the files, which can take a long time with tons of files.

That said, perhaps it is the most flexible to continue making the product available in more files rather than fewer so that users can 1) only download the files they need and 2) not have to worry about accidentally loading a huge file into memory. What does everyone else think?

In that case, then we could recommend that eccov4py users 'reformat' the data into a format that benefits their workflow the best. For instance, users could load and re-save all 2D variables in a single file, or one file per year, or however they want it. They could additionally save it to zarr, or whatever file format they prefer and are familiar with.

What do you think? I'm not in any decision making position and I'm not in charge of the file formats. I'm just providing (hopefully useful?!) suggestions :) I would be happy to provide some suggested lines as above for a README that @gaelforget is recommending, just let me know if you'd like that.

[hongandyan]

That said, perhaps it is the most flexible to continue making the product available in more files rather than fewer so that users can 1) only download the files they need and 2) not have to worry about accidentally loading a huge file into memory. What does everyone else think?

I don't think scenario(2) will happen because "read_nctiles" is flexible in
reading time records or vertical levels.
Actually "read_nctiles" is OK for r4, the major problem is "grid_load" which
can't handle the incompatibility between r3/nctiles_grid/GRID*.nc and r4/ nctiles_grid/ECCO-GRID*.nc as pointed here
#40 (comment)

[gaelforget]

@gaelforget I submitted a pull request to the gcmfaces git for an updated read_nctiles.m that should improve the performance when reading V4r4 files. It now takes about the same time to read V4r3 or V4r4 files. See the pull at MITgcm/gcmfaces#12.

Will take a look as soon as possible & report back after I've had a chance to test on standard analysis for r2, r3, and r4. Hopefully by next week (but ...)

Thanks!!!