Hierarchical chunk storage

[ome]

Hierarchical chunk storage

[joshmoore]

Status update:

• bioformats2raw is moving towards 0.3.0. There will already be some breaking layout changes.
• zarr-python has the FSStore fixe merged to the mainline. The 2.7.0 release will contain it.
• The jzarr work will be taken over by @SabineEmbacher but is moving forward.

With all of this in mind, I'd propose that we go ahead and bump to OME-Zarr version to 0.2.0 stating in the spec that nested storage is a MUST. A client can then detect whether the fileset is 0.1.0 or 0.2.0 from the metadata and make a choice as to the chunk index separator.

cc: @manzt (since I've failed to engage on the JS front)

[joshmoore]

Here's what I've been working on:

• n5 supports nested (of course) but zarr+s3 is missing: saalfeldlab/n5-zarr#5
• jzarr doesn't support nested: bcdev/jzarr#17
• NestedDirectoryStore supports zarr+nesting but not s3. The newer FSStore which does support s3 had a bug in the nesting: zarr-developers/zarr-python#699

Conversations starting https://gitter.im/zarr-developers/community?at=601d471fc83ec358be27944f and during the 2021-02-10 community call suggest that even before the V3 spec (which defaults to nested storage) it might be possible for us to transition to using it more heavily with the V2 spec. A requirement would be a heuristic or metadata entry to determine if chunks are nested or flat. zarr-python is waiting on a PR from me.

[joshmoore]

Feels like most implementations have now implemented this new strategy. Likely a solid next step would be to add v0.1 and v0.2 implementations to https://github.com/ome/ome_zarr_test_suite

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

[joshmoore]

see: zarr-developers/zarr-python#715

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/open-an-image-from-omero-in-imagej-js/47747/17