ariellubonja/zarr-performance-experiments

Are you wondering whether to use Zarr to store your scientific data? Here's a list of results to help you decide

Experiments on the Read Speed of the Zarr package under different circumstances

These include:

• How much Compression affects read speed
• Effects of using Sharding on performance
• Effects of increasing chunk_size
• Whether joining data variables together improves read speed

These are done both in repeated (warm) or first-time (cold) access

Accompanying Articles

• To Compress or Not to Compress — A Zarr Question
• Efficiently Querying Large Scientific Data Using Zarr’s partial decompress

Please open an issue if you'd like to see other experiments!

Interesting Results

1. Effects of Increasing chunk_size on Cold Access Speed

2. Same but Warm (Repeated) Access

3. Cold Access Speed w/ 60% Compression Ratio

4. Same but Warm Access

More results can be found in the experiment_results folder