Optimize merging logic to handle massive number of records (300B+)

[googlegenomics]

Optimize merging logic to handle massive number of records (300B+)

[arostamianfar]

Thanks for investigating the Cloud Dataflow Shuffle, Nima!
Update: using Cloud Dataflow Shuffle actually solves the issue of handling massive number of records when merging. However, it can still be expensive since it's attempting to merge data from all chromosomes. We can optimize this further by partitioning the data by chromosome first. Saman is going to work on partitioning data for sharded BQ export, and this also fits nicely there (we can "kill two birds with one stone" :p ). Reassigning to Saman.

[arostamianfar]

Saman, now that Cloud Dataflow Shuffle pricing is updated, could you please try with --experiments=shuffle_mode=service with the platinum1000 test to see what cost/performance you get? I think we can close this issue once that experiment is done.

[arostamianfar]

Closing this bug. Summary:

• Using the shuffle service is actually more expensive than our native partition-based merge (turned on with the --optimize_for_large_inputs flag).
• Using a large number of small workers is much more efficient than using a small number of large workers.
• Using SSDs can further reduce cost if a large enough worker pool is not available.

More details are provided in the handling large inputs doc.