Harmonize tensorization parallelization

[erikr]

Harmonize tensorization parallelization

[StevenSong]

yes, let me address in next PR, I'm doing some things with the tensor writer code anyways and this is easy to address

[lucidtronix]

Our current tensorization process involves 4 main steps:

Run Dataflow jobs to tensorize data from big query database. This means 6 separate data flow runs for each of the fields: ['categorical', 'continuous', 'icd', 'disease', 'death', 'phecode_disease']

Tensorize bulk data with tensor_writer_ukbb.py. This file has code to tensorize abdominal, cardiac, and brain MRI as well as resting and exercise ECG.

Merge tensors with merge_hd5s.py. Doing intersections or in-place merging as appropriate.

Append any CSV or TSV data with the `append[categorical,continuous]_[csv,tsv]' recipe modes.

Each step is currently run separately. We would also like to be able to run them all at once and store intermediate tensors in google cloud buckets.

[StevenSong]

Partners data is tensorized with tensor_writer_partners.py. Currently only tensorizes Partners ECGs. Parallelization is achieved using joblib, should probably convert to strategy of tensor_writer_ukbb.py or rest of codebase

[erikr]

Partners tensorize should switch from joblib to multiprocess to be consistent with the rest of the codebase.

@ndiamant any thoughts?

[erikr]

@StevieSong is this still an issue?