Explore mode for recipes

[erikr]

Explore mode for recipes

[erikr]

@paolodi You made this work with interpretations and tested it on UKBB tensors. What else must be done before the PR?

Also we should merge my Partners branch with master soon so I can revise my tmaps and explore to use interpretations. Can we discuss next week with the team?

[erikr]

I need to:

• Resolve difference between intersect and union counts for test cohort.
• Add missing_fraction to summary_stats_string
• Add missing_fraction to summary_stats_continuous
• Add variance to summary_stats_continuous
• Parallelize

[erikr]

@paolodi note the latest version of explore is in recipes.py in my branch er_tensorize_partners_ecgs rather than er_explore_tensormaps. I want to delete the latter branch as I have not used it for anything. What do you think?

[paolodi]

I'd leave it there, as that's the branch the PR refers to. Deleting it would close the PR again. I'll take care of merging it with er_tensorize_partners_ecgs and once it works again with master we can ask for an additional pair of eyes

[erikr]

Thanks!

[erikr]

@paolodi I finished more modifications (378a4ab), so please checkout recipes.py from the latest er_tensorize_partners_ecgs branch.

• Resolve difference between intersect and union counts for test cohort.

Differences are expected behavior because ∩ across tmaps will be lower than union. Duh.

• Add missing_fraction to summary_stats_string
• Add missing_fraction to summary_stats_continuous
• Add variance to summary_stats_continuous
• Parallelize

iterating through many hd5s is i/o bound; parallelizing slowed down overall run time

I also now calculate summary stats using Pandas functions whenever possible, since they are fast, and work on pd.Series of np.arrays. I had trouble using Numpy functions with our nested data structure.

The most important change is that tensor_to_df is not called for each interpretation.

tensor_to_df iterates through every tensor and consolidates them into a big dataframe.

Now, for each interpretation, we only iterate through the keys (of the df) that belong to that interpretation.

Calling it once instead of three times reduces the run time to 33% of before!

Other changes for future PRs are:

• Improving performance with JIT compilation via Numba
• Modularizing most of explore into functions, and housing those functions inside of explore.py
• Plotting histograms for some of the summary stats (especially continuous)