Conditional analysis run time

[broadinstitute]

Conditional analysis run time

[francois-a]

Hi,

While the conditional analysis will be a bit slower than the standard permutations mode, it should be much faster than that. The runtime will depend on the number of significant phenotypes resulting from the permutations. For example, if the permutations take ~30min and yield ~10k significant phenotypes, the conditional analysis should not take more than 1-2 hours (for typical QTL mapping).

I suspect this is a data complexity issue -- how many independent variants did you obtain for the genes you tested so far?

[JonMarten]

I don't have an easy answer unfortunately, the job times out without saving any output. I'm trying to run a toy example with just 3 phenotypes to get an answer but the GPU cluster is unfortunately being overused this week. I'll update when I can.

For perspective, if it helps, I'm running with 4,136 samples and 67 covariates, and on Chr2 for an example there are 1,179 phenotypes and 807,409 variants. For the map_cis results the number of variants per feature ranges averages around 6,000

[francois-a]

This should not take more than a few seconds per phenotype at most. Is the GPU running, and how much memory is used/free?

Are you able to share scrambled/residualized data? There should not be any data protection issues if you do that, and we can discuss further by email if needed.

[JonMarten]

I think I might have to get permission to share the actual data because in trying to find a set of two or three phenotypes to replicate the error I've found another weird problem.

When I have 10 phenotypes, one of them hangs with the error (WARNING: scipy.optimize.newton failed to converge (running scipy.optimize.minimize) which appears multiple times until it times out.

When I cut that down to just 2 phenotypes, the one that hangs and another that doesn't, it works fine. The phenotype that hung before had two independent SNPs, the other had just one.

But when I add another phenotype to that set, it does still run, but now the phenotype that hung has THREE independent SNPs in the results, and the other phenotype from before has two. The new one has just one.

Something very odd is happening. I'll get permission to share data and work up a script to replicate the error.

[JonMarten]

Hi Francois, unfortunately I'm out of time - I'm changing jobs and won't be working on this analysis going forward. Hopefully this will be picked up by someone else on the project, but in the meantime I'll close this.