Cis analysis fails to find R package qvalue

[JonMarten]

Cis analysis fails to find R package qvalue

[JonMarten]

Ah. Well, small update - I realised I wasn't explicitly specifying that the same version of R should be called in the job scheduler. I fixed that and it appears to have resolved the error - presumably meaing it was looking at a blank R library for a different version.

Unfortunately, I have a new even more opaque error:

    processing phenotype 17666/17666

Traceback (most recent call last):
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4730, in get_value
    return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
  File "pandas/_libs/index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 88, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/site-packages/tensorqtl/__main__.py", line 2, in <module>
    tensorqtl.main()
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/site-packages/tensorqtl/tensorqtl.py", line 111, in main
    calculate_qvalues(res_df, fdr=args.fdr, qvalue_lambda=args.qvalue_lambda, logger=logger)
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/site-packages/tensorqtl/post.py", line 49, in calculate_qvalues
    ub = res_df.loc[res_df['qval']>fdr, 'pval_beta'].sort_values()[0]
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/site-packages/pandas/core/series.py", line 1068, in __getitem__
    result = self.index.get_value(self, key)
  File "/home/jm2294/.conda/envs/tensorQTL/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4736, in get_value
    return libindex.get_value_box(s, key)
  File "pandas/_libs/index.pyx", line 51, in pandas._libs.index.get_value_box
  File "pandas/_libs/index.pyx", line 47, in pandas._libs.index.get_value_at
  File "pandas/_libs/util.pxd", line 98, in pandas._libs.util.get_value_at
  File "pandas/_libs/util.pxd", line 89, in pandas._libs.util.validate_indexer
IndexError: index out of bounds
  Time elapsed: 45.48 min
done.
  * writing output
Computing q-values
  * Number of phenotypes tested: 17666
  * Correlation between Beta-approximated and empirical p-values: : 1.0000
  * Proportion of significant phenotypes (1-pi0): 0.95
  * QTL phenotypes @ FDR 0.05: 17666

[francois-a]

Hi,

This is indeed opaque.

Have you tried running this interactively, by importing tensorqtl as module? If so, can you run map_cis and share the resulting dataframe?

[JonMarten]

                 num_var  beta_shape1  beta_shape2      true_df   pval_true_df   variant_id  tss_distance  ...       maf  ref_factor   pval_nominal     slope  slope_se  pval_perm      pval_beta
phenotype_id                                                                                               ...
ENSG00000223972     1322     1.033777   380.794495  2433.491699   7.576844e-04  rs144137219        943405  ...  0.006062           1   3.870159e-04 -0.538294  0.151493   0.237276   2.363449e-01
ENSG00000227232     1393     1.014911   395.193115  2433.273926   4.265920e-03  rs112618790        812282  ...  0.099303           1   2.599590e-03  0.082266  0.027292   0.809219   8.112105e-01
ENSG00000233750     1915     1.047369   456.706665  2406.245117   4.632885e-03  rs568927205        -80124  ...  0.415220           1   2.695241e-03 -0.063475  0.021135   0.870813   8.704000e-01
ENSG00000268903     1919     1.034786   482.007446  2417.564209   3.384300e-03  rs142329267        833883  ...  0.049339           1   1.947521e-03 -0.072601  0.023411   0.797320   7.946825e-01
ENSG00000269981     1924     1.050884   469.730133  2402.008789   2.705939e-03    rs9723307        872790  ...  0.028390           1   1.467018e-03 -0.095396  0.029957   0.706929   7.010368e-01
...                  ...          ...          ...          ...            ...          ...           ...  ...       ...         ...            ...       ...       ...        ...            ...
ENSG00000205559     6311     1.020132   967.323853  2385.615723  9.814792e-211      rs86337         -5401  ...  0.424349           1  2.082032e-238  0.658249  0.017991   0.000100  6.396193e-212
ENSG00000008735     6214     1.045838   831.689941  2316.647705   8.270214e-11   rs76733141          7074  ...  0.066111           1   2.296552e-12 -0.340904  0.048370   0.000100   3.165674e-08
ENSG00000100299     6132     1.050933   812.577148  2323.872070   0.000000e+00    rs6151429         -3124  ...  0.078228           1   0.000000e+00 -0.982828  0.017313   0.000100   0.000000e+00
ENSG00000184319     5276     1.024658   788.057983  2383.427002   3.595846e-20    rs6010080        -40679  ...  0.069399           1   1.181032e-22 -0.433737  0.043884   0.000100   1.095780e-17
ENSG00000079974     5349     1.018324   786.077881  2384.526611   1.956143e-33    rs2285395        -44001  ...  0.060284           1   1.164918e-37  0.319797  0.024552   0.000100   4.336756e-31

[17666 rows x 16 columns]

Print output attached. This is run using the code from here and the same inputs as the command line version.

Running the final line tensorqtl.calculate_qvalues(cis_df, qvalue_lambda=0.85) seems to complete without error and yields the following:

>>> cis_df
                 num_var  beta_shape1  beta_shape2      true_df   pval_true_df   variant_id  ...     slope  slope_se  pval_perm      pval_beta           qval  pval_nominal_threshold
phenotype_id                                                                                 ...
ENSG00000223972     1322     1.033777   380.794495  2433.491699   7.576844e-04  rs144137219  ... -0.538294  0.151493   0.237276   2.363449e-01   1.378329e-02                0.006882
ENSG00000227232     1393     1.014911   395.193115  2433.273926   4.265920e-03  rs112618790  ...  0.082266  0.027292   0.809219   8.112105e-01   4.421187e-02                0.006547
ENSG00000233750     1915     1.047369   456.706665  2406.245117   4.632885e-03  rs568927205  ... -0.063475  0.021135   0.870813   8.704000e-01   4.728907e-02                0.005795
ENSG00000268903     1919     1.034786   482.007446  2417.564209   3.384300e-03  rs142329267  ... -0.072601  0.023411   0.797320   7.946825e-01   4.336314e-02                0.005445
ENSG00000269981     1924     1.050884   469.730133  2402.008789   2.705939e-03    rs9723307  ... -0.095396  0.029957   0.706929   7.010368e-01   3.853997e-02                0.005648
...                  ...          ...          ...          ...            ...          ...  ...       ...       ...        ...            ...            ...                     ...
ENSG00000205559     6311     1.020132   967.323853  2385.615723  9.814792e-211      rs86337  ...  0.658249  0.017991   0.000100  6.396193e-212  4.444391e-212                0.002690
ENSG00000008735     6214     1.045838   831.689941  2316.647705   8.270214e-11   rs76733141  ... -0.340904  0.048370   0.000100   3.165674e-08   2.488204e-09                0.003183
ENSG00000100299     6132     1.050933   812.577148  2323.872070   0.000000e+00    rs6151429  ... -0.982828  0.017313   0.000100   0.000000e+00   0.000000e+00                0.003269
ENSG00000184319     5276     1.024658   788.057983  2383.427002   3.595846e-20    rs6010080  ... -0.433737  0.043884   0.000100   1.095780e-17   1.101945e-18                0.003311
ENSG00000079974     5349     1.018324   786.077881  2384.526611   1.956143e-33    rs2285395  ...  0.319797  0.024552   0.000100   4.336756e-31   5.599857e-32                0.003305

[17666 rows x 18 columns]

[francois-a]

Thanks.

Does the following command work in your environment?

python3 -c "import numpy as np; import rpy2; from rpy2.robjects.packages import importr; qvalue = importr('qvalue'); print(np.array(qvalue.qvalue(rpy2.robjects.vectors.FloatVector([0.001,0.01,0.1,0.4,0.4,0.5]), **{'lambda':rpy2.robjects.vectors.FloatVector([0.5])}).rx2('qvalues')))"

Otherwise, are you able to share the full dataframe? Or a subset of it that reproduces the error?

[JonMarten]

The command works. I don't think I can share the dataframe but I'll see if I can find a way to recreate the error. In the meantime, I can just run in python rather than using the command line.

[francois-a]

I finally ran across this error. fd79ad4 should fix this.