k_selection_plot Error: Last 2 dimensions of the array must be square

[dylkot]

k_selection_plot Error: Last 2 dimensions of the array must be square

[dpcook]

Update: Just confirmed that I hit the same error running through the sample PBMC notebook without any modifications to the code, so at least it's not something with the input data.

HTML of notebook can be found here

I'm running python in a Conda environment set up as described in the README.

# packages in environment at /Users/dpcook/miniconda3/envs/cnmf_env:
#
# Name                    Version                   Build  Channel
anndata                   0.6.22.post1               py_0    bioconda
appnope                   0.1.0            py36hf537a9a_0  
attrs                     19.3.0                     py_0  
backcall                  0.1.0                    py36_0  
backports                 1.0                        py_2    conda-forge
bleach                    3.1.0                    py36_0  
blosc                     1.17.0               h6de7cb9_1    conda-forge
bzip2                     1.0.8                h01d97ff_1    conda-forge
ca-certificates           2019.10.16                    0  
cairo                     1.16.0            he1c11cd_1002    conda-forge
certifi                   2019.9.11                py36_0  
configparser              3.7.3                    py36_1    conda-forge
contextlib2               0.6.0.post1                py_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cython                    0.29.13          py36h0a44026_0  
decorator                 4.4.1                      py_0    conda-forge
defusedxml                0.6.0                      py_0  
entrypoints               0.3                      py36_0  
fastcluster               1.1.25          py36h1702cab_1000    conda-forge
fontconfig                2.13.1            h6b1039f_1001    conda-forge
freetype                  2.10.0               h24853df_1    conda-forge
gettext                   0.19.8.1          h46ab8bc_1002    conda-forge
glib                      2.58.3            h9d45998_1002    conda-forge
gmp                       6.1.2             h0a44026_1000    conda-forge
h5py                      2.9.0           nompi_py36h3d62f72_1103    conda-forge
hdf5                      1.10.4          nompi_h0cbb7df_1106    conda-forge
icu                       64.2                 h6de7cb9_1    conda-forge
igraph                    0.7.1             h02d7eb0_1007    conda-forge
importlib-metadata        0.19                     py36_0    conda-forge
importlib_metadata        0.23                     py36_0  
ipykernel                 5.1.3            py36h39e3cac_0  
ipython                   7.8.0            py36h39e3cac_0  
ipython_genutils          0.2.0            py36h241746c_0  
jedi                      0.15.1                   py36_0  
jinja2                    2.10.3                     py_0  
joblib                    0.14.0                     py_0    conda-forge
json5                     0.8.5                      py_0  
jsonschema                3.1.1                    py36_0  
jupyter_client            5.3.4                    py36_0  
jupyter_core              4.6.1                    py36_0  
jupyterlab                1.1.4              pyhf63ae98_0  
jupyterlab_server         1.0.6                      py_0  
kiwisolver                1.1.0            py36h770b8ee_0    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libcxx                    4.0.1                hcfea43d_1  
libcxxabi                 4.0.1                hcfea43d_1  
libedit                   3.1.20181209         hb402a30_0  
libffi                    3.2.1                h475c297_4  
libgfortran               3.0.1                         0    conda-forge
libiconv                  1.15              h01d97ff_1005    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libopenblas               0.3.7                hd44dcd8_1    conda-forge
libpng                    1.6.37               h2573ce8_0    conda-forge
libsodium                 1.0.16               h3efe00b_0  
libxml2                   2.9.10               h53d96d6_0    conda-forge
llvm-openmp               9.0.0                h40edb58_0    conda-forge
llvmlite                  0.30.0           py36h98b8051_0  
louvain                   0.6.1            py36h0a44026_2    conda-forge
lzo                       2.10              h1de35cc_1000    conda-forge
markupsafe                1.1.1            py36h1de35cc_0  
matplotlib                3.1.1            py36h54f8f79_0  
mistune                   0.8.4            py36h1de35cc_0  
mock                      3.0.5                    py36_0    conda-forge
more-itertools            7.2.0                      py_0    conda-forge
natsort                   6.0.0                      py_0    conda-forge
nbconvert                 5.6.1                    py36_0  
nbformat                  4.4.0            py36h827af21_0  
ncurses                   6.1                  h0a44026_1  
networkx                  2.4                        py_0    conda-forge
notebook                  6.0.2                    py36_0  
numba                     0.46.0           py36h6440ff4_0  
numexpr                   2.7.0            py36h86efe34_0    conda-forge
numpy                     1.17.3           py36hde6bac1_0    conda-forge
openssl                   1.1.1d               h1de35cc_3  
palettable                3.3.0                      py_0    conda-forge
pandas                    0.25.2           py36h0a44026_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py36_1  
parallel                  20170422             pl5.22.0_0    bioconda
parso                     0.5.1                      py_0  
pathlib2                  2.3.5                    py36_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.43                 h0a44026_0  
perl                      5.22.0.1                      0    conda-forge
pexpect                   4.7.0                    py36_0  
pickleshare               0.7.5                    py36_0  
pip                       19.3.1                   py36_0  
pixman                    0.38.0            h01d97ff_1003    conda-forge
prometheus_client         0.7.1                      py_0  
prompt_toolkit            2.0.10                     py_0  
ptyprocess                0.6.0                    py36_0  
pycairo                   1.18.2           py36h83b1bec_0    conda-forge
pygments                  2.4.2                      py_0  
pyparsing                 2.4.4                      py_0    conda-forge
pyrsistent                0.15.4           py36h1de35cc_0  
pytables                  3.6.1            py36h5bccee9_0  
python                    3.6.9                h359304d_0  
python-dateutil           2.8.1                      py_0    conda-forge
python-igraph             0.7.1.post7      py36h0b31af3_0    conda-forge
pytz                      2019.3                     py_0    conda-forge
pyyaml                    5.1.2            py36h0b31af3_0    conda-forge
pyzmq                     18.1.0           py36h0a44026_0  
readline                  7.0                  h1de35cc_5  
scanpy                    1.4.4.post1                py_0    bioconda
scikit-learn              0.21.3           py36hd4ffd6c_0    conda-forge
scipy                     1.3.1            py36hab3da7d_2    conda-forge
seaborn                   0.9.0                      py_0    conda-forge
send2trash                1.5.0                    py36_0  
setuptools                41.6.0                   py36_0  
six                       1.13.0                   py36_0    conda-forge
sqlite                    3.30.1               ha441bb4_0  
statsmodels               0.10.1           py36h3b54f70_1    conda-forge
tbb                       2018.0.5             h2d50403_0    conda-forge
terminado                 0.8.2                    py36_0  
testpath                  0.4.2                    py36_0  
tk                        8.6.8                ha441bb4_0  
tornado                   6.0.3            py36h0b31af3_0    conda-forge
tqdm                      4.7.2                    py36_0    bioconda
traitlets                 4.3.3                    py36_0  
umap-learn                0.3.10                   py36_0    conda-forge
wcwidth                   0.1.7            py36h8c6ec74_0  
webencodings              0.5.1                    py36_1  
wheel                     0.33.6                   py36_0  
xz                        5.2.4                h1de35cc_4  
yaml                      0.1.7             h1de35cc_1001    conda-forge
zeromq                    4.3.1                h0a44026_3  
zipp                      0.6.0                      py_0    conda-forge
zlib                      1.2.11               h1de35cc_3  

[dylkot]

Thanks for posting this! I'm looking into it now and will add an update asap.

[dylkot]

Interesting, I created a conda environment following the instructions and ran the PBMC example from start to finish without any errors.

I also ran the pipeline on your data (with similar parameters to what you tried) and didn't hit any errors there either (K selection plot below)

I'm guessing this must be some kind of software installation or version issue. But its odd because the versions in your conda environment seem to match what I found.

The error is happening on the step where the Frobenius norm is being computed

LINE 562: prediction_error = ((norm_counts.X.todense() - rf_pred_norm_counts)**2).sum().sum()

Maybe try replacing it with something like this?

LINE 562: prediction_error = ((np.array(norm_counts.X.todense()) - rf_pred_norm_counts)**2).sum().sum()

I'm a bit stuck on how to figure out where the discrepancies in the behavior are coming from.

[dpcook]

Thanks for looking into this--I really appreciate it!

Not sure why it's behaving differently. I rebuilt the environment and still had the error, but your fix of explicitly calling np.array worked like a charm!

I'll push onward and let you know if I hit any other hurdles--thanks again, Dylan

[Zifeng-L]

有趣的是，我按照说明创建了一个conda环境，并从头到尾运行了PBMC示例，没有任何错误。

我还在您的数据上运行了管道（参数与您尝试的参数相似），并且也没有遇到任何错误（下面的K选择图）

我猜想这一定是某种软件安装或版本问题。但这很奇怪，因为您的conda环境中的版本似乎与我发现的版本匹配。

错误发生在正在计算Frobenius范数的步骤上

LINE 562: prediction_error = ((norm_counts.X.todense() - rf_pred_norm_counts)**2).sum().sum()

也许尝试用这样的东西代替它？

LINE 562: prediction_error = ((np.array(norm_counts.X.todense()) - rf_pred_norm_counts)**2).sum().sum()

我对如何弄清楚行为差异来自何处有些困惑。

I hit the same error！ Can you tell me how to replace the code? from a python novice.

[Zifeng-L]

update: I rebuilt the environment. The error changed as follows

K selection plot command: python /cluster/home/zfli/cNMF/cnmf.py k_selection_plot --output-dir /cluster/home/zfli/RProject/LYS/cnmf --name S132TB_cNMF
Traceback (most recent call last):
  File "/cluster/home/zfli/cNMF/cnmf.py", line 883, in <module>
    cnmf_obj.k_selection_plot()
  File "/cluster/home/zfli/cNMF/cnmf.py", line 696, in k_selection_plot
    stats.append(self.consensus(k, skip_density_and_return_after_stats=True).stats)
  File "/cluster/home/zfli/cNMF/cnmf.py", line 500, in consensus
    norm_counts = sc.read(self.paths['normalized_counts'])
  File "/cluster/home/zfli/anaconda2/envs/cnmf_env/lib/python3.6/site-packages/scanpy/readwrite.py", line 97, in read
    backup_url=backup_url, cache=cache, **kwargs,
  File "/cluster/home/zfli/anaconda2/envs/cnmf_env/lib/python3.6/site-packages/scanpy/readwrite.py", line 499, in _read
    return read_h5ad(filename, backed=backed)
  File "/cluster/home/zfli/anaconda2/envs/cnmf_env/lib/python3.6/site-packages/anndata/readwrite/read.py", line 447, in read_h5ad
    constructor_args = _read_args_from_h5ad(filename=filename, chunk_size=chunk_size)
  File "/cluster/home/zfli/anaconda2/envs/cnmf_env/lib/python3.6/site-packages/anndata/readwrite/read.py", line 502, in _read_args_from_h5ad
    return AnnData._args_from_dict(d)
  File "/cluster/home/zfli/anaconda2/envs/cnmf_env/lib/python3.6/site-packages/anndata/core/anndata.py", line 2157, in _args_from_dict
    if key in d_true_keys[true_key].dtype.names:
AttributeError: 'dict' object has no attribute 'dtype'