aertslab / scenicprotocol Goto Github PK
View Code? Open in Web Editor NEWA scalable SCENIC workflow for single-cell gene regulatory network analysis
License: GNU General Public License v3.0
scenicprotocol's People
Stargazers
Watchers
Forkers
huuthoth b97jre 514419407 kant duosu yuqimiao zorrodong michaelzh24 geng-lee taoshengxu anu-bioinfo yangmean yang050036 eijynagai yu-1011 abu601 jiajia7179 oubounyt jun-lizst chenmengpin misaka-dayu rojinsafavi laurenpmarazzi dimitri-sid zh-bian nikmerlock97 takshan zic12345 xchromosome219 hs-zheng zala-m jimmayxu iceshadows weinformatics luxiancao abuchin ruiyuraywang sjohri20 bgudenas rebecaarroyo natnaelt mkyriak pudima farewellznm kayihui gauravsinghrathore gusgml63 zgyaru tilmanimmisch gmaciag chaetognatha dotsonga zhangshuli-stu smarted rutendos jywang-ucas erhei130 66huge mattcfbrown mahibose likaitanscenicprotocol's Issues
error: No module named remote_ikernel
when I tried to run:
python3 -m remote_ikernel manage --add
--kernel_cmd="source /home/zhang/miniconda3/etc/profile.d/conda.sh && conda activate scenic_protocol && /home/zhang/miniconda3/envs/scenic_protocol/bin/ipython3 kernel -f {connection_file}"
--name="pyscenic-0918-conda"
--interface=ssh
--host=hostname
--workdir="~/"
--language=python3
it says: No module named remote_ikernel.
How to solve this?
Thanks,
Fan
Can't run pyscenic grn
First off, thanks for publishing the package!
I'm having trouble running the example notebook though. I've got the error when running
!pyscenic grn {f_loom_path_scenic} {f_tfs} -o adj.csv --num_workers 20
I'm on python 3.6.11 and macOS 10.15.6. I've installed the protocol using Conda.
Is this something you have seen?
/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
2020-09-06 22:29:53,586 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2020-09-06 22:29:55,980 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
OMP: Info #270: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
preparing dask client
parsing input
/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/arboreto/algo.py:214: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
expression_matrix = expression_data.as_matrix()
creating dask graph
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.core - ERROR - tuple is not allowed for map key
Traceback (most recent call last):
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/core.py", line 347, in handle_comm
msg = yield comm.read()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/comm/tcp.py", line 218, in read
frames, deserialize=self.deserialize, deserializers=deserializers
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/comm/utils.py", line 85, in from_frames
res = _from_frames()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/comm/utils.py", line 71, in _from_frames
frames, deserialize=deserialize, deserializers=deserializers
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
not shutting down client, client was created externally
finished
Traceback (most recent call last):
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 421, in main
args.func(args)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 74, in find_adjacencies_command
network = method(expression_data=ex_mtx, tf_names=tf_names, verbose=True, client_or_address=client, seed=args.seed)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/arboreto/algo.py", line 41, in grnboost2
early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/arboreto/algo.py", line 128, in diy
seed=seed)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/arboreto/core.py", line 403, in create_graph
future_tf_matrix = client.scatter(tf_matrix, broadcast=True)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/client.py", line 2071, in scatter
hash=hash,
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/client.py", line 753, in sync
return sync(self.loop, func, *args, **kwargs)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/utils.py", line 331, in sync
six.reraise(*error[0])
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/utils.py", line 316, in f
result[0] = yield future
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/client.py", line 1916, in _scatter
timeout=timeout,
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/core.py", line 739, in send_recv_from_rpc
result = yield send_recv(comm=comm, op=key, **kwargs)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/core.py", line 533, in send_recv
response = yield comm.read(deserializers=deserializers)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/comm/tcp.py", line 218, in read
frames, deserialize=self.deserialize, deserializers=deserializers
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/comm/utils.py", line 85, in from_frames
res = _from_frames()
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/comm/utils.py", line 71, in _from_frames
frames, deserialize=deserialize, deserializers=deserializers
File "/Users/f6v/opt/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
Abount mm_tfs.txt
Hello, I have a naive question.
Where can I download the mm_tfs.txt?
Thank you any comments
Problem to install VSN_Pipelines
putting *.tsv file as input to run pySCENIC
Hi, the SCENICprotocol in docker is maintained, while scenic-nf wasn't maintained.
my question is whether the *.tsv could be input for SCENICprotocol by parameter --expr like scenic-nf.
regards
pyscenic ctx output information
Hello,
I would like to know what is the meaning of the numerical values that I see in the TargetGenes column near each gene name.
Thank you.
Error when calculating mitochondrial read fraction on a sparse matrix
If the loom file contains a sparse matrix, this line will cause a problem:
SCENICprotocol/bin/filtering-basic.py
Line 65 in 36db0a5
by writing the following into adata.obs['percent_mito']:
In [61]: adata.obs['percent_mito']
Out[61]:
rna_AAACCCAAGCGCCCAT-1 (0, 1)\t0.00045372050816696913\n (0, 2)\t0....
rna_AAACCCACAGAGTTGG-1 (0, 1)\t0.0001699524133242692\n (0, 2)\t0.0...
rna_AAACCCACAGGTATGG-1 (0, 1)\t0.00018083182640144665\n (0, 2)\t0....
rna_AAACCCACATAGTCAC-1 (0, 1)\t0.00019584802193497845\n (0, 2)\t0....
rna_AAACCCACATCCAATG-1 (0, 1)\t0.00021872265966754156\n (0, 2)\t0....
...
rna_TTTGTTGGTGCGTCGT-1 (0, 1)\t0.00019550342130987292\n (0, 2)\t0....
rna_TTTGTTGGTGTCATGT-1 (0, 1)\t0.000254000508001016\n (0, 2)\t0.00...
rna_TTTGTTGGTTTGAACC-1 (0, 1)\t0.00023557126030624264\n (0, 2)\t0....
rna_TTTGTTGTCCAAGCCG-1 (0, 1)\t0.00017857142857142857\n (0, 2)\t0....
rna_TTTGTTGTCTTACTGT-1 (0, 1)\t0.0001479946721918011\n (0, 2)\t0.0...
Name: percent_mito, Length: 9432, dtype: object
The error occurs when trying to subset adata based on this filter:
In [60]: adata = adata[adata.obs['percent_mito'] < 0.5, :]
/opt/venv/lib/python3.7/site-packages/scipy/sparse/compressed.py:287: SparseEfficiencyWarning: Comparing a sparse matrix with a scalar greater than zero using < is inefficient, try using >= instead.
warn(bad_scalar_msg, SparseEfficiencyWarning)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-60-6dc37c3cd667> in <module>
----> 1 adata = adata[adata.obs['percent_mito'] < 0.5, :]
/opt/venv/lib/python3.7/site-packages/pandas/core/ops/__init__.py in wrapper(self, other, axis)
1227
1228 with np.errstate(all="ignore"):
-> 1229 res = na_op(values, other)
1230 if is_scalar(res):
1231 raise TypeError(
/opt/venv/lib/python3.7/site-packages/pandas/core/ops/__init__.py in na_op(x, y)
1089
1090 if is_object_dtype(x.dtype):
-> 1091 result = _comp_method_OBJECT_ARRAY(op, x, y)
1092
1093 elif is_datetimelike_v_numeric(x, y):
/opt/venv/lib/python3.7/site-packages/pandas/core/ops/__init__.py in _comp_method_OBJECT_ARRAY(op, x, y)
1067 result = libops.vec_compare(x, y, op)
1068 else:
-> 1069 result = libops.scalar_compare(x, y, op)
1070 return result
1071
pandas/_libs/ops.pyx in pandas._libs.ops.scalar_compare()
/opt/venv/lib/python3.7/site-packages/scipy/sparse/base.py in __bool__(self)
285 return self.nnz != 0
286 else:
--> 287 raise ValueError("The truth value of an array with more than one "
288 "element is ambiguous. Use a.any() or a.all().")
289 __nonzero__ = __bool__
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
One solution is to convert to dense for the calculation:
adata.obs['percent_mito'] = np.ravel(np.sum(np.asmatrix(adata[:, mito_genes].X.todense()), axis=1)) / np.ravel(np.sum(adata.X, axis=1))
Metadata for aucell output
The example for creating a SCope compatible loom file calls an objects metadata. However, the loomfile, at least as it is outputed by PySCENIC, seems to contain no metadata field.
Tornado.util.TimeoutError timed out after 20 s
Hey everyone! I am trying to run scenic protocol, for network regulatory inference.
The code command is
pyscenic grn --num_workers 20 --output adj.tsv --method grnboost2 filtered.loom hs_hgnc_tfs.txt
And I am getting the following error
home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. data = yaml.load(f.read()) or {} 2020-07-21 22:55:37,466 - pyscenic.cli.pyscenic - INFO - Loading expression matrix. 2020-07-21 22:55:37,616 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks. /home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. data = yaml.load(f.read()) or {} tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 501, in callback result_list.append(f.result()) File "/home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 748, in run yielded = self.gen.send(value) File "/home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/deploy/local.py", line 316, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list File "/home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 501, in callback result_list.append(f.result()) File "/home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/tornado/gen.py", line 748, in run yielded = self.gen.send(value) File "/home/sidrah19220/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/distributed/deploy/local.py", line 316, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start Exception ignored in: <bound method="" localcluster.del="" of="" localcluster('tcp:="" 127.0.0.1:38723',="" workers="0," ncores="0)>" traceback="" (most="" recent="" call="" last):="" file="" "="" home="" sidrah19220="" anaconda3="" envs="" scenic_protocol="" lib="" python3.6="" site-packages="" distributed="" deploy="" local.py",="" line="" 444,="" in="" del="" file="" "="" home="" sidrah19220="" anaconda3="" envs="" scenic_protocol="" lib="" python3.6="" site-packages="" distributed="" deploy="" local.py",="" line="" 385,="" in="" close="" file="" "="" home="" sidrah19220="" anaconda3="" envs="" scenic_protocol="" lib="" python3.6="" site-packages="" distributed="" deploy="" local.py",="" line="" 249,="" in="" sync="" file="" "="" home="" sidrah19220="" anaconda3="" envs="" scenic_protocol="" lib="" python3.6="" site-packages="" distributed="" utils.py",="" line="" 326,="" in="" sync="" <strong="">tornado.util.TimeoutError: timed out after 20 s
Please suggest a solution
AttributeError: 'DataFrame' object has no attribute 'TF'
I tried the tutorial named pySCENIC protocol: PBMC10k using the dataset from 10X Genomics.
However, I got the error as followed. I checked the adj.csv, there seemed nothing wrong with it. Can anyone help me?
!pyscenic ctx /cluster/home/zfli/RProject/Lymphoma/SCENIC/pbmc10k/adjacencies.csv \
/cluster/home/zfli/Rcistarget/hg38/hg38__refseq-r80__500bp_up_and_100bp_down_tss.mc9nr.feather /cluster/home/zfli/Rcistarget/hg38/hg38__refseq-r80__10kb_up_and_down_tss.mc9nr.feather \
--annotations_fname /cluster/home/zfli/Rcistarget/motifs-v9-nr.hgnc-m0.001-o0.0.tbl \
--expression_mtx_fname /cluster/home/zfli/RProject/Lymphoma/SCENIC/pbmc10k/pbmc10k_filtered_scenic.loom \
--output /cluster/home/zfli/RProject/Lymphoma/SCENIC/pbmc10k/reg.csv \
--mask_dropouts \
--num_workers 4
2020-06-16 21:02:43,453 - pyscenic.cli.pyscenic - INFO - Creating modules.
2020-06-16 21:02:46,486 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2020-06-16 21:02:50,491 - pyscenic.utils - INFO - Calculating Pearson correlations.
2020-06-16 21:02:50,491 - pyscenic.utils - WARNING - Note on correlation calculation: the default behaviour for calculating the correlations has changed after pySCENIC verion 0.9.16. Previously, the default was to calculate the correlation between a TF and target gene using only cells with non-zero expression values (mask_dropouts=True). The current default is now to use all cells to match the behavior of the R verision of SCENIC. The original settings can be retained by setting 'rho_mask_dropouts=True' in the modules_from_adjacencies function, or '--mask_dropouts' from the CLI.
Dropout masking is currently set to [True].
Traceback (most recent call last):
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 421, in main
args.func(args)
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 140, in prune_targets_command
modules = adjacencies2modules(args)
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 109, in adjacencies2modules
keep_only_activating=(args.all_modules != "yes"))
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/lib/python3.6/site-packages/pyscenic/utils.py", line 268, in modules_from_adjacencies
rho_threshold=rho_threshold, mask_dropouts=rho_mask_dropouts)
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/lib/python3.6/site-packages/pyscenic/utils.py", line 129, in add_correlation
col_idx_pairs = _create_idx_pairs(adjacencies, ex_mtx)
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/lib/python3.6/site-packages/pyscenic/utils.py", line 70, in _create_idx_pairs
genes = set(adjacencies.TF).union(set(adjacencies.target))
File "/cluster/home/zfli/anaconda2/envs/scanpy_env/lib/python3.6/site-packages/pandas/core/generic.py", line 4376, in __getattr__
return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'TF'
Memory error while running on CLI
Hello,
I am using CLI to run the pyscenic. and it seems it has been installed properly.
I wanted to run the grn for a big dataset (17627, 23657) and after loading the expression matrix, I dealt with the following error. Does anyone have idea how i can fix it?
I play a lot with the --cpus-per-task and --mem-per-cpu, but error is repeated.
I would appreciate if anyone have an idea about it.
2022-08-21 00:52:03,871 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
Traceback (most recent call last):
File "/opt/venv/bin/pyscenic", line 8, in
sys.exit(main())
File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 675, in main
args.func(args)
File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 61, in find_adjacencies_command
args.gene_attribute,
File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/utils.py", line 143, in load_exp_matrix
df = pd.read_csv(fname, sep=suffixes_to_separator(extension), header=0, index_col=0)
File "/opt/venv/lib/python3.7/site-packages/pandas/io/parsers.py", line 610, in read_csv
return _read(filepath_or_buffer, kwds)
File "/opt/venv/lib/python3.7/site-packages/pandas/io/parsers.py", line 468, in _read
return parser.read(nrows)
File "/opt/venv/lib/python3.7/site-packages/pandas/io/parsers.py", line 1069, in read
df = DataFrame(col_dict, columns=columns, index=index)
File "/opt/venv/lib/python3.7/site-packages/pandas/core/frame.py", line 529, in init
mgr = init_dict(data, index, columns, dtype=dtype)
File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/construction.py", line 287, in init_dict
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/construction.py", line 95, in arrays_to_mgr
return create_block_manager_from_arrays(arrays, arr_names, axes)
File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1701, in create_block_manager_from_arrays
blocks = _form_blocks(arrays, names, axes)
File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1759, in _form_blocks
float_blocks = _multi_blockify(items_dict["FloatBlock"])
File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1852, in _multi_blockify
values, placement = _stack_arrays(list(tup_block), dtype)
File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1880, in _stack_arrays
stacked = np.empty(shape, dtype=dtype)
numpy.core._exceptions.MemoryError: Unable to allocate 3.11 GiB for an array with shape (17627, 23657) and data type float64
Problems with saving AUCell and regulon data into loom file using export2loom
Hi, I am trying to save AUCell and regulon data as loom file for further analysis in R.
I tried two codes,
export2loom(ex_mtx = ex_matrix, auc_mtx = auc_mtx, regulons = [r.rename(r.name.replace('(',' (')) for r in regulons], out_fname = "/path/file_name.loom")
and
export2loom(ex_mtx = ex_matrix, auc_mtx = auc_mtx, regulons = [r.rename(r.name.replace('(+)',' ('+str(len(r))+'g)')) for r in regulons], out_fname = "/path/file_name.loom")
However, it is taking more than 1 day to run.
Does this normally take that long or there is something wrong with my code?
I could save auc_mtx into csv files.
Thank you
pyscenic ctx: error: argument module_fname: can't open '{ADJACENCIES_FNAME}'
There is something wrong when I fellow the example.
It said that No such file or directory: '{ADJACENCIES_FNAME}'.
But I checked the file, it had no problem.
Can anyone help me?
Can the number of cells exceed 65536?
It seems that when runSCENIC_4_aucell_binarize call the hclust function in stats and the number of cells exceed 65536, it can't work.
scenicOptions <- runSCENIC_4_aucell_binarize(scenicOptions)
output:
Binary regulon activity: 26 TF regulons x 79211 cells.
(43 regulons including 'extended' versions)
26 regulons are active in more than 1% (792.11) cells.
Error in hclust(d, method = hclustfun) :
size cannot be NA nor exceed 65536
version
scenic 1.2.2
stats 4.0.3
Dask workers failing when working with large dataset on HPC
Hi there,
thanks for the great package. It makes it very easy to run the complete SCENIC workflow.
I have SCENICprotocol setup on an HPC running SLURM, singularity (aertslab-pySCENIC-0.9.18.img) and submitting the small test dataset works without a problem. When I run a small subset of my data (5% of complete dataset, ~3k cells, converted to loom from a Seurat object) it also works without an issue using 10 threads and 4G per thread.
When I try to run my entire dataset consisting of 62k cells, the pipeline fails during GRNinference.
I don't know where the issue is coming from, as the loom objects for the subset and the full dataset are similar except for cell number. Could this be related to RAM available on each worker?
Any help with this would be much appreciated!
Thanks, here are the command and errors:
This is the command executed in the pipeline:
executor > local (3)
[89/7e3133] process > filter [100%] 1 of 1 ✔
[16/745f91] process > preprocess [100%] 1 of 1, failed: 1
[- ] process > pca -
[- ] process > visualize -
[- ] process > cluster -
[b2/fbba24] process > GRNinference [100%] 1 of 1, failed: 1 ✘
[- ] process > cisTarget -
[- ] process > AUCell -
[- ] process > visualizeAUC -
[- ] process > integrateOutput -
WARN: Killing pending tasks (1)
Error executing process > 'GRNinference'
pyscenic grn --num_workers 11 -o adj.tsv --method grnboost2 --cell_id_attribute CellID --gene_attribute Gene filtered.loom Mouse_TFs.txt
And this is the error message:
Command exit status:
1
Command output:
(empty)
Command error:
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.send(value)
File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 268, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
Traceback (most recent call last):
File "/usr/local/bin/pyscenic", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 408, in main
args.func(args)
File "/usr/local/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 64, in find_adjacencies_command
client, shutdown_callback = _prepare_client(args.client_or_address, num_workers=args.num_workers)
File "/usr/local/lib/python3.7/site-packages/pyscenic/prune.py", line 63, in _prepare_client
threads_per_worker=1)
File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 177, in init
self.start(ip=ip, n_workers=n_workers)
File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 214, in start
self.sync(self._start, **kwargs)
File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 207, in sync
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 321, in sync
six.reraise(*error[0])
File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 306, in f
result[0] = yield future
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
value = future.result()
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 239, in _start
yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
value = future.result()
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.send(value)
File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 268, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-6, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-7, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-8, started daemon)>
distributed.nanny - WARNING - Worker process 203921 was killed by unknown signal
distributed.nanny - WARNING - Worker process 203925 was killed by signal 15
running jupyter notebook Error: Kernel died
[E 22:46:08.468 remote_ikernel] Kernel died.
[I 22:46:08.853 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel 59e2d9fd-b056-45da-95cf-05e6543e657b restarted
[E 22:46:08.854 NotebookApp] KernelRestarter: restart callback <bound method ZMQChannelsHandler.on_kernel_restarted of ZMQChannelsHandler(59e2d9fd-b056-45da-95cf-05e6543e657b)> failed
Traceback (most recent call last):
File "/home/zhang/miniconda3/lib/python3.8/site-packages/jupyter_client/restarter.py", line 86, in _fire_callbacks
callback()
File "/home/zhang/miniconda3/lib/python3.8/site-packages/notebook/services/kernels/handlers.py", line 476, in on_kernel_restarted
self._send_status_message('restarting')
File "/home/zhang/miniconda3/lib/python3.8/site-packages/notebook/services/kernels/handlers.py", line 472, in _send_status_message
self.write_message(json.dumps(msg, default=date_default))
File "/home/zhang/miniconda3/lib/python3.8/site-packages/tornado/websocket.py", line 339, in write_message
raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
[I 22:46:08.976 remote_ikernel] Remote kernel version: 0.4.6.
[I 22:46:08.976 remote_ikernel] File location: /home/zhang/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/remote_ikernel/kernel.py.
[I 22:46:08.977 remote_ikernel] Launching kernel over SSH.
[I 22:46:08.977 remote_ikernel] Login command: 'ssh -o StrictHostKeyChecking=no hostname2'.
[I 22:46:08.996 remote_ikernel] Established connection; starting kernel.
[I 22:46:08.996 remote_ikernel] Remote working directory /home/zhang//Desktop/pySCENIC/notebooks.
[I 22:46:09.147 remote_ikernel] Running kernel command: 'source /home/zhang/miniconda3/etc/profile.d/conda.sh && conda activate scenic_protocol && /home/zhang/miniconda3/envs/scenic_protocol/bin/ipython3 kernel -f ./rik_kernel-59e2d9fd-b056-45da-95cf-05e6543e657b.json'.
[I 22:46:09.476 remote_ikernel] Setting up tunnels on ports: 60137, 48085, 52635, 41725, 47981.
[E 22:46:12.254 remote_ikernel] Kernel died.
[I 22:46:12.639 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
[I 22:46:12.760 remote_ikernel] Remote kernel version: 0.4.6.
[I 22:46:12.760 remote_ikernel] File location: /home/zhang/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/remote_ikernel/kernel.py.
[I 22:46:12.760 remote_ikernel] Launching kernel over SSH.
[I 22:46:12.760 remote_ikernel] Login command: 'ssh -o StrictHostKeyChecking=no hostname'.
[I 22:46:12.777 remote_ikernel] Established connection; starting kernel.
[I 22:46:12.777 remote_ikernel] Remote working directory ~/.
[I 22:46:12.928 remote_ikernel] Running kernel command: 'source /home/zhang/miniconda3/etc/profile.d/conda.sh && conda activate scenic_protocol && /home/zhang/miniconda3/envs/scenic_protocol/bin/ipython3 kernel -f ./rik_kernel-77cfb9ea-6645-42ed-a876-48a465519bf4.json'.
[I 22:46:13.259 remote_ikernel] Setting up tunnels on ports: 45899, 39335, 40115, 59567, 34567.
[E 22:46:14.482 remote_ikernel] Kernel died.
Why Kernel always died? Thanks,
Fan
In running: PBMC10k_SCENIC-protocol-CLI.ipynb, please let know where this file: "exprMat" came from? thanks!
Would it be better to use raw/normalized/batch-effect-removed input count matrix?
Hi,
Thank you for the cool software! I noticed that the pipeline in your Nature Protocol paper used raw counts as input of the pySCENIC software. I'm wondering if it is better to use raw, normalized or batch effect removed matrix, or it doesn't matter (they would perhaps produce similar results)?
Thanks again!
H
Question about the input of `pyscenic ctx`
Hi,
Could you please give a further instructions on the database_fname of pyscenic ctx, as I encountered an error when use following command base on the turtorial
pyscenic ctx adj.csv \
test*.feather \
--annotations_fname CIS_BP.tbl \
--expression_mtx_fname CIM7D.loom\
--output reg.csv \
--mask_dropouts \
--num_workers 20
ls test*.feather
test.motifs_vs_regions.rankings.feather test.motifs_vs_regions.scores.feather
test.regions_vs_motifs.rankings.feather test.regions_vs_motifs.scores.featherThe feather files were generated according the instruction in https://github.com/aertslab/create_cisTarget_databases
The following was the error I encountered
File "/home/xxx/miniconda3/envs/pyscenic/lib/python3.7/site-packages/pyscenic/transform.py", line 301, in modules2df
[module2df(db, module, motif_annotations, weighted_recovery, False, module2features_func) for module in modules]
File "/home/xxx/miniconda3/envs/pyscenic/lib/python3.7/site-packages/pyscenic/transform.py", line 301, in <listcomp>
[module2df(db, module, motif_annotations, weighted_recovery, False, module2features_func) for module in modules]
File "/home/xxx/miniconda3/envs/pyscenic/lib/python3.7/site-packages/pyscenic/transform.py", line 231, in module2df
db, module, motif_annotations, weighted_recovery=weighted_recovery
File "/home/xxx/miniconda3/envs/pyscenic/lib/python3.7/site-packages/pyscenic/transform.py", line 152, in module2features_auc1st_impl
df = db.load(module)
File "/home/xxx/miniconda3/envs/pyscenic/lib/python3.7/site-packages/ctxcore/rnkdb.py", line 327, in load
df.set_index(self._index_name, inplace=True)
File "/home/xxx/miniconda3/envs/pyscenic/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(args, **kwargs)
File "/home/xxx/miniconda3/envs/pyscenic/lib/python3.7/site-packages/pandas/core/frame.py", line 5451, in set_index
raise KeyError(f"None of {missing} are in the columns")
KeyError: "None of ['features'] are in the columns"
Installation says python3.6 but pandas1.3.5 requires python3.7+
On conda installation page it says conda create -n scenic_protocol python=3.6 however when doing pip install pyscenic it tells me no available pandas1.3.5, and according to pandas their 1.3.5 is only available for python 3.7 or higher version.
(installed successfully with re-creating a python=3.7 env, opening issue to suggest maybe modify the installation instruction?)
pyscenic issue with the example data set
Dear SCENIC developers,
I was trying to follow the tutorial (except for skipping the Scanpy part) for the PBMC10k protocol.
But I got an error in the first grn step.
!pyscenic grn {f_loom_path_scenic} {f_tfs} -o adj.csv --num_workers 20
/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
2020-01-24 14:09:44,508 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2020-01-24 14:09:44,534 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
Traceback (most recent call last):
File "", line 1, in
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 243, in main
fds = reduction.recvfds(s, MAXFDS_TO_SEND + 1)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 155, in recvfds
raise EOFError
EOFError
distributed.nanny - WARNING - Worker process 47169 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47167 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47165 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47163 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47161 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47160 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47168 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47166 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47162 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47171 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47170 was killed by unknown signal
distributed.nanny - WARNING - Worker process 47164 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self._semlock = _multiprocessing.SemLock._rebuild(*state)
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
FileNotFoundError: [Errno 2] No such file or directory
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/synchronize.py", line 110, in setstate
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 196, in _start
response = yield self.instantiate()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 276, in instantiate
timedelta(seconds=self.death_timeout), self.process.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 439, in start
yield self.process.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 35, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/process.py", line 192, in _start
process.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/popen_forkserver.py", line 35, in init
super().init(process_obj)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/forkserver.py", line 70, in connect_to_new_process
child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 196, in _start
response = yield self.instantiate()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 276, in instantiate
timedelta(seconds=self.death_timeout), self.process.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 416, in start
self.init_result_q = init_q = mp_context.Queue()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/context.py", line 102, in Queue
return Queue(maxsize, ctx=self.get_context())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/queues.py", line 41, in init
self._reader, self._writer = connection.Pipe(duplex=False)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/connection.py", line 517, in Pipe
fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 196, in _start
response = yield self.instantiate()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 276, in instantiate
timedelta(seconds=self.death_timeout), self.process.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 416, in start
self.init_result_q = init_q = mp_context.Queue()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/context.py", line 102, in Queue
return Queue(maxsize, ctx=self.get_context())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/queues.py", line 41, in init
self._reader, self._writer = connection.Pipe(duplex=False)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/multiprocessing/connection.py", line 517, in Pipe
fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 184, in _start
listen_args=self.listen_args,
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/core.py", line 322, in listen
self.listener.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 421, in start
self.port, address=self.ip, backlog=backlog
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/netutil.py", line 140, in bind_sockets
sock = socket.socket(af, socktype, proto)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/socket.py", line 151, in init
_socket.socket.init(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 184, in _start
listen_args=self.listen_args,
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/core.py", line 322, in listen
self.listener.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 421, in start
self.port, address=self.ip, backlog=backlog
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/netutil.py", line 140, in bind_sockets
sock = socket.socket(af, socktype, proto)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/socket.py", line 151, in init
_socket.socket.init(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 184, in _start
listen_args=self.listen_args,
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/core.py", line 322, in listen
self.listener.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 421, in start
self.port, address=self.ip, backlog=backlog
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/netutil.py", line 140, in bind_sockets
sock = socket.socket(af, socktype, proto)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/socket.py", line 151, in init
_socket.socket.init(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 184, in _start
listen_args=self.listen_args,
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/core.py", line 322, in listen
self.listener.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 421, in start
self.port, address=self.ip, backlog=backlog
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/netutil.py", line 140, in bind_sockets
sock = socket.socket(af, socktype, proto)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/socket.py", line 151, in init
_socket.socket.init(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 306, in _start_worker
**kwargs
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.await())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/nanny.py", line 184, in _start
listen_args=self.listen_args,
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/core.py", line 322, in listen
self.listener.start()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 421, in start
self.port, address=self.ip, backlog=backlog
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/netutil.py", line 140, in bind_sockets
sock = socket.socket(af, socktype, proto)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/socket.py", line 151, in init
_socket.socket.init(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files
Traceback (most recent call last):
File "/Users/levylabmini2/opt/anaconda3/bin/pyscenic", line 10, in
sys.exit(main())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 408, in main
args.func(args)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 64, in find_adjacencies_command
client, shutdown_callback = _prepare_client(args.client_or_address, num_workers=args.num_workers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/pyscenic/prune.py", line 63, in _prepare_client
threads_per_worker=1)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 218, in init
self.start(ip=ip, n_workers=n_workers)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 256, in start
self.sync(self._start, **kwargs)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 249, in sync
return sync(self.loop, func, *args, **kwargs)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/utils.py", line 331, in sync
six.reraise(*error[0])
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/utils.py", line 316, in f
result[0] = yield future
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 284, in _start
yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/Users/levylabmini2/opt/anaconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
error in: read in the adjacencies matrix: FileNotFoundError: [Errno 2] File b'adj.tsv' does not exist: b'adj.tsv'
read in the adjacencies matrix:
adjacencies = pd.read_csv("adj.tsv", index_col=False, sep='\t')
FileNotFoundError Traceback (most recent call last)
in
----> 1 adjacencies = pd.read_csv("adj.tsv", index_col=False, sep='\t')
~/miniconda3/envs/scenic_protocol_2/lib/python3.6/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)
683 )
684
--> 685 return _read(filepath_or_buffer, kwds)
686
687 parser_f.name = name
~/miniconda3/envs/scenic_protocol_2/lib/python3.6/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
455
456 # Create the parser.
--> 457 parser = TextFileReader(fp_or_buf, **kwds)
458
459 if chunksize or iterator:
~/miniconda3/envs/scenic_protocol_2/lib/python3.6/site-packages/pandas/io/parsers.py in init(self, f, engine, **kwds)
893 self.options["has_index_names"] = kwds["has_index_names"]
894
--> 895 self._make_engine(self.engine)
896
897 def close(self):
~/miniconda3/envs/scenic_protocol_2/lib/python3.6/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
1133 def _make_engine(self, engine="c"):
1134 if engine == "c":
-> 1135 self._engine = CParserWrapper(self.f, **self.options)
1136 else:
1137 if engine == "python":
~/miniconda3/envs/scenic_protocol_2/lib/python3.6/site-packages/pandas/io/parsers.py in init(self, src, **kwds)
1915 kwds["usecols"] = self.usecols
1916
-> 1917 self._reader = parsers.TextReader(src, **kwds)
1918 self.unnamed_cols = self._reader.unnamed_cols
1919
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.cinit()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._setup_parser_source()
FileNotFoundError: [Errno 2] File b'adj.tsv' does not exist: b'adj.tsv'
#####################
Let me how to solve this problem. Thanks!
Fan
Problem with further exploring module files in iRegulon
Hi here,
I followed the jupyter notebook named PBMC10k_downstream-analysis to explore module files in iRegulon.
When I picked out modules for my target genes, there's something wrong.
adjacencies = pd.read_csv("/cluster/home/zfli/RProject/MM_single/SCENIC/results/MM.adjacencies.tsv", index_col=False, sep='\t')
exprMat=pd.read_csv(EXP_MTX_QC_FNAME,index_col=0)
from pyscenic.utils import modules_from_adjacencies
modules = list(modules_from_adjacencies(adjacencies, exprMat))
tf = 'MYBL2'
tf_mods = [ x for x in modules if x.transcription_factor==tf ]
for i,mod in enumerate( tf_mods ):
print( f'{tf} module {str(i)}: {len(mod.genes)} genes' )`
MYBL2 module 0: 4301 genes
MYBL2 module 1: 3240 genes
MYBL2 module 2: 51 genes
MYBL2 module 3: 47 genes
MYBL2 module 4: 644 genes
MYBL2 module 5: 3351 genes
print( f'{tf} regulon: {len(regulons[tf+"(+)"])} genes' )
TypeError Traceback (most recent call last)
in
----> 1 print( f'{tf} regulon: {len(regulons[tf+"(+)"])} genes' )
TypeError: list indices must be integers or slices, not str
Then I checked the regulon list.
regulons[1]
Regulon(name='ALX1(+)', gene2weight=<frozendict {'NFATC3': 0.4666430402753485, 'LGALS3': 2.9417811594561583, 'PJA1': 0.7149351863285957, 'CA1': 2.9417811594561583, 'FLNA': 1.3756471078801764, 'TUBA1A': 0.37228583969436796, 'PRMT2': 1.0, 'RUNX1T1': 1.6734605118620007, 'LRRC23': 1.1354697828746674, 'ADI1': 0.9461970789505432, 'CD36': 1.0787344073180478, 'DTNB': 1.6851222353044415, 'BOLA2B': 1.9447241642326292, 'PACSIN2': 1.3756471078801764, 'ENTPD1': 1.0, 'ADGRE1': 1.6851222353044415, 'CPPED1': 4.292454650246219}>, gene2occurrence=<frozendict {}>, transcription_factor='ALX1', context=frozenset({'activating', 'taipale_cyt_meth__PROP1_TAATTRCGTTA_FL_meth.png'}), score=3.026860825133384, nes=0.0, orthologous_identity=0.0, similarity_qvalue=0.0, annotation='')
How can I adjust my codes? Thanks a lot!!!
OSError: Unable to open file (file signature not found)
Hi,
I am following the procedure outlined in the Nature Protocols paper. At step 7, cellular enrichment, I run into an error. The command I'm using is: pyscenic aucell PBMC10k_filtered.loom reg.csv --output PBMC10k_SCENIC.loom --num_workers 20
This results in the following output in the CLI:
2021-04-09 17:54:44,095 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
Traceback (most recent call last):
File "/home/luna.kuleuven.be/u0124158/miniconda3/envs/pyscenic/bin/pyscenic", line 8, in
sys.exit(main())
File "/home/luna.kuleuven.be/u0124158/miniconda3/envs/pyscenic/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 485, in main
args.func(args)
File "/home/luna.kuleuven.be/u0124158/miniconda3/envs/pyscenic/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 216, in aucell_command
args.gene_attribute)
File "/home/luna.kuleuven.be/u0124158/miniconda3/envs/pyscenic/lib/python3.6/site-packages/pyscenic/cli/utils.py", line 116, in load_exp_matrix
return load_exp_matrix_as_loom(fname, return_sparse, attribute_name_cell_id, attribute_name_gene)
File "/home/luna.kuleuven.be/u0124158/miniconda3/envs/pyscenic/lib/python3.6/site-packages/pyscenic/cli/utils.py", line 69, in load_exp_matrix_as_loom
with lp.connect(fname,mode='r',validate=False) as ds:
File "/home/luna.kuleuven.be/u0124158/.local/lib/python3.6/site-packages/loompy/loompy.py", line 1389, in connect
return LoomConnection(filename, mode, validate=validate)
File "/home/luna.kuleuven.be/u0124158/.local/lib/python3.6/site-packages/loompy/loompy.py", line 84, in init
self._file = h5py.File(filename, mode)
File "/home/luna.kuleuven.be/u0124158/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 427, in init
swmr=swmr)
File "/home/luna.kuleuven.be/u0124158/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (file signature not found)
My OS is Ubuntu 18.0.4.5 LTS.
Any tips on how to solve this issue?
Thank you very much in advance!
Best,
Tim
Protocol
Hi, I am Steven,
Thank you for your work, I have issues concerning the link for the protocol :
https://github.com/aertslab/SCENICprotocol/blob/master/notebooks/PBMC10k_SCENIC-protocol-CLI.ipynb
The code doesn't appear.
Cordially,
Steven
ValueError: Intersection of gene_names and tf_names is empty.
Hello,
Thank you for this resource! I am running the PBMC10k_SCENIC-protocol-CLI.ipynb protocol, and when I run SCENIC Steps phase 1a, I get an error.
!pyscenic grn {f_loom_path_scenic} {f_tfs} -o adj.csv --num_workers 20
ValueError: Intersection of gene_names and tf_names is empty.
I have tried transposing the original matrix (as as is suggested in another issues post) but I still get the same error.
Is there another reason I could be getting this error?
Thank you!
ImportError: cannot import name 'plot_scatter' from 'scanpy.plotting._tools.scatterplots'
Hello, there is something wrong when I import the package, and I found that there is no function called "scatterplots" in the .py file. Could you help me deal with the issue? Thank you so much!
Singularity: can't open file 'pyscenic'
I've installed singularity on fresh ubuntu instance and trying to run scenic according to the documentation:
singularity pull --name aertslab-pyscenic-0.9.18.sif shub://aertslab/pySCENIC:0.9.18
However, I get the error when running:
singularity run aertslab-pyscenic-0.9.18.sif pyscenic -h
python3: can't open file 'pyscenic': [Errno 2] No such file or directory
I have also set up a kernel for Jypyter from singularity container, however Jupyter can't connect to it.
Are the instructions up to date? Am I missing something?
cannot run scenic protocol
Hi, I am trying to run scenic on google colab but I get this error can you help me out?
!pyscenic grn {'Opto_d14_filtered_scenic.loom'} {f_tfs} -o adj.csv --num_workers 8
2020-12-09 18:27:17,987 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2020-12-09 18:27:21,650 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
preparing dask client
parsing input
/usr/local/lib/python3.6/dist-packages/arboreto/algo.py:214: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
expression_matrix = expression_data.as_matrix()
creating dask graph
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.core - ERROR - tuple is not allowed for map key
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/distributed/core.py", line 297, in handle_comm
msg = yield comm.read()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/tcp.py", line 207, in read
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 326, in wrapper
yielded = next(result)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 82, in from_frames
res = _from_frames()
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 68, in _from_frames
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
not shutting down client, client was created externally
finished
Traceback (most recent call last):
File "/usr/local/bin/pyscenic", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/pyscenic/cli/pyscenic.py", line 470, in main
args.func(args)
File "/usr/local/lib/python3.6/dist-packages/pyscenic/cli/pyscenic.py", line 74, in find_adjacencies_command
network = method(expression_data=ex_mtx, tf_names=tf_names, verbose=True, client_or_address=client, seed=args.seed)
File "/usr/local/lib/python3.6/dist-packages/arboreto/algo.py", line 41, in grnboost2
early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
File "/usr/local/lib/python3.6/dist-packages/arboreto/algo.py", line 128, in diy
seed=seed)
File "/usr/local/lib/python3.6/dist-packages/arboreto/core.py", line 403, in create_graph
future_tf_matrix = client.scatter(tf_matrix, broadcast=True)
File "/usr/local/lib/python3.6/dist-packages/distributed/client.py", line 1874, in scatter
asynchronous=asynchronous, hash=hash)
File "/usr/local/lib/python3.6/dist-packages/distributed/client.py", line 675, in sync
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/distributed/utils.py", line 277, in sync
six.reraise(*error[0])
File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/distributed/utils.py", line 262, in f
result[0] = yield future
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/client.py", line 1736, in _scatter
timeout=timeout)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/core.py", line 580, in send_recv_from_rpc
result = yield send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/core.py", line 457, in send_recv
response = yield comm.read(deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/tcp.py", line 207, in read
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 326, in wrapper
yielded = next(result)
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 82, in from_frames
res = _from_frames()
File "/usr/local/lib/python3.6/dist-packages/distributed/comm/utils.py", line 68, in _from_frames
deserializers=deserializers)
File "/usr/local/lib/python3.6/dist-packages/distributed/protocol/core.py", line 114, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
Requirements:
Requirement already satisfied: pyscenic in /usr/local/lib/python3.6/dist-packages (0.10.4)
Requirement already satisfied: frozendict in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.2)
Requirement already satisfied: umap-learn in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.4.6)
Requirement already satisfied: arboreto in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.1.5)
Requirement already satisfied: boltons in /usr/local/lib/python3.6/dist-packages (from pyscenic) (20.2.1)
Requirement already satisfied: llvmlite in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.31.0)
Requirement already satisfied: numba in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.48.0)
Requirement already satisfied: pandas<1.0.0,>=0.20.1 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.25.3)
Requirement already satisfied: cloudpickle in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.3.0)
Requirement already satisfied: loompy in /usr/local/lib/python3.6/dist-packages (from pyscenic) (3.0.6)
Requirement already satisfied: distributed<2.0.0,>=1.21.6 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.25.3)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pyscenic) (4.41.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.4.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from pyscenic) (50.3.2)
Requirement already satisfied: multiprocessing-on-dill in /usr/local/lib/python3.6/dist-packages (from pyscenic) (3.5.0a4)
Requirement already satisfied: cytoolz in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.11.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from pyscenic) (2.5)
Requirement already satisfied: interlap in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.2.7)
Requirement already satisfied: dask==1.0.0 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.0.0)
Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from pyscenic) (20.3.0)
Requirement already satisfied: pyarrow<0.17.0,>=0.11.1 in /usr/local/lib/python3.6/dist-packages (from pyscenic) (0.14.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pyscenic) (1.18.5)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from pyscenic) (3.13)
Requirement already satisfied: scikit-learn>=0.20 in /usr/local/lib/python3.6/dist-packages (from umap-learn->pyscenic) (0.22.2.post1)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas<1.0.0,>=0.20.1->pyscenic) (2018.9)
Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas<1.0.0,>=0.20.1->pyscenic) (2.8.1)
Requirement already satisfied: numpy-groupies in /usr/local/lib/python3.6/dist-packages (from loompy->pyscenic) (0.9.13)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from loompy->pyscenic) (7.1.2)
Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from loompy->pyscenic) (2.10.0)
Requirement already satisfied: msgpack in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (1.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (2.3.0)
Requirement already satisfied: tblib in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (1.7.0)
Requirement already satisfied: tornado>=4.5.1 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (5.1.1)
Requirement already satisfied: psutil>=5.0 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (5.4.8)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (1.15.0)
Requirement already satisfied: toolz>=0.7.4 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (0.11.1)
Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from distributed<2.0.0,>=1.21.6->pyscenic) (2.0.0)
Requirement already satisfied: dill in /usr/local/lib/python3.6/dist-packages (from multiprocessing-on-dill->pyscenic) (0.3.3)
Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->pyscenic) (4.4.2)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20->umap-learn->pyscenic) (0.17.0)
Requirement already satisfied: heapdict in /usr/local/lib/python3.6/dist-packages (from zict>=0.1.3->distributed<2.0.0,>=1.21.6->pyscenic) (1.0.1)
Interactive use with jupyter notebook and docker command
Hello,
I found this in the installation section:
python3 -m remote_ikernel manage --add \ --kernel_cmd="singularity run -B /data /path/to/aertslab-pyscenic-0.9.18.sif ipython kernel -f {connection_file}" \ --name="pyscenic-0918-singularity" \ --interface=ssh \ --host=hostname \ --workdir="~/" \ --language=python3
I would like to use docker instead of singularity. I tried to adapt the command line but it is not working.
Could you please help me to adapt the command line in order to use jupyter in the scenic docker image?
Thank you.
AttributeError: module 'numba' has no attribute 'jitclass'
I tried the tutorial to install the pySCENIC followed by the file named installation.md.
However,when I tried the basic test for the whole environment, I got the error as followed. There is no attribute 'jitclass' for 'numba' . Can anyone help me?
pyscenic -h
/Users/lichengyu/opt/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
Traceback (most recent call last):
File "/Users/lichengyu/opt/anaconda3/envs/scenic_protocol/bin/pyscenic", line 5, in <module>
from pyscenic.cli.pyscenic import main
File "/Users/lichengyu/opt/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 19, in <module>
from pyscenic.utils import modules_from_adjacencies, add_correlation
File "/Users/lichengyu/opt/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/utils.py", line 6, in <module>
from .math import masked_rho4pairs
File "/Users/lichengyu/opt/anaconda3/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/math.py", line 5, in <module>
from numba import *
AttributeError: module 'numba' has no attribute 'jitclass'
Here are all my installed packages in the environment.
anndata 0.7.5 pypi_0 pypi
arboreto 0.1.5 pypi_0 pypi
attrs 20.3.0 pypi_0 pypi
boltons 20.2.1 pypi_0 pypi
ca-certificates 2020.12.5 h033912b_0 conda-forge
cached-property 1.5.2 pypi_0 pypi
cairo 1.16.0 h4ec454a_1007 conda-forge
certifi 2020.12.5 py36h79c6626_1 conda-forge
cffi 1.14.4 py36hfe0777f_1 conda-forge
click 7.1.2 pypi_0 pypi
cloudpickle 1.6.0 pypi_0 pypi
cycler 0.10.0 py_2 conda-forge
cytoolz 0.11.0 py36haf1e3a3_0 anaconda
dask 1.0.0 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
dill 0.3.3 pypi_0 pypi
distributed 1.28.1 pypi_0 pypi
fontconfig 2.13.1 hd23ceaa_1004 conda-forge
freetype 2.10.4 h4cff582_1 conda-forge
frozendict 1.2 pypi_0 pypi
get-version 2.1 pypi_0 pypi
gettext 0.19.8.1 h7937167_1005 conda-forge
gmp 6.2.1 h2e338ed_0 conda-forge
h5py 3.1.0 pypi_0 pypi
heapdict 1.0.1 pypi_0 pypi
icu 68.1 h74dc148_0 conda-forge
importlib-metadata 3.4.0 pypi_0 pypi
interlap 0.2.7 pypi_0 pypi
joblib 1.0.0 pypi_0 pypi
jpeg 9d hbcb3906_0 conda-forge
kiwisolver 1.3.1 py36h615c93b_1 conda-forge
lcms2 2.11 h11f7e16_1 conda-forge
legacy-api-wrap 1.2 pypi_0 pypi
libblas 3.9.0 7_openblas conda-forge
libcblas 3.9.0 7_openblas conda-forge
libcxx 11.0.1 habf9029_0 conda-forge
libffi 3.3 h046ec9c_2 conda-forge
libgfortran 5.0.0 9_3_0_h6c81a4c_17 conda-forge
libgfortran5 9.3.0 h6c81a4c_17 conda-forge
libglib 2.66.4 h7424822_1 conda-forge
libiconv 1.16 haf1e3a3_0 conda-forge
liblapack 3.9.0 7_openblas conda-forge
libopenblas 0.3.12 openmp_h54245bb_1 conda-forge
libpng 1.6.37 h7cec526_2 conda-forge
libtiff 4.2.0 h355d032_0 conda-forge
libwebp-base 1.1.0 hbcb3906_3 conda-forge
libxml2 2.9.10 h802ca4f_3 conda-forge
llvm-openmp 11.0.1 h7c73e74_0 conda-forge
llvmlite 0.35.0 pypi_0 pypi
loompy 3.0.6 pypi_0 pypi
louvain 0.7.0 py36h59ef8ac_1 conda-forge
lz4-c 1.9.3 h046ec9c_0 conda-forge
matplotlib 3.3.3 py36h79c6626_0 conda-forge
matplotlib-base 3.3.3 py36hbcb57cb_0 conda-forge
msgpack 1.0.2 pypi_0 pypi
multicore-tsne 0.1_d4ff4aab py36h3e44d54_0 conda-forge
multiprocessing-on-dill 3.5.0a4 pypi_0 pypi
natsort 7.1.0 pypi_0 pypi
ncurses 6.2 h2e338ed_4 conda-forge
networkx 2.5 pypi_0 pypi
numba 0.52.0 pypi_0 pypi
numexpr 2.7.2 pypi_0 pypi
numpy 1.19.5 py36h08dc641_1 conda-forge
numpy-groupies 0.9.13 pypi_0 pypi
olefile 0.46 pyh9f0ad1d_1 conda-forge
openssl 1.1.1i h35c211d_0 conda-forge
packaging 20.8 pypi_0 pypi
pandas 0.25.3 pypi_0 pypi
patsy 0.5.1 py_0 conda-forge
pcre 8.44 hb1e8313_0 conda-forge
pillow 8.1.0 py36hc04f27d_1 conda-forge
pip 20.3.3 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 hbcb3906_0 conda-forge
psutil 5.8.0 pypi_0 pypi
pyarrow 0.16.0 pypi_0 pypi
pycairo 1.20.0 py36h7d97dce_1 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyscenic 0.10.4 pypi_0 pypi
python 3.6.12 h6c3b2c9_0_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-igraph 0.8.3 py36he18b813_3 conda-forge
python_abi 3.6 1_cp36m conda-forge
pytz 2020.5 pyhd8ed1ab_0 conda-forge
pyyaml 5.4.1 pypi_0 pypi
readline 8.0 h0678c8f_2 conda-forge
scanpy 1.6.1 pypi_0 pypi
scikit-learn 0.24.1 pypi_0 pypi
scipy 1.5.3 py36h04de62b_0 conda-forge
seaborn 0.11.1 h694c41f_0 conda-forge
seaborn-base 0.11.1 pyhd8ed1ab_0 conda-forge
setuptools 49.6.0 py36h79c6626_3 conda-forge
setuptools-scm 5.0.1 pypi_0 pypi
sinfo 0.3.1 pypi_0 pypi
six 1.15.0 pyh9f0ad1d_0 conda-forge
sortedcontainers 2.3.0 pypi_0 pypi
sqlite 3.34.0 h17101e1_0 conda-forge
statsmodels 0.12.1 py36h809c706_2 conda-forge
stdlib-list 0.8.0 pypi_0 pypi
tables 3.6.1 pypi_0 pypi
tblib 1.7.0 pypi_0 pypi
texttable 1.6.3 pyh9f0ad1d_0 conda-forge
threadpoolctl 2.1.0 pypi_0 pypi
tk 8.6.10 h0419947_1 conda-forge
toolz 0.11.1 py_0 anaconda
tornado 6.1 py36h20b66c6_1 conda-forge
tqdm 4.56.0 pypi_0 pypi
typing-extensions 3.7.4.3 pypi_0 pypi
umap-learn 0.4.6 pypi_0 pypi
wheel 0.36.2 pyhd3deb0d_0 conda-forge
xz 5.2.5 haf1e3a3_1 conda-forge
zict 2.0.0 pypi_0 pypi
zipp 3.4.0 pypi_0 pypi
zlib 1.2.11 h7795811_1010 conda-forge
zstd 1.4.8 hf387650_1 conda-forge```
Memory required for running pyscenic
Hello,
I am trying to run the dummy dataset using the given pyscenic workflow on the cluster using slurm script. However, when I assigned 180G memory usage, still it is giving me the error for memory exhaustion. When I submitted the job without any slurm script, it gave an overconsumption issue.
Showing you a glimpse of the error generated:
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
Can I please be suggested an alternative for it? Or can I please get to know how much the memory will be needed for running? This is a sample data given in the tutorial.
I will be grateful for your help.
Thanks.
Not able to change the pyscenic version to 0.10.4 in the nextflow pipeline --pyscenic_container
Hi,
I am trying to use the nextflow implementation of pyscenic using the pipeline given below,
Original Pipeline:
nextflow run aertslab/SCENICprotocol -profile singularity --expr raw_gene_expression_df.csv --output pyscenic_output.csv --TFs allTFs_mm.txt --motifs motifs-v9-nr.mgi-m0.001-o0.0.tbl --db '*feather' --nb_runs 3
Modified Pipeline:
In the modified one I wanted to use the singularity pyscenic container aertslab-pyscenic-0.10.4.sif as given below,
nextflow run aertslab/scenic-nf --pyscenic_container aertslab-pyscenic-0.10.4.sif --expr raw_gene_expression_df.csv --output pyscenic_output.csv --TFs allTFs_mm.txt --motifs motifs-v9-nr.mgi-m0.001-o0.0.tbl --db '*feather' --nb_runs 3
But when I run the modified version I get the following error,
cheema@gauss07: $ nextflow run aertslab/scenic-nf --pyscenic_container example/aertslab-pyscenic-0.10.4.sif --expr raw_gene_expression_df.csv --output pyscenic_output.csv --TFs allTFs_mm.txt --motifs motifs-v9-nr.mgi-m0.001- o0.0.tbl --db '*feather' --nb_runs 3
N E X T F L O W ~ version 21.10.6
Launching aertslab/scenic-nf [astonishing_poitras] - revision: 7f253de81c [master]
***
Parameters in use:
expr=raw_gene_expression_df.csv
TFs=allTFs_mm.txt
motifs=motifs-v9-nr.mgi-m0.001-o0.0.tbl
db=*feather
output=pyscenic_output.csv
outdir=.
grn=grnboost2
threads=6
cell_id_attribute=CellID
gene_attribute=Gene
nb_runs=3
tag=0.9.9
qsubaccount=
pyscenic_container=aertslab- pyscenic-0.10.4.sif
***
Using 2 feather databases:
/mnt/DOSI/MDALAB/USERS/Active/Ammar_Sabir_Cheema/scenic_analysis/example/mm10__refseq- r80__10kb_up_and_down_tss.mc9nr.feather
/mnt/DOSI/MDALAB/USERS/Active/Ammar_Sabir_Cheema/scenic_analysis/example/mm10__refseq-r80__500bp_up_and_100bp_down_tss.mc9nr.feather
***
executor > local (3)
executor > local (3)
[9e/cdc19a] process > GRNinference (2) [100%] 3 of 3, failed: 3 ✘
[- ] process > cisTarget -
[- ] process > AUCell -
Error executing process > 'GRNinference (3)'
Caused by:
Process `GRNinference (3)` terminated with an error exit status (127)
Command executed:
pyscenic grn --num_workers 6 -o "run_3__adj.tsv" --method grnboost2 --cell_id_attribute CellID --gene_attribute Gene raw_gene_expression_df.csv allTFs_mm.txt
Command exit status:
127
Command output:
(empty)
Command error:
.command.sh: line 2: pyscenic: command not found
Work dir:
/mnt/DOSI/MDALAB/USERS/Active/Ammar_Sabir_Cheema/scenic_analysis/example/work/fd/b0efb5ef71d5c2f8571426b4e3fb21
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Can you please let me know if there is any way to solve this problem? So that I can use 0.10.4 version of pyscenic singularity image?
Thanks,
Ammar
Nextflow broken pipe - GRNinference never starts
Hi,
First thanks for developing this great tool!
I am trying to run the full SCENIC pipeline using Nextflow and docker on our AWS linux instance (m5.12xlarge) but the pipeline seems to get stuck at the GRNinference step.
There is no error but after a while (~2h stuck at 0%), the instance disconnect and return "client_loop: send disconnect: Broken pipe".
On my local ubuntu, same thing happen, pipeline get stuck at GRNinference.
Do you have any idea what could be the problem?
Thanks in advance! I enclosed the CLI log file.
Theo
AUCell - AttributeError: 'Series' object has no attribute 'iteritems'
I get the following error while trying to run AUCell step from the tutorial (http://htmlpreview.github.io/?https://github.com/aertslab/SCENICprotocol/blob/master/notebooks/PBMC10k_SCENIC-protocol-CLI.html)
/home/xx/.local/bin/pyscenic aucell filtered_scenic.loom reg.csv --output pyscenic_output.loom --num_workers 4
2023-05-03 15:40:13,037 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2023-05-03 15:40:14,208 - pyscenic.cli.pyscenic - INFO - Loading gene signatures.
Create regulons from a dataframe of enriched features.
Additional columns saved: []
2023-05-03 15:40:14,746 - pyscenic.cli.pyscenic - INFO - Calculating cellular enrichment.
2023-05-03 15:40:22,311 - pyscenic.cli.pyscenic - INFO - Writing results to file.
Traceback (most recent call last):
File "/home/xx/.local/bin/pyscenic", line 8, in
sys.exit(main())
File "/home/xx/.local/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
args.func(args)
File "/home/xx/.local/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 296, in aucell_command
append_auc_mtx(
File "/home/xx/.local/lib/python3.10/site-packages/pyscenic/cli/utils.py", line 347, in append_auc_mtx
for name, threshold in auc_thresholds.iteritems()
File "/home/xx/.local/lib/python3.10/site-packages/pandas/core/generic.py", line 5989, in getattr
return object.getattribute(self, name)
AttributeError: 'Series' object has no attribute 'iteritems'
I tried running this from CLI as well as notebook but get the same error
system info -
anndata 0.9.1
scanpy 1.9.3
MulticoreTSNE NA
PIL 9.5.0
asttokens NA
backcall 0.2.0
cffi 1.15.1
cloudpickle 2.2.1
colorama 0.4.6
comm 0.1.3
cycler 0.10.0
cython_runtime NA
cytoolz 0.12.1
dask 2023.4.0
dateutil 2.8.2
debugpy 1.6.7
decorator 5.1.1
defusedxml 0.7.1
executing 1.2.0
h5py 3.8.0
igraph 0.10.4
ipykernel 6.22.0
ipython_genutils 0.2.0
jedi 0.18.2
jinja2 3.1.2
joblib 1.2.0
kiwisolver 1.4.4
llvmlite 0.39.1
loompy 3.0.7
louvain 0.8.0
lz4 4.3.2
markupsafe 2.1.2
matplotlib 3.7.1
mpl_toolkits NA
natsort 8.3.1
nt NA
ntsecuritycon NA
numba 0.56.4
numexpr 2.8.4
numpy 1.23.5
numpy_groupies 0.9.20
packaging 23.1
pandas 2.0.1
parso 0.8.3
pickleshare 0.7.5
pkg_resources NA
platformdirs 3.3.0
prompt_toolkit 3.0.38
psutil 5.9.5
pure_eval 0.2.2
pyarrow 11.0.0
pycparser 2.21
pydev_ipython NA
pydevconsole NA
pydevd 2.9.5
pydevd_file_utils NA
pydevd_plugins NA
pydevd_tracing NA
pygments 2.15.1
pyparsing 3.0.9
pythoncom NA
pytz 2023.3
pywin32_bootstrap NA
pywin32_system32 NA
pywintypes NA
scipy 1.10.1
seaborn 0.12.2
session_info 1.0.0
setuptools 67.7.2
six 1.16.0
sklearn 1.2.2
stack_data 0.6.2
statsmodels 0.13.5
tblib 1.7.0
texttable 1.6.7
threadpoolctl 3.1.0
tlz 0.12.1
toolz 0.12.0
tornado 6.3.1
traitlets 5.9.0
wcwidth 0.2.6
win32api NA
win32com NA
win32security NA
yaml 6.0
zipp NA
zmq 25.0.2
zoneinfo NA
IPython 8.12.0
jupyter_client 8.2.0
jupyter_core 5.3.0
notebook 6.5.4
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:00:38) [MSC v.1934 64 bit (AMD64)]
Windows-10-10.0.19044-SP0
Singularity containers do not fully isolate the user home
In cases where there is already a local Scanpy install in the user home directory:
~/.local/lib/python3.7/site-packages/scanpy, Singularity tries to use this install before the container version, resulting in:
Error executing process > 'filter'
Caused by:
Process `filter` terminated with an error exit status (1)
Command executed:
filtering-basic.py --loom_input expr_mat.loom --loom_filtered filtered.loom --thr_min_genes 200 --thr_min_cells 3 --thr_n_genes 5000 --thr_pct_mito 0.25
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File "/path/to/SCENICprotocol/bin/filtering-basic.py", line 5, in <module>
import scanpy as sc
File "/user/leuven/325/vsc32528/.local/lib/python3.7/site-packages/scanpy/__init__.py", line 3, in <module>
from ._utils import pkg_version, check_versions, annotate_doc_types
File "/user/leuven/325/vsc32528/.local/lib/python3.7/site-packages/scanpy/_utils.py", line 17, in <module>
from packaging import version
ModuleNotFoundError: No module named 'packaging'
Work dir:
/ddn1/vol1/staging/leuven/stg_00002/lcb/cflerin/testruns/scenic-protocol/contain_test/work/ce/731fb8cee41978b6352a534825a95d
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
This can be fixed by using the contain option:
singularity run --contain aertslab/pyscenic:0.9.18 [...]
or, in Nextflow:
singularity.runOptions = '--contain'
High variable genes
Hi,
All genes or high variable genes, which is better for pyscenic ?
Running example notebook is taking a long time
I'm trying to run the PBMC tutorial Jupyter notebook (PBMC10k_SCENIC-protocol-CLI.ipynb).
It's taking some time to run pyscenic ctx. Right now it's been two days and it's still running. I'm running it with an on-campus HPC service. I'm starting to think maybe there's something that I overlooked.
How long should it typically take to run pyscenic ctx for the PBMC example?
Thanks in advance!
Boxun
Error: buffer source array is read-only
Hi,
I'm trying to run pyscenic in jupyter-lab but got an error.
I created a docker container that pulled from aertslab/pyscenic:0.10.4 (also tried 'latest') and also installed jupyter-lab within it.
I was following this tutorial:
https://github.com/aertslab/SCENICprotocol/blob/master/notebooks/PBMC10k_SCENIC-protocol-CLI.ipynb
All the modules were loaded successfully and I was able to create a anndata object to contain the raw read count matrix. However I got an error when I ran:
!pyscenic grn {f_loom_path_scenic} {f_tfs} -o adj.csv --num_workers 20
The main error seems to be:
'infer_data failed for target BOK-AS1' Retry (1/10). Failure caused by ValueError("Regression for target gene BOK-AS1 failed. Cause ValueError('buffer source array is read-only').").
The same kind of error repeated for many other genes.
The output file "adj.csv" is empty.
Any help? Thanks.
nextflow pipeline error: Signatures dataframe is empty
Hi there! I tried the nextflow tool for running scenic. The command error happened in the following code :
2022-05-17 00:43:43,002 - pyscenic.cli.pyscenic - INFO - Loading gene signatures.
Traceback (most recent call last):
File "/opt/venv/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 470, in main
args.func(args)
File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 220, in aucell_command
signatures = load_signatures(args.signatures_fname.name)
File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/utils.py", line 175, in load_signatures
return df2regulons(load_motifs(fname, sep=suffixes_to_separator(extension)))
File "/opt/venv/lib/python3.7/site-packages/pyscenic/transform.py", line 308, in df2regulons
assert not df.empty, 'Signatures dataframe is empty!
AssertionError: Signatures dataframe is empty!`
I checked the config and thought it may be associated with the threshold in the config:
cistarget {
adj = 'adj.tsv'
type = ''
rank_threshold = 5000
auc_threshold = 0.05
nes_threshold = 3.0
min_orthologous_identity = 0.0
max_similarity_fdr = 0.001
annotations_fname = ''
thresholds = '0.75,0.90'
top_n_targets = 50
top_n_regulators = '5,10,50'
min_genes = 20
all_modules = false
motifsDb = 'pyscenic/mm10/mm10__refseq-r80__*feather'
motifsAnnotation = 'pyscenic/mm10/motifs-v9-nr.mgi-m0.001-o0.0.tbl'
}
which threshold should I change? Will it result in different regulons? Can anyone help me?
KeyError: 'Celltype_Garnett'
Hi,
I follow the pySCENIC protocol: PBMC10k downstream analyses.I get the following error when trying to extract relevant data from the integrated loom file:
cellAnnot = pd.concat(
[
pd.DataFrame( lf.ca.Celltype_Garnett, index=lf.ca.CellID ),
pd.DataFrame( lf.ca.ClusterID, index=lf.ca.CellID ),
pd.DataFrame( lf.ca.Louvain_clusters_Scanpy, index=lf.ca.CellID ),
pd.DataFrame( lf.ca.Percent_mito, index=lf.ca.CellID ),
pd.DataFrame( lf.ca.nGene, index=lf.ca.CellID ),
pd.DataFrame( lf.ca.nUMI, index=lf.ca.CellID ),
],
axis=1
)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/loompy/attribute_manager.py", line 115, in getattr
vals = self.dict["storage"][name]
KeyError: 'Celltype_Garnett'
Any suggestions?
Thanks!
where is the pbmc10k_garnett_results.txt file?
where is the pbmc10k_garnett_results.txt file? Thank you very much.
No module named 'sklearn.manifold.t_sne'
I'm running the following import:
from pyscenic.export import export2loom, add_scenic_metadata
And get the error:
>>> from pyscenic.export import export2loom, add_scenic_metadata
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/f6v/miniconda3/envs/scenic_protocol/lib/python3.6/site-packages/pyscenic/export.py", line 7, in <module>
from sklearn.manifold.t_sne import TSNE
ModuleNotFoundError: No module named 'sklearn.manifold.t_sne'
Seems like TSNE is just under sklearn.manifold. See docs.
sklearn version: '0.24.1'
Help with Analysis for Multiple samples, Counts normalization?
Hi, I have questions about how to carry out pySCENIC analysis correctly using data from more than a single sample. For example, I am working with data from 3 biological replicates which I intend to integrate using Harmony later on. My questions are:
- do the pyscenic steps of the protocol need to be carried out separately for each dataset or can the datasets be merged into a single one first and then that used as an input for the grn? If the latter is fine, can the counts used come from an already integrated dataset?
- do the counts need to be normalised? I see that this is something shown in many tutorials, however not in your protocol paper?
Thank you for your time.
can I use nextflow output reg.csv and adj.csv file?
how to apply SCENIC to another species(zebrafish) except for human,mouse and fly?
I've generated cistarget database for zebrafish. but I'm stopped at the initializeScenic step.
May I ask how to bypass the requirement in initializeScenic() that 'org' should be one of: mgi, hgnc, dmel?
The rss results are different between R and python
Hi
Tanks for your packages for TF analysis.
I have a problem here:
I did SCENIC analysis using pySCENIC, and finally I got the a loom file after pyscenic auc analysis.
I read the loom file in R and python to calculate rss, respectively. However, I got different results.
R:
code
sce_SCENIC <- open_loom("sample_SCENIC.loom") regulonAUC <- get_regulons_AUC(sce_SCENIC, column.attr.name='RegulonsAUC') rss <- calcRSS(AUC=getAUC(regulonAUC), cellAnnotation=cellTypes[colnames(regulonAUC), selectedResolution])
Python:
code:
lf = lp.connect( f_pyscenic_output, mode='r+', validate=False ) auc_mtx = pd.DataFrame( lf.ca.RegulonsAUC, index=lf.ca.CellID) lf.close() rss_cellType = regulon_specificity_scores( auc_mtx, sce_test.obs.celltype ) rss_cellType
The value in auc_mtx was same in R and python!
thanks so much
How to switch the revision of nextflow and jumpover the preprocess steps in nf?
Hi there!
I tried these brilliant work tools following the tutorial. Then I tried the mp_sparse_fixes revision. How can I switch the revision into the initial revision and jumpover the steps before scenic? so that I can use our own processed data for analysis. Many thanks in advance.
Generating final loom file
Hello,
while making the final loom file, I am getting this error:
Column attribute 'Embeddings_X' is not the same length (478) as number of columns in main matrix (239)
please help in this regard
What is required to run SCENIC in a custom species?
Hi - I am interested in running SCENIC on zebrafish and chick datasets, but am unclear as to which files I need to generate to run this in a custom species. Any help would be much appreciated - thanks
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.