Feature: sanity checks

Check all zero cells
Check all zero (or constant) features
...

Fix reported imputation metrics on readme for `GraphSCI` and `DeepImpute`

Use MSE
Compute on log1p normalized expression instead of the raw counts

Running scmogcn.py example - fatal error

Hello,

I am trying to reproduce the scmogcn.py example from the DANCE documentation. I run this script:

in the same folder where scmogcn.py is located. I'm using MSU's HPCC and a conda environment. After submitting the job, my slurm file gives this error: "dgl._ffi.base.DGLError: [18:00:11] /opt/dgl/src/random/random.cc:36: Check failed: e == CURAND_STATUS_SUCCESS: CURAND Error: CURAND_STATUS_INITIALIZATION_FAILED at /opt/dgl/src/random/random.cc:36"

Do you know what I might be doing wrong or how I can fix this error so that I can successfully reproduce this example?

Where can I find documenation?

I don't see any place with documentation for how I actually go about using the models in the package. Does any documentation exist?

Add random_state option to PCA transforms

It turns out that the random seed of the SVD solver impacts the PCA loadings (except for the top few) quite a bit, even when the tol option is set to 0. To enable reproducibility, set the random_state option.

Model object abstraction and refactoring

Model metadata, e.g., description, training info, params
Abstract methods (standardize in/out format)
- fit(*args, **kwargs) -> None
- predict(*args, **kwargs) -> Union[np.ndarray, torch.Tensor, other?]
- fit_predict shortcut function combining fit and predict
- score(*args, **kwargs) -> float (determine the type of metric via metric: Union[str, Callable], default setting saved in class attr. Abstract class per type of tasks?)
- __repr__ (show model metadata and info)
Model saving and loading (checkpointing)?

TODOs

single_modality/cell_type_annotation (#163, #164)
single_modality/clustering
single_modality/imputation
multi_modality/joint_embedding
multi_modality/match_modality
multi_modality/predict_modality
spatial/cell_type_deconvo
spatial/spatial_domain

Make utils for generating `dgl` or `pyg` graphs from processed data

Some current graph transforms methods directly save the processed dgl graphs in .uns. The limitations for doing so are

Limits the choice of the framework (dgl vs. pyg) to use in the downstream model.
.uns is not a good place to store large data

Solution

Create utils that generate dgl or pyg graphs given raw feature, edge data, or adjacency matrix.
Pass the raw data to method's fit function and construct the graph using the appropriate framework within the fit function.

Add option for `return_type` to take a list of values aligned with the specified channels

We need both the return types for feature_channels and label_channels..

init.py missing from dance/metadata

I work in an HPC environment where conda installation is not possible, pip installation results in a ModuleNotFoundError for dance.metadata. The directory appears to be missing its __init__.py file. Cloning the repo, adding the file and pip installing from the local clone worked for me.

error about running dataset(openproblems_bmmc_cite_phase2 ) with scMoGCN

After I installed dance, I run python scmogcn.py --subtask openproblems_bmmc_cite_phase2 --device cuda, but I found code have this error

, AddReverse is occurred on the 125 line of cell_feature_graph.py, It should be a function of dgl, but it was not imported

Make util for auto device selection

Pipeline bypass mechanism to replace `*PlaceHolder` transforms

pip install pydance

When running pip install pydance on a local environment running on an M1 macbook air, the following exception is raised. I believe the issue stems from "tables":

Random `segfault` when running pipeline tuning with `wandb` on certain machines

When running the tuning examples recently introduced in #398 (and #406), there appears to be some random chance of having segfault. The issue was later observed to be machine specific. I have only been getting this random segfault on MSU ICER HPCC (Python 3.8.16). Running the same example script on papermachine does not throw this segfault.

Looking at the core dump file (using pystack), it appears that the issue was related to Python's threading. More particularly when calling sklearn's randomized svd func (maybe some other similar packages as well). See detail core dump log below.

(dance) bash-4.2$ pystack core core.113134
Using executable found in the core file: /mnt/home/liurenmi/software/anaconda3/envs/dance/bin/python

Core file information:
state: D zombie: True niceness: 0
pid: 113134 ppid: 112816 sid: 112816
uid: 790872 gid: 2362 pgrp: 113134
executable: python arguments: python main.py

The process died due a segmentation fault accessing address: 0xffffffffffffff70
Traceback for thread 114928 [] (most recent call last):
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
        self.run()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 300, in check_internal_messages
        self._loop_check_status(
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 251, in _loop_check_status
        join_requested = self._join_event.wait(timeout=wait_time)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 558, in wait
        signaled = self._cond.wait(timeout)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 306, in wait
        gotit = waiter.acquire(True, timeout)

Traceback for thread 114927 [] (most recent call last):
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
        self.run()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 268, in check_network_status
        self._loop_check_status(
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 251, in _loop_check_status
        join_requested = self._join_event.wait(timeout=wait_time)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 558, in wait
        signaled = self._cond.wait(timeout)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 306, in wait
        gotit = waiter.acquire(True, timeout)

Traceback for thread 114926 [] (most recent call last):
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
        self.run()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 286, in check_stop_status
        self._loop_check_status(
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 251, in _loop_check_status
        join_requested = self._join_event.wait(timeout=wait_time)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 558, in wait
        signaled = self._cond.wait(timeout)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 306, in wait
        gotit = waiter.acquire(True, timeout)

Traceback for thread 114874 [] (most recent call last):
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
        self.run()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/interface/router.py", line 70, in message_loop
        msg = self._read_message()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/interface/router_sock.py", line 27, in _read_message
        resp = self._sock_client.read_server_response(timeout=1)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 285, in read_server_response
        data = self._read_packet_bytes(timeout=timeout)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 255, in _read_packet_bytes
        data = self._sock.recv(self._bufsize)

Traceback for thread 114845 [Has the GIL] (most recent call last):
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
        self.run()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 298, in _run_job
        self._function()
    (Python) File "main.py", line 83, in evaluate_pipeline
        preprocessing_pipeline(data)
    (Python) File "/mnt/ufs18/home-026/liurenmi/repo/dance/dance/pipeline.py", line 238, in __call__
        func(*args, **kwargs)
    (Python) File "/mnt/ufs18/home-026/liurenmi/repo/dance/dance/transforms/cell_feature.py", line 56, in __call__
        gene_feat = gene_pca.fit_transform(feat.T)  # decompose into gene features
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
        data_to_wrap = f(self, X, *args, **kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/base.py", line 1152, in wrapper
        return fit_method(estimator, *args, **kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/decomposition/_pca.py", line 460, in fit_transform
        U, S, Vt = self._fit(X)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/decomposition/_pca.py", line 512, in _fit
        return self._fit_truncated(X, n_components, self._fit_svd_solver)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/decomposition/_pca.py", line 616, in _fit_truncated
        U, S, Vt = randomized_svd(
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/utils/extmath.py", line 449, in randomized_svd
        Q = randomized_range_finder(
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/utils/extmath.py", line 277, in randomized_range_finder
        Q, _ = linalg.lu(safe_sparse_dot(A, Q), permute_l=True)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/scipy/linalg/_decomp_lu.py", line 220, in lu
        p, l, u, info = flu(a1, permute_l=permute_l, overwrite_a=overwrite_a)

Traceback for thread 114844 [] (most recent call last):
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
        self.run()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 178, in _heartbeat
        time.sleep(5)

Traceback for thread 113134 [] (most recent call last):
    (Python) File "main.py", line 108, in <module>
        wandb.agent(sweep_id, function=evaluate_pipeline, count=3)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/wandb_agent.py", line 581, in agent
        return pyagent(sweep_id, function, entity, project, count)
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 348, in pyagent
        agent.run()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 326, in run
        self._run_jobs_from_queue()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 220, in _run_jobs_from_queue
        thread.join()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 1011, in join
        self._wait_for_tstate_lock()
    (Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
        elif lock.acquire(block, timeout):

More sysinfo below.

Machine that failed:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Machine that did not fail:

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Skipping for now but might come back later to fix this issue if it appears to be happening to more users other than myself.

Improve train/val/test data retrieval mech (instead of getting them individually?)

Make `Compose` transformation object and add `transform` option to `Data`

Add more info about key error from `get_feature`, e.g., what channel type

Fix minor inconsistency at param init for CARD

0d26d29#r105213164

Add support for containerizing using, e.g., Singularity and Docker

Some users pointed out that fully installing the dance package in a conda environment may take up a lot of space (~12GB), and suggested using solutions like Singularity to containerize the package to take advantage of the scratch space on HPC. I think @JiayuanDing100 might have some experience with containerizing/virtualizing. Could you suggest some ways to approach this?

Unify kwarg: `seed` -> `random_state`

Exclude benchmark runs from default pytest runs, make `lite-bench` and `full-bench` test options (tox)

Update contribution guidelines with notes about bench tests vs. unit tests

Convention for tensor device

Problem

Currently, there is no convention for which device the data is stored when a function, e.g., fit(), is called. For example, even though the computation device to use is cuda, the input graph may not be on the GPU and will need to be transferred later after subsampling the neighborhoods for mini-batch training. This inconsistency causes many issues for development.

One naive solution is to call tensor.to(device) using the correct device every time a computation is being performed, which is certainly unsatisfactory and makes the code base not as clean.

Solution

By default, all data should sit on cpu
When a compute function, e.g., fit(), is called, perform any necessary device conversion inside that function.
The only exceptions are predict() and score(), which can be more flexible, since they will be used in various places with various configurations, e.g., training on GPU with mini-batch training and evaluating on CPU with full-batch.

The following PR is a corresponding example of fixing the issue: #30

Need to check

grep "\.to(" -r dance | awk -F":" '{print $1}' | sort -u

Release date

This work is quite interesting. Do you have a plan for the code release date?

Errors with Installation

dance/install.sh

Line 58 in 1d94be9

    
           pip install torch==${PYTORCH_VERSION} torchvision=${TORCHVISION_VERSION} ${PYTORCH_CHANNEL}

~~torchvision= should be replaced with torchvision==, otherwise 'install.sh' raises error~~
Fixed by latest PR

Tutorial notebook is outdated

The official tutorial is not compatible with v1.0 release. Please refer to the issue here.

Config support

Use omegaconf DictConfig object for storing and managing configurations. Update preprocessing pipelines construction with config parsing.

Data selection
Preprocessing pipelines
Model params and pipelines
Eval pipelines
(Additional) Results report generation

Fix ``scmm`` error due to zero total counts calling ``torch.lgamma`` (called by ``torch.distributions.NegativeBinomial``)

libcusparse.so.11

I followed the install.sh file to install dance in a conda env. Now I am following the tutorial. I am getting this error
"OSError: libcusparse.so.11: cannot open shared object file: No such file or directory"
when running
"from dance.datasets.singlemodality import ClusteringDataset"
I don't know where the issue is coming from. I appreciate any suggestions.

Unifying base data object

Currently, there are several different dataset objects specialized for each task and model (e.g., CellTypeDataset, ClusteringDataset), each of them takes a variety of specialized arguments that are not directly related to the underlying data, e.g., save path, processing scheme, choice of tissue. This complexity makes it quite hard to maintain the code base and implement new methods/datasets.

To improve this situation, we need to isolate raw dataset objects from transformation/processing methods.

Base data object
- Take AnnData as an input and save it as a private attribute (read-only?).
- Construct data loaders that load g, x, y, etc., to be passed to the model for training/evaluation.
Dataset object
- Download option
- Transformation option
- Dataset from paper (preprocessed) -> used to benchmark the reproducibility of the reimplemented model
Transformation
- Leverage functionalities from scanpy (recall that now the base data object store an AnnData object as a (private) attribute

To fix

Single modality

Spatial

examples/spatial/spatial_domain/stagate.py (#127)
examples/spatial/spatial_domain/louvain.py (#124)
examples/spatial/spatial_domain/stlearn.py (#126)
examples/spatial/spatial_domain/spagcn.py (#83)
examples/spatial/cell_type_deconvo/spotlight.py (#107)
examples/spatial/cell_type_deconvo/dstg.py (#103)
examples/spatial/cell_type_deconvo/card.py (#93)
examples/spatial/cell_type_deconvo/spatialdecon.py (#94)

Multi modality

GraphSCI implementation

For the imputation model GraphSCI, I think the AE part is wrong, the size_factor is cell_specific, so it should have the same number as cells, but in your implementation the size_factor number is gene-specific. For the inputs of AE, it should be the transpose of the input of GNN, so the input's shape should be cell*gene.

Add option to `.score` method to take `index` option, which indicates which samples we want to calculate the scores

index option
mask option

Some questions about general wrapper for datasets

Hi, I intend to apply this model to different datasets rather than the competition datasets, and I wonder if you have any general loading data structure to load public datasets or not. Moreover, is it possible for me to use a lighter structure comparing the jointembedding structure if I have already processed the given dataset? Thanks.

Use the `download_unzip` utility in `ImputationDataset`

          - [ ] TODO later: use the `download_unzip` utility instead of doing these system calls.

Originally posted by @RemyLau in #223 (comment)

Refactor multimodal dataset object to use the dance abstract dataset class

          TODO in a future PR: refactor dataset object to use the dance abstract dataset class, see [`ClusteringDataset`](https://github.com/OmicsML/dance/blob/f3e0e83ef3dbf25339e1a6ffa10ffb18ac81d72d/dance/datasets/singlemodality.py#L235) for example

Originally posted by @RemyLau in #228 (comment)

Questions about conda installing

Hi, I have a question about conda installing process:

I think we should use

conda activate dance

rather than

conda activate dance-env

Is it correct? Thanks.

Add community guidelines

Code of conduct (#112)
Contributing/development guidelines (pre-commit, auto tests, etc.) (#121)

Create tutorial notebook for wrapping ``AnnData`` (or ``MuData``) objects into dance ``Data`` object

DSTG link graph add connections between real spots?

According to the link graph construction description from the paper, the graph should also contain interaction between real-spots.

However, in the current implementation (modified from the original implementation, see Su-informatics-lab/DSTG#16), only the cross interactions between pseudo-spots and real-spots are used.

dance/dance/transforms/graph/dstg_graph.py

Line 59 in 9966c47

graph = construct_link_graph(pseudo_st_df, real_st_df, k_filter, num_cc)

Should we add the interactions between real-spots as described in the manuscript in the DANCE implementation? If so, it could be achieved quite easily by simply calling construct_link_graph one more time by passing real_st_df as both first two args and combine the edge list with the old graph.

Caution: be aware that one might need to modify the the new graph node indexes (or pass real_st_df and another copy of real_st_df with offsetted index)

ImportError when directly running the code in the terminal

(base) [zhan2210@gateway-03 ~]$ conda activate dance-env
(dance-env) [zhan2210@gateway-03 ~]$ export LD_LIBRARY_PATH=/mnt/home/zhan2210/
ENTER/lib: $LD_LIBRARY_PATH
(dance-env) [zhan2210@gateway-03 ~]$ python
Python 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:53:32) [GCC 12
.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dance.datasets.multimodality import Joint EmbeddingNIPSDataset
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/ufs18/home-249/zhan2210/dance/dance/datasets/__init__.py", line 1,
in <module>
from dance.datasets.multimodality import JointEmbeddingNIPSDataset, Modalit
yMatchingDataset, ModalityPrediction Dataset
File "/mnt/ufs18/home-249/zhan2210/dance/dance/datasets/multimodality.py", li
ne 12, in <module>
from dance import logger
ImportError: cannot import name 'logger' from 'dance' (unknown location)
>>>

Replace model specific score functions with generic ones?

Currently, each model comes with a .score function for evaluating the model performance. Most of them do the same thing, e.g., compute model accuracy for cell-type annotation tasks. Why not just have one generic function that calculates the accuracy (and similarly for other tasks)?

Dataset dowmload

I can't download your dataset because of VPN restrictions and can't access the content under 'https://www.dropbox.com/'.

omicsml / dance Goto Github PK

dance's People

Contributors

Stargazers

Watchers

Forkers

dance's Issues

TODOs

Solution

Problem

Solution

Need to check

To fix

Single modality

Spatial

Multi modality

Recommend Projects

Recommend Topics

Recommend Org