jf-lab/cnmfe-reviewer

CNMF-E Reviewer

Source code for the paper:

Automated curation of CNMF-E-extracted ROI spatial footprints and calcium traces using open-source AutoML tools

Tran LM, Mocle AJ, Ramsaran AI, Jacob AD, Frankland PW, Josselyn SA. (2020) In preparation.

Getting Started

The tutorial jupyter notebook in notebooks/ has information about the tool, and how to how to use it for your own data after you've installed everything. You can make a copy of the notebook and the config.py file and configure the project for your own data.

Download example dataset

Most of necessary files are included in the repo, however the processed spatial footprints and traces in the ground truth dataset which can be used in the tutorial can be downloaded from here (due to GitHub file size limits). Place them in the cnmfe-reviewer/data folder.

Software requirements

Note: The main AutoML tool used, Autosklearn, works best in Linux environments.

1. Python 3.5+ (ideally with Anaconda)
2. Clone or download this repository onto a local folder on your computer.
3. Change to the directory where you downloaded cnmfe-reviewer. cd /path/to/cnfme-reviewer
4. Create a dedicated conda environment for use of this package (highly recommended to prevent versioning issues with packages you already have installed). Instructions below, but for more detailed instructions, click here for the conda documentation.

# replace myenv_name with your environment name (e.g. cnmfereview)
conda create -n myenv_name python=3.6

# when conda asks to proceed, type 'y'
# this puts your environment in a default folder under the hood

# activate the environment
source activate myenv_name

5. While in the conda environment, follow the instructions to install AutoSklearn. Namely: Make sure you have these installed:

• C++ compiler (with C++11 supports) (get gcc here)
• SWIG (version 3.0 or later) (get SWIG here)

# install dependancies
curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | xargs -n 1 -L 1 pip install

# (ON LINUX ONLY, skip on MacOS) get the correct compilers and swig
conda install gxx_linux-64 gcc_linux-64 swig

# install auto-sklearn
pip install 'auto-sklearn==0.6.0'

6. Finally, install cnfmereview:

# install cnmfereview (you should be in the cnmfe-reviewer directory)
pip install ./

7. To make the environment available in Jupyter Notebooks:

conda install ipykernel
ipython kernel install --user --name=<any_name_for_kernel>

8. When you're done, you can deactivate the environment:

conda deactivate

cnmfe-reviewer's Issues

auto-sklearn version issue

Have to install the auto-sklearn manually but got an error: “cnmfereview 0.1 has requirement auto-sklearn>0.6.0, but you'll have auto-sklearn 0.6.0 which is incompatible.”
I could not find an newer version than 0.6.0.

create config tempate file for users to populate to suit their experimental specifications

To unify a place to make experimenter specific changes. Should include:

• trace length to use to generate dataset from calcium traces (should be smaller than the max, but larger than a 3min equivalent to allow
• spatial and temporal downsample factors
• average footprint image size (heightxwidth in pixels)
  • depends on the type of miniature endoscope used, field of view and what brain region is being imaged, the key is to make sure the size will not cut off any pixels of the largest footprint in the dataset)

Deal with large data files hosting (using something other than git-lfs)

Files larger than 100MB are blocked from upload, which includes all the pickled .npy datset files.
Get around this using git-lfs, but then users would need to install and use git-lfs to retrieve the data and not just download the pointers

Is scores.npy broken?

I can't load the scores.npy in this repo. Is this a broken file?

codes (Tutorial.ipynb)

import cnmfereview as cr
import config as cfg

data = cr.Dataset(
    data_paths=cfg.data_paths,
    exp_id=cfg.exp_id,
    img_shape=cfg.img_shape,
    img_crop_size=cfg.img_crop_size,
    max_trace=cfg.max_trace_len,
    )

outputs

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-fee02d6f19fd> in <module>
      4     img_shape=cfg.img_shape,
      5     img_crop_size=cfg.img_crop_size,
----> 6     max_trace=cfg.max_trace_len,
      7     )

~/Documents/research/cnmfe-reviewer/cnmfereview/utils.py in __init__(self, data_paths, exp_id, img_shape, img_crop_size, max_trace, preprocess)
    201                   exists and has been loaded instead.")
    202 
--> 203         self.targets = np.load(self.DATADIR / data_paths['targets']
    204                                ).astype(self.dtype)
    205 

~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    455             # Try a pickle
    456             if not allow_pickle:
--> 457                 raise ValueError("Cannot load file containing pickled data "
    458                                  "when allow_pickle=False")
    459             try:

ValueError: Cannot load file containing pickled data when allow_pickle=False

I also tried allow_pickle=True option for line 203, 204 in utils.py like below then another error occurred.
codes (utils.py)

        self.targets = np.load(self.DATADIR / data_paths['targets'],
                               allow_pickle=True).astype(self.dtype)

outputs

---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    459             try:
--> 460                 return pickle.load(fid, **pickle_kwargs)
    461             except Exception:

UnpicklingError: invalid load key, 'v'.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-3-fee02d6f19fd> in <module>
      4     img_shape=cfg.img_shape,
      5     img_crop_size=cfg.img_crop_size,
----> 6     max_trace=cfg.max_trace_len,
      7     )

~/Documents/research/cnmfe-reviewer/cnmfereview/utils.py in __init__(self, data_paths, exp_id, img_shape, img_crop_size, max_trace, preprocess)
    202 
    203         self.targets = np.load(self.DATADIR / data_paths['targets'],
--> 204                                allow_pickle=True).astype(self.dtype)
    205 
    206         assert (len(self.targets) == len(self.spatial) and

~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    461             except Exception:
    462                 raise IOError(
--> 463                     "Failed to interpret file %s as a pickle" % repr(file))
    464     finally:
    465         if own_fid:

OSError: Failed to interpret file PosixPath('../data/scores.npy') as a pickle