Giter Site home page Giter Site logo

mountainsort5's Introduction

MountainSort 5

latest-release tests codecov

This is the most recent version of the MountainSort spike sorting algorithm. An implementation of the previous version of this algorithm can be found here.

  • Uses Isosplit clustering
  • Runs faster than previous versions, especially for large channel counts
  • Better handles time-overlapping events and drifting waveforms
  • Runs fast on CPU
  • Uses SpikeInterface for I/O and preprocessing
  • Supports multiple sorting schemes, each suited for different experimental setups

image

Installation

pip install --upgrade mountainsort5

Dependencies:

Python, SpikeInterface, scikit-learn, isosplit6

Usage

MountainSort5 utilizes SpikeInterface recording and sorting objects. See the SpikeInterface documentation to learn how you can load and preprocess your ephys data.

Once you have loaded a SpikeInterface recording object, you can run MountainSort5 using the following code:

import spikeinterface as si
import spikeinterface.preprocessing as spre
import mountainsort5 as ms5

recording = ... # load your recording using SpikeInterface

# Make sure the recording is preprocessed appropriately
# lazy preprocessing
recording_filtered = spre.bandpass_filter(recording, freq_min=300, freq_max=6000)
recording_preprocessed: si.BaseRecording = spre.whiten(recording_filtered, dtype='float32')

# use scheme 1
sorting = ms5.sorting_scheme1(
    recording=recording_preprocessed,
    sorting_parameters=ms5.Scheme1SortingParameters(...)
)

# or use scheme 2
sorting = ms5.sorting_scheme2(
    recording=recording_preprocessed,
    sorting_parameters=ms5.Scheme2SortingParameters(...)
)

# or use scheme 3
sorting = ms5.sorting_scheme3(
    recording=recording_preprocessed,
    sorting_parameters=ms5.Scheme3SortingParameters(...)
)

# Now you have a sorting object that you can save to disk or use for further analysis

To give it a try with simulated data, run the following scripts in the examples directory:

Scheme 1: examples/scheme1/toy_example.py

Scheme 2: examples/scheme2/toy_example.py

Scheme 3: examples/scheme3/toy_example.py

Preprocessing

MountainSort5 is designed to operate on preprocessed data. You should bandpass filter and whiten the recording as shown in the examples. SpikeInterface provides a variety of lazy preprocessing tools so that intermediate files do not need to be stored to disk.

Sorting schemes

MountainSort5 is organized into three sorting schemes. Different schemes are appropriate for different experimental setups.

Sorting scheme 1

This is the simplest sorting scheme and is useful for quick tests. The entire recording is loaded into memory, and clustering is performed in a single pass. In general, scheme 1 should only be used for testing and debugging as scheme 2 does a better job handling events that overlap in time, and works with larger datasets on limited RAM systems. Scheme 1 is used as a first pass in scheme 2, so reading about the parameters of scheme 1 will help you understand the other schemes better.

Read more about scheme 1

Sorting scheme 2

The second sorting scheme is generally preferred over scheme 1 because it can handle larger datasets that cannot be fully loaded into memory, and also has other advantages in terms of accurately detecting and labeling spikes.

In phase 1, the first scheme is used as a training step, performing unsupervised clustering on a subset of the dataset. Then in phase 2, a set of classifiers are trained based on the labels of the training step. The classifiers are then used to label the spikes in the entire recording.

Read more about scheme 2

Sorting scheme 3

Sorting scheme 3 is designed to handle long recordings that may involve waveform drift. The recording is divided into blocks, and each block is spike sorted using scheme 2. Then the snippet classifiers for the blocks are used to associate matching units between blocks.

Read more about scheme 3

Citing MountainSort

For now, please cite the original MountainSort paper:

@article{chung2017fully,
  title={A fully automated approach to spike sorting},
  author={Chung, Jason E and Magland, Jeremy F and Barnett, Alex H and Tolosa, Vanessa M and Tooker, Angela C and Lee, Kye Y and Shah, Kedar G and Felix, Sarah H and Frank, Loren M and Greengard, Leslie F},
  journal={Neuron},
  volume={95},
  number={6},
  pages={1381--1394},
  year={2017},
  publisher={Elsevier}
}

In addition, if you use the SpikeInterface framework, please cite the following paper:

@article{buccino2020spikeinterface,
  title={SpikeInterface, a unified framework for spike sorting},
  author={Buccino, Alessio Paolo and Hurwitz, Cole Lincoln and Garcia, Samuel and Magland, Jeremy and Siegle, Joshua H and Hurwitz, Roger and Hennig, Matthias H},
  journal={Elife},
  volume={9},
  pages={e61834},
  year={2020},
  publisher={eLife Sciences Publications Limited}
}

Contributing

Feel free to open an issue or pull request if you have any questions or suggestions.

Please star this repository if you find it useful!

Authors

Jeremy Magland, Center for Computational Mathematics, Flatiron Institute

Acknowledgements

Thank you to Loren Frank and members of his lab for their support of this project at all stages of development.

Thank you to Alex Barnett, Leslie Greengard, and Jason Chung for their work on the original Isosplit and MountainSort algorithms.

Thank you to the SpikeInterface team, especially Alessio Buccino and Samuel Garcia, for their work on the SpikeInterface framework, which supports pre- and post-processing and makes it easy to use MountainSort5 with a variety of file formats.

Thank you to Jeff Soules for his work on sortingview and related visualization tools that make it possible to inspect the results of MountainSort5 and other algorithms.

Finally, thank you to all the users of the previous version of MountainSort who have provided feedback and suggestions.

License

Apache-2.0

mountainsort5's People

Contributors

magland avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

mountainsort5's Issues

error: could not build wheels for isosplit6

Hello! I keep running into a specific issue while installing mountainsort5, which is specifically due to issues installing isosplit6. I'm running this on a Mac:

pip install --upgrade isosplit6

Collecting isosplit6
Using cached isosplit6-0.1.2.tar.gz (25 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: isosplit6
Building wheel for isosplit6 (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for isosplit6 (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [72 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.9-x86_64-cpython-38
creating build/lib.macosx-10.9-x86_64-cpython-38/isosplit6
copying isosplit6/init.py -> build/lib.macosx-10.9-x86_64-cpython-38/isosplit6
running build_ext
building 'isosplit6_cpp' extension
creating build/temp.macosx-10.9-x86_64-cpython-38
creating build/temp.macosx-10.9-x86_64-cpython-38/src
/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/haelabdulrazeq/anaconda3/envs/main/include -arch x86_64 -I/Users/haelabdulrazeq/anaconda3/envs/main/include -arch x86_64 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/private/var/folders/kz/rjjrnft52xvfg1655351jmr00000gn/T/pip-build-env-dl4_3vl4/overlay/lib/python3.8/site-packages/pybind11/include -I/Users/haelabdulrazeq/anaconda3/envs/main/include/python3.8 -c src/isocut5.cpp -o build/temp.macosx-10.9-x86_64-cpython-38/src/isocut5.o -fvisibility=hidden -g0 -stdlib=libc++
In file included from src/isocut5.cpp:16:
src/isocut5.h:25:25: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
bool already_sorted = false;
^
1 warning generated.
/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/haelabdulrazeq/anaconda3/envs/main/include -arch x86_64 -I/Users/haelabdulrazeq/anaconda3/envs/main/include -arch x86_64 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/private/var/folders/kz/rjjrnft52xvfg1655351jmr00000gn/T/pip-build-env-dl4_3vl4/overlay/lib/python3.8/site-packages/pybind11/include -I/Users/haelabdulrazeq/anaconda3/envs/main/include/python3.8 -c src/isocut6.cpp -o build/temp.macosx-10.9-x86_64-cpython-38/src/isocut6.o -fvisibility=hidden -g0 -stdlib=libc++
In file included from src/isocut6.cpp:16:
src/isocut5.h:25:25: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
bool already_sorted = false;
^
In file included from src/isocut6.cpp:17:
src/isocut6.h:25:25: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
bool already_sorted = false;
^
2 warnings generated.
/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/haelabdulrazeq/anaconda3/envs/main/include -arch x86_64 -I/Users/haelabdulrazeq/anaconda3/envs/main/include -arch x86_64 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/private/var/folders/kz/rjjrnft52xvfg1655351jmr00000gn/T/pip-build-env-dl4_3vl4/overlay/lib/python3.8/site-packages/pybind11/include -I/Users/haelabdulrazeq/anaconda3/envs/main/include/python3.8 -c src/isosplit5.cpp -o build/temp.macosx-10.9-x86_64-cpython-38/src/isosplit5.o -fvisibility=hidden -g0 -stdlib=libc++
In file included from src/isosplit5.cpp:16:
In file included from src/isosplit5.h:20:
src/isocut5.h:25:25: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
bool already_sorted = false;
^
In file included from src/isosplit5.cpp:16:
src/isosplit5.h:25:29: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
double isocut_threshold = 1.0;
^
src/isosplit5.h:26:26: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
int min_cluster_size = 10;
^
src/isosplit5.h:27:16: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
int K_init = 200;
^
src/isosplit5.h:28:26: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
bool refine_clusters = false;
^
src/isosplit5.h:29:33: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
int max_iterations_per_pass = 500;
^
src/isosplit5.h:36:27: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
bigint num_iterations = 0;
^
src/isosplit5.h:58:25: warning: default member initializer for non-static data member is a C++11 extension [-Wc++11-extensions]
bool final_reassign = false; //not yet implemented
^
src/isosplit5.cpp:228:154: warning: format specifies type 'long' but the argument has type 'bigint' (aka 'long long') [-Wformat]
printf("Warning in isosplit5: new parcel has no points -- perhaps dataset contains duplicate points? -- original size = %ld.\n", sz);
~~~ ^~
%lld
src/isosplit5.cpp:153:14: warning: variable 'something_changed' set but not used [-Wunused-but-set-variable]
bool something_changed=false;
^
src/isosplit5.cpp:870:35: error: a space is required between consecutive right angle brackets (use '> >')
std::vector<std::vector> dists(K);
^~
> >
src/isosplit5.cpp:1163:35: error: a space is required between consecutive right angle brackets (use '> >')
std::vector<std::vector> distances(N);
^~
> >
10 warnings and 2 errors generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for isosplit6
Failed to build isosplit6
ERROR: Could not build wheels for isosplit6, which is required to install pyproject.toml-based projects

How to limit number of cores to use?

I am using this repo on a strong computer with 64 cores, which ends up going really slowly because they all have to wait for each other, leading to >12 sorting times for a 45 minute NPIX recording. Is there a way to limit the number of cores used?

parallelize detection and classification

@magland I am finding mountainsort5 to be excellent but a bit slow for long (2+ hour recordings) with many channels. Is there a reason why the detection and classification in scheme 2 cannot be threaded or parallelized with multiprocessing? If its simply the case that nobody has had the time to implement this, I will take a stab - but I just was wondering if there was something I was missing that prevented this from parallelization.

Parameters list for compatibility for spikeinterface

Hi, is there a list of spike sorting parameters in the code somewhere that I can directly reference in my spikeinterface jobs? Reading the documentation, the parameters are organised into sorting schemes, but I would like to know if I can extract the actual parameters corresponding to these schemes. Right now, I only have two parameters implemented (filter, whiten) in my json file which seems to work fine:

    "sorter_params":
    {
        "mountainsort5":
        {
            "filter": false,
            "whiten": false
        }
    },

However, I can't easily find the full list of adjustable parameters within the source code.

For instance, I'd like to know if there is a parameter to handle multiple cores (in ms4 I believe it was num_workers, but I can't find something similar in ms5).

Otherwise, ms5 seems to be much faster than ms4, which is very exciting.
Many thanks for your time.

threshold

Hi Dr.Margland,
I noticed that you suggested 5.5sd for ms5 for spike detection, do you suggest changing the threshold of the previous version ms4 to 5.5 as well or this behavior is specific to ms5? Separately do you kindly have any insight on how might the recording duration change this parameter? Thanks.

channel locations for spike sorting

I'm wanting to use ms5 for spike sorting but I'm running into the following error with data collected using openephys.

Exception                                 Traceback (most recent call last)
Cell In[8], line 8
      4 # need to sort the recording to continue!
      5 # 1. Build singularity continaer for matlab
      6 # 2. Run Sorter
      7 folder = "sorter_output"
----> 8 sorting = ms5.sorting_scheme1(
      9     recording["clean_data"],
     10     sorting_parameters = ms5.Scheme1SortingParameters(detect_channel_radius=0,
     11                                                      snippet_mask_radius = 0))

File ~/.conda/envs/spike_sort/lib/python3.10/site-packages/mountainsort5/schemes/sorting_scheme1.py:41, in sorting_scheme1(recording, sorting_parameters)
     38 N = recording.get_num_frames()
     39 sampling_frequency = recording.sampling_frequency
---> 41 channel_locations = recording.get_channel_locations()
     43 print(f'Number of channels: {M}')
     44 print(f'Number of timepoints: {N}')

File ~/.conda/envs/spike_sort/lib/python3.10/site-packages/spikeinterface/core/baserecordingsnippets.py:305, in BaseRecordingSnippets.get_channel_locations(self, channel_ids, axes)
    303 locations = self.get_property('location')
    304 if locations is None:
--> 305     raise Exception('There are no channel locations')
    306 locations = np.asarray(locations)[channel_indices]
    307 return select_axes(locations, axes)

Exception: There are no channel locations

I was looking at early documentation and I thought setting radius to 0 would let me move past this, but it doesn't. Is there anything I can do to get this information or by-pass this requirement?

Python version support

Hi, do you know which versions of Python mountainsort5 supports? I'm running into compilation errors when installing mountainsort5 on a virtual environment on a cluster computing account using Python 3.8.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.