DeepLabStream

DeepLabStream is a python based multi-purpose tool that enables the realtime tracking and manipulation of animals during ongoing experiments. Our toolbox was orginally adapted from the previously published DeepLabCut (Mathis et al., 2018) and expanded on its core capabilities, but is now able to utilize a variety of different network architectures for online pose estimation (SLEAP, DLC-Live, DeepPosekit's StackedDenseNet, StackedHourGlass and LEAP).

DeepLabStreams core feature is the utilization of real-time tracking to orchestrate closed-loop experiments. This can be achieved using any type of camera-based video stream (incl. multiple streams). It enables running experimental protocols that are dependent on a constant stream of bodypart positions and feedback activation of several input/output devices. It's capabilities range from simple region of interest (ROI) based triggers to headdirection or behavior dependent stimulation, including online classification (SiMBA, B-SOID).

Read the news:

Watch an introductory talk about DLStream on Youtube
Real-time behavioral analysis using artificial intelligence @ Federal Ministry of Education and Research, Uni Bonn and Phys.org
We are featured on OpenBehavior
We are also featured on open-neuroscience.com!

New features:

03/2021: Online Behavior Classification using SiMBA and B-SOID:

full integration of online classification of user-defined behavior using SiMBA and B-SOID.
SOCIAL CLASSIFICATION with SiMBA 14bp two animal classification (more to come!)
Unsupervised Classification with B-SOID
New wiki guide and example experiment to get started with online classification: Advanced Behavior Classification
this version has new requirements (numba, pure, scikit-learn), so be sure to install them (e.g. pip install -r requirements.txt).

02/2021: Multiple Animal Experiments (Pre-release): Full SLEAP integration (Full release coming soon!)

Updated Installation (for SLEAP support)
Single Instance and Multiple Instance models (TopDown & BottomUp) integration
New Multiple Animal Experiment Tutorial

01/2021: DLStream was published in Communications Biology

12/2021: New pose estimation model integration

(DLC-Live) and pre-release of further integration (DeepPosekit's StackedDenseNet, StackedHourGlass and LEAP)

Quick Reference:

Check out or wiki: DLStream Wiki

Read the paper: Schweihoff, et al. 2021

Contributing

If you have feature requests or questions regarding the design of experiments join our slack group.

We are constantly working to update and increase the capabilities of DLStream. We welcome all feedback and input from your side.

1. Updated Installation & Testing

2. How to use DLStream GUI

3. Check out our Out-of-the-Box

4. Design an Out-of-the-Box Experiment

What's underneath?:

5. Introduction to experiments

For advanced users:

6. Design your first experiment

7. Adapting an existing experiment to your own needs

How to use DeepLabStream

Just run

cd DeepLabStream
python app.py

You will see the main control panel of a GUI app.

To start working with DeepLabStream, press the Start Stream button. It will activate the camera manager and show you the current view from the connected cameras.

After that you can Start Analysis to start DeepLabCut and receive a pose estimations for each frame, or, additionally, you can Start Recording to record a video of the current feed (visible in the stream window). You will see your current video timestamp (counted in frames) and FPS after you pressed the Start Analysis button.

As you can see, we track three points that represent three body parts of the mouse - nose, neck and tail root. Every single frame where the animal was tracked is outputted to the dataframe, which would create a .csv file after the analysis is finished.

After you finish with tracking and/or recording the video, you can stop either function by specifically pressing on corresponding "stop" button (so, Stop Analysis or Stop Recording) or you can stop the app and refresh all the timing at once, by pressing Stop Streaming button.

Experiments

DeepLabStream was build specifically for closed-loop experiments, so with a properly implemented experiment protocol, running experiments on this system is as easy as pressing the Start Experiment button. Depending on your protocol and experimental goals, experiments could run and finish without any further engagement from the user.

In the provided ExampleExperiment two regions of interest (ROIs) are created inside an arena. The experiment is designed to count the number of times the mouse enters a ROI and trigger a corresponding visual stimulus on a screen. The high contrast stimuli (image files) are located within the experiments/src folder and specified within the experiments.py ExampleExperiments Class.

As a visual representation of this event, the border of the ROI will turn green.

All experimental output will be stored to a .csv file for easy postprocessing. Check out Working with DLStream output for further details.

Look at the Introduction to experiments to get an idea how to design your own experiment in DeepLabStream or learn how to adapt one of the already published experiments at Adapting an existing experiment.

How does this work

DeepLabStream uses the camera's video stream to simultaneously record a raw (read as unmodified) video of the ongoing experiment, send frames one-by-one to the neuronal network for analysis, and use returned analysed data to plot and show a video stream for the experimenter to observe and control the experiment. Analysed data will also be utilized to enable closed-loop experiments without any human interference, using triggers to operate equipment on predefined conditions and to end, prolong or modify parts of experimental protocol.

Known issues

If you encounter any issues or errors, you can check out the wiki article (Help there is an error!). If your issue is not listed yet, please refer to the issues and either submit a new issue or find a reported issue (which might be already solved) there. Thank you!

References:

If you use this code or data please cite:

Schweihoff, J.F., Loshakov, M., Pavlova, I. et al. DeepLabStream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection.

Commun Biol 4, 130 (2021). https://doi.org/10.1038/s42003-021-01654-9

License

This project is licensed under the GNU General Public License v3.0. Note that the software is provided "as is", without warranty of any kind, expressed or implied.

Authors

Developed by:

Jens Schweihoff, [email protected]
Matvey Loshakov, [email protected]

Corresponding Author: Martin Schwarz, [email protected]

Other References

If you are using any of the following open-source code please cite them accordingly:

Simple Behavioral Analysis (SimBA) – an open source toolkit for computer classification of complex social behaviors in experimental animals; Simon RO Nilsson, Nastacia L. Goodwin, Jia Jie Choong, Sophia Hwang, Hayden R Wright, Zane C Norville, Xiaoyu Tong, Dayu Lin, Brandon S. Bentzley, Neir Eshel, Ryan J McLaughlin, Sam A. Golden bioRxiv 2020.04.19.049452; doi: https://doi.org/10.1101/2020.04.19.049452

B-SOiD: An Open Source Unsupervised Algorithm for Discovery of Spontaneous Behaviors; Alexander I. Hsu, Eric A. Yttri bioRxiv 770271; doi: https://doi.org/10.1101/770271

SLEAP: Multi-animal pose tracking; Talmo D. Pereira, Nathaniel Tabris, Junyu Li, Shruthi Ravindranath, Eleni S. Papadoyannis, Z. Yan Wang, David M. Turner, Grace McKenzie-Smith, Sarah D. Kocher, Annegret L. Falkner, Joshua W. Shaevitz, Mala Murthy bioRxiv 2020.08.31.276246; doi: https://doi.org/10.1101/2020.08.31.276246

Real-time, low-latency closed-loop feedback using markerless posture tracking; Gary A Kane, Gonçalo Lopes, Jonny L Saunders, Alexander Mathis, Mackenzie W Mathis; eLife 2020;9:e61909 doi: 10.7554/eLife.61909

natalia-enid / deeplabstream Goto Github PK

deeplabstream's Introduction