kushalkolar/mesmerize-viz

Widgets for calcium imaging analysis visualization in notebooks

mesmerize-viz

Motion Correction and CNMF visualizations should just work. CNMFE will work without "rcb" and "residuals" image_data_options.

❗ Harware requirements The large CNMF visualizations with contours etc. usually require either a dedicated GPU or integrated GPU with access to at least 1GB of VRAM.

https://www.youtube.com/watch?v=GWvaEeqA1hw

Installation

After you have mesmerize-core installed:

pip install mesmerize-viz

If you want to use %gui qt you will need pyqt6 or PySide6:

pip install PyQt6
# or
pip install PySide6

Usage

Explore parameter variants

Click on different rows to view the results of different runs of motion correction, CNMF or CNMFE.

mesviz-variants.mp4

Explore components

Explore components using the heatmap selector, or the component index slider. Auto-zoom into components if desired using the checkbox, set the zoom scale using the slider.

mesviz-explore-components.mp4

Visualize component evaluation metrics

View the evaluation metrics by setting the contour colors based on the metrics. Select to show "all", or only "accepted" or only "rejected" components based on the current evaluation criteria. You can also choose to make the accepted or rejected components semi-transparent instead of entirely opague or invisible using the alpha slider.

Colormaps used:

accepted/rejected: Set1, accepted - blue, rejected - red

snr, r_values, and cnn_preds: spring: low value: pink, high value: yellow

mesviz-eval-metrics.mp4

Interactive component evaluation using metrics and manully accept or reject components

Interactively change the metric thresholds for the sliders. See the caiman docs for info on the evaluation params: https://caiman.readthedocs.io/en/latest/Getting_Started.html#component-evaluation

After setting the metric thresholds, you can manually accept or reject components by clicking on them and pressing "a" (accept) or "r" (reject) keys on your keyboard.

When you are happy with component evaluation, click "Save eval to disk". This overwrites the existing hdf5 file with the state of the hdf5 file as shown in the visualization, i.e. estimates.idx_components and estimates.edx_components_bad gets set with respect to the visualization.

mesviz-eval-interactive.mp4

Voila app

Install voila:

pip install voila

Use as a voila app (as shown in the demo video).

cd mesmerize-viz
voila examples/app.ipynb --enable_nbextensions=True

Note that the voila app is a prototype

mesmerize-viz's Issues

Unable to display image correctly

I was trying out the mcorr_cnmf demo notebook the motion correction was finished successfully. However, I have trouble with the "Visualize with mesmerize-viz!" section. I could not have the image display properly it is all black as you can see in the following screen shot. I tried resetting the contrast but it does not help.

add eval stuff

similar to what exists in the mesmerize_core demo nb, it should also update the reconstructed movie, residuals, etc. live

WrongAlgorithmExtensionError

I tried to install mesmerize-viz on a win 11 machine and I got all kinds on errors regarding fastplotlib not finding modules and pygfx incompatibility, I tried pulling the latest ones from the links I also tried every version combination possible and it didn't work, the last error I got is in this attached file
Error last build.txt
Now I saw that you updated some things and I was happy, I pulled the version and tried it and now I'm getting this

WrongAlgorithmExtensionError              Traceback (most recent call last)
File ~\mambaforge\envs\mestest\lib\site-packages\ipywidgets\widgets\interaction.py:240, in interactive.update(self, *args)
    238     value = widget.get_interact_value()
    239     self.kwargs[widget._kwarg] = value
--> 240 self.result = self.f(**self.kwargs)
    241 show_inline_matplotlib_plots()
    242 if self.auto_display and self.result is not None:

Cell In[3], line 10, in start_widget(parent_path, batch_path)
      7 tab = Tab()
      9 mcorr_container = df.mcorr.viz()
---> 10 cnmf_container = df.cnmf.viz(start_index=2)
     12 tab.children = [mcorr_container.show(), cnmf_container.show()]
     13 tab.titles = ["mcorr", "cnmf"]

File c:\users\ba81\mestest\mesmerize-viz\mesmerize_viz\_cnmf.py:1164, in CNMFDataFrameVizExtension.viz(self, start_index, temporal_data_option, image_data_options, temporal_kwargs, reset_timepoint_on_change, input_movie_kwargs, image_widget_kwargs, data_grid_kwargs)
   1088 def viz(
   1089     self,
   1090     start_index: int = None,
   (...)
   1097     data_grid_kwargs: dict = None,
   1098 ):
   1099     """
   1100     Visualize CNMF output and other data columns such as behavior video (optional).
   1101 
   (...)
   1161         Example: `image_widget_kwargs={"cmap": "viridis"}`
   1162     """
-> 1164     container = CNMFVizContainer(
   1165         dataframe=self._dataframe,
   1166         start_index=start_index,
   1167         temporal_data_option=temporal_data_option,
   1168         image_data_options=image_data_options,
   1169         temporal_kwargs=temporal_kwargs,
   1170         reset_timepoint_on_change=reset_timepoint_on_change,
   1171         input_movie_kwargs=input_movie_kwargs,
   1172         image_widget_kwargs=image_widget_kwargs,
   1173         data_grid_kwargs=data_grid_kwargs,
   1174 
   1175     )
   1177     return container

File c:\users\ba81\mestest\mesmerize-viz\mesmerize_viz\_cnmf.py:570, in CNMFVizContainer.__init__(self, dataframe, start_index, temporal_data_option, image_data_options, temporal_kwargs, reset_timepoint_on_change, input_movie_kwargs, image_widget_kwargs, data_grid_kwargs)
    566 self._component_index = 0
    568 self._cnmf_obj: CNMF = None
--> 570 data_arrays = self._get_row_data(index=start_index)
    571 self._set_data(data_arrays)
    573 self._set_params_text_area(index=start_index)

File c:\users\ba81\mestest\mesmerize-viz\mesmerize_viz\_cnmf.py:596, in CNMFVizContainer._get_row_data(self, index)
    589 def _get_row_data(self, index: int) -> Dict[str, np.ndarray]:
    590     data_mapping = get_cnmf_data_mapping(
    591         series=self._dataframe.iloc[index],
    592         input_movie_kwargs=self.input_movie_kwargs,
    593         temporal_kwargs=self.temporal_kwargs
    594     )
--> 596     temporal = data_mapping[self.temporal_data_option]()
    598     rcm = data_mapping["rcm"]()
    600     shape = rcm.shape

File c:\users\ba81\mestest\mesmerize-viz\mesmerize_viz\_cnmf.py:94, in ExtensionCallWrapper.__call__(self, *args, **kwargs)
     93 def __call__(self, *args, **kwargs):
---> 94     rval = self.func(**self.kwargs)
     96     if self.attr is not None:
     97         return getattr(rval, self.attr)

File ~\mestest\mesmerize-core\mesmerize_core\caiman_extensions\_utils.py:18, in validate.<locals>.dec.<locals>.wrapper(self, *args, **kwargs)
     16 if algo is not None:
     17     if algo not in self._series["algo"]:
---> 18         raise WrongAlgorithmExtensionError(
     19             f"<{algo}> extension called for a <{self._series.algo}> item"
     20         )
     22 if not self._series["outputs"]["success"]:
     23     tb = self._series["outputs"]["traceback"]

WrongAlgorithmExtensionError: <cnmf> extension called for a <mcorr> item

which doesn't look like a problem with my setup, so I though, instead of spending two days debugging I could ask you for advice.
Thank you so much for the help :)

AWS or cloud example

Make an example of this running on some cloud compute service

pandas series extensions for viz

It might be better to create a set of series extensions for visualization of cnmf and mcorr. These extensions can return something like a container class instance with attributes such as fastplotlib.Graphic and functions to update frame index.

This works, kinda messy but something to start with

from mesmerize_core import *
from fastplotlib import Image, Plot, Line, Heatmap, GridPlot
import pandas as pd
from dataclasses import dataclass
from ipywidgets import IntSlider, VBox
from ipywidgets import link as widget_link
import numpy as np
from functools import partial
from typing import *


@dataclass
class GraphicManager:
    """A dataclass which is just a container for a graphic and an observe handler for sliders to use"""
    name: str
    graphics: Dict[str, Union[Image, Line, Heatmap]]
    data: np.ndarray
    
    def observe_handler(self, change: dict):  # abstract method
        """Implemented in extension function"""
        pass
    
    def update_component_index(self, index: int):  # abstract method
        """Implemented in extension function"""
        pass


@pd.api.extensions.register_series_accessor("cnmf_viz")
class VizCNMF:
    """Extensions for visualization"""
    def __init__(self, s):
        self._s = s
        
    def make_frame_slider(self, **kwargs):
        movie = self._s.caiman.get_input_movie()
        
        slider_kwargs = dict(
            min=0,
            max=movie.shape[0],
            step=1,
            description="frame slider"
        )
        
        kwargs = {**slider_kwargs, **kwargs}
        
        return IntSlider(**kwargs)
    
    def get_input_movie(
        self, 
        plot=True, 
        ext_kwargs: dict = None, 
        graphic_kwargs: dict = None,
        make_slider: bool = True
    ):
        """
        Returns a GraphicManager containing a fastplotlib.Graphic that can be added 
        to a GridPlot and a slider that is already set to update the graphic using 
        slider.observe(<update_frame_index_function>)
        """
        if ext_kwargs is None:
            ext_kwargs = dict()
            
        if graphic_kwargs is None:
            graphic_kwargs = dict()
            
        movie = self._s.caiman.get_input_movie(**ext_kwargs)
        ig = Image(movie[0], **graphic_kwargs)
        
        gm = GraphicManager(
            name="mcorr-movie",
            graphics=[ig],
            data=movie
        )
        
        def _observe_handler(_gm, change):
            index = change["new"]
            _gm.graphics[0].update_data(_gm.data[index])
            
        gm.update_frame_index = partial(_observe_handler, gm)
        
        if make_slider:
            slider = self.make_frame_slider()
            slider.observe(gm.update_frame_index, names="value")
        else:
            slider = None
        
        return gm, slider
    
    def get_contours():
        pass
    
    def get_rcm(self, make_slider=True):
        frame0 = self._s.cnmf.get_rcm(component_indices="good", frame_indices=0)[0]
        
        ig = Image(frame0)
        
        gm = GraphicManager(
            name="rcm-movie",
            graphics=[ig],
            data=None
        )

        def _observe_handler(_gm, change):
            index = change["new"]
            frame = self._s.cnmf.get_rcm(component_indices="good", frame_indices=index)[0]
            _gm.graphics[0].update_data(frame)
            
        gm.update_frame_index = partial(_observe_handler, gm)
        
        if make_slider:
            slider = self.make_frame_slider()
            slider.observe(gm.update_frame_index, names="value")
        else:
            slider = None
        
        return gm, slider

    def get_rcb(self, make_slider=True):
        pass
    
    def get_residuals():
        pass

gp = GridPlot(shape=(1, 2), controllers="sync")
df = load_batch("/data/meso-data/Opto_inter_region/mesmerize_batch/batch.pickle")

r = df.iloc[35]
gm_movie, slider_movie = r.cnmf_viz.get_input_movie()
gm_rcm, slider_rcm = r.cnmf_viz.get_rcm()

widget_link((slider_movie, "value"), (slider_rcm, "value"))

gp.subplots[0, 0].add_graphic(gm_movie.graphics[0])
gp.subplots[0, 1].add_graphic(gm_rcm.graphics[0])

# show only one slider, the other is hidden but linked
VBox([gp.show(), slider_movie])

The above code results in this:

@clewis7 thoughts? I think this would make mesmerize-viz easier to develop, and I think it would make it much easier to have a multi-item selection widget with these extensions and a proper "data container" class to manage the graphic and frame updates. It would also allow arbitrarily adding and linking many more subplots and visualizations without code getting messy.

CNMF widget

• mcorr movie with colored contours
• A ⨂ C
• A ⨂ C with customizable C, like dF/F0 etc.
• residuals
• b ⨂ f
• heatmap of traces with infline that indicates frame slider position
• stacked lineplot of traces

for CNMF allow contours to be drawn on any of the subplots

useful to have contours drawn on the RCM, residuals and bg movie as well

improvements

1. Customizing what data to plot: A toggle button for each option, like input, rcm, temporal, heatmap etc. and an Add Plot button which adds the selected items to a list of plots that are then spit out.
2. When a CNMF item is selected just show the first component to start with.

@EricThomson welcome to add

setting metric sliders' limits fails if it violates pre-existing min-max conditions

mesmerize-viz/mesmerize_viz/_cnmf.py line 224. if the metric_array min is larger than the existing default max, the widget fails. You can imagine the same if the max is lower than the default min. Easy fix using a try-except statements or conditionals.

allow using any column in batch item list for blinding

When creating an instance of DataFrameWidget use an item_names arg to set the DataFrame column to use for names displayed in the batch_item_list_widget. Create an blind_name column which is a single uuid for each unique batch item name. This way the blind_name that originates from the same input movie will be identical, allowing the user to determine the best params per-movie.

playing with nvjpeg with jupyter_rfb

Just noting this for future reference.

On a laptop, the remote frame buffer is quite CPU-heavy due to the jpeg encoder, even if using simplejpeg which has turbo-jpeg. The use of nvjpeg frees up CPU time, and the power is shifted to the GPU, however the performance is still the same on a T1200 GPU. Maybe fancier GPUs can perform better with nvjpeg.

The issue is that fps is quite limited on a laptop, it seems like the power budget is heavily shifted towards the CPU for jpeg encoding. Using nvjpeg shifts the power budget to the GPU, so maybe this is still useful with more complex visualizations.

Still need to test nvjpeg on desktop. nvjpeg is twice as fast as simplejpeg on my laptop, so that might mean there's some other bottleneck in jupyter_rfb

So far I'm just using this for nvjpeg: https://github.com/UsingNet/nvjpeg-python

Failed to create view for 'FloatSliderView' in eval params tab

Hi,
When creating a Mesmerize-viz visualization (using viz_cnmf = df.cnmf.viz() then viz_cnmf.show()), the sliders for min_SNR and SNR_lowest fail to load.

Any idea what could be wrong here?

Thanks

Universal way to get_input_movie() for CNMF

If it's not a memmap, use implementation specific to the file format or have a widget for the user to select the correct loader.

gsig filt viz for mcorr, use a checkbox to enable it

customizable coloring of contours

based on things like:

• component index
• vals from a metric, like SNR, cnn value, etc.
• good vs. bad component

use `pint` to manage timepoints and sampling rates

display both frame index and time in seconds/ms for sliders, pint should make this easy.

requirements for getting this on conda-forge and constructor

These need to be put on conda-forge:

• jupyter_rfb
• fastplotlib
• wgpu-py
• pygfx

Even with the above on conda-forge, it will not be very usable without turbo-jpeg. Potential solutions:

• swap out simplejpeg for pyturbojpeg which is already on conda-forge: https://github.com/lilohuang/PyTurboJPEG/

  • pyturbojpeg gives the same performance as simplejpeg

• use libnvjpeg, it's already on conda-forge: https://anaconda.org/conda-forge/libnvjpeg

Things to add to the recipe:

• libturbojpeg system library so that it gets found by windows

Plot feature: Error bar or scale bar?

Hi,
Just a quick question: what is the bar to the left of the temporal trace? A scale bar? Or some measure of noise or std ? What variable is it based on?

Thanks,
Vincent

Better contour interaction

Maintain alpha when highlighting component

Ignore click if alpha = 0

Checkbox for highlighting in white vs. just increasing thickness, useful when contours have cmap_values.

user-end API

instead of making complex widgets with dropdown etc. we can just have the user specify what they want in each part of the gridplot.

Simple example

from mesmerize_viz.widgets import CNMFWidget
from mesmerize_viz.visuals import cnmf_visuals as cvs

shape = (2, 2)
visualizations = [
    [cvs.input + cvs.contours,  # input movie + contours
     cvs.rcm + cvs.contours,  # these get stacked in the z-axis in the order of addition 
     cvs.lineplot],

    [cvs.rcb,
     cvs.residuals,
     cvs.heatmap(cmap="viridis")]  # maybe allow these to be optionally callable with kwargs
]

names = [
    ["input", "rcm", "traces"],
    ["rcb", "residuals", "heatmap"]
]

sliders =\  # slider(s) linked according to this
    [[0,     0,     None],
     [0,     0,     None]]

# events to link visuals together, the details are predefined by us using the fastplotlib API
events = [
    cvs.contours >> cvs.heatmap,
    cvs.heatmap >> cvs.contours,
    cvs.heatmap >> cvs.lineplot
]

Make the visualization
cnmf_viz = CNMFWidget(
    shape=(2, 3),  # makes gridplot of [2, 3]
    visualizations=visualizations,
    sliders=sliders,
    events=events,
    selection="single"  # single select batch items
)

cnmf_viz.show()

Multi selection

# set properties of visuals like this
cnmf_viz["input"].cmap = "gnuplot2"

cnmf_viz["input"].contours = "good-bad"
cnmf_viz["rcm"].contours = "index"

# multi-selection works exactly the same, stuff is added to grid in the order that they user selects them

visualizations_multi = [
    [cvs.input + cvs.contours,
     cvs.input + cvs.contours,
     cvs.input + cvs.contours,]

    [cvs.input + cvs.contours,
     cvs.input + cvs.contours,
     cvs.input + cvs.contours,]
]

cnmf_viz_multi = CNMFWidget(
    shape=(2, 3),  # makes gridplot of [2, 3]
    visualizations=visualizations_multi,
    sliders="sync-all", # all subplots get a single slider that synced them all
    events=None,
    selection="multi"  # multi select batch items
)

Customizability

# customize visuals even more
# a function that just returns dfof traces
def custom_temporal(cnmf_item: pd.Series) -> np.ndarray:
    cnmf_item.run_detrend_dfof()
    return cnmf_item.get_detrend_dfof()

# make reconstructed movie with dfof traces instead of raw `cnmf_obj.C` temporal components
rcm_custom= cvs.rcm(temporal=custom_temporal)

visualizations = [
    [cvs.input + cvs.contours,
     rcm_custom,  # RCM with the dfof used for temporal
     cvs.lineplot],

    [cvs.rcb,
     cvs.residuals,
     cvs.heatmap(cmap="viridis")]  # maybe allow these to be optionally callable with kwargs
]

cnmf_viz = CNMFWidget(
    shape=(2, 3),  # makes gridplot of [2, 3]
    visualizations=visualizations_multi,
    sliders="sync-all", # all subplots get a single slider that synced them all
    events=None,
    selection="multi"  # multi select batch items
)

@clewis7 this will be so much easier to implement
@EricThomson comments?

button to edit params of a batch item and add it as a new item

useful for parameter optimization when looking at a batch item output and then thinking of new params to tryout.

Interactive component initialization

Make an interactive component evaluation tool that uses a few hundred samples frames for fast initialization where sliders can be used to interactively change the params.

If it's too slow, param ranges can be defined and each variant run in a Process Pool.

pypi?

The installation instructions have the user clone and install the library in edit mode. Is this package on pypi? If not, I would like to request that it is published for ease of installation and for reproducibility.

Organize by `item_name`

Show only the unique names in the listwidget of batch items, clicking on an item_name should then show the unique parameter variants for that item within another listwidget using df.caiman.get_params_diffs().

@clewis7 new idea, don't work on it yet, can see next week

Make a better default layout for cnmf

Mcorr, rcm, rcb and Residuals on left, temporal and heatmap on right

Maybe allow data_options to be a dict with "left" and "right" keys, and each of them accepts a list of lists

add heatmap to cnmf viz

once this is merged fastplotlib/fastplotlib#32

Some kind of general neuro viz widget would be useful

Takes in arrays for calcium video, behavior video, timeseries (traces, scatters etc), along with all their sampling rates.

ideas

• Base visualization consisting of a list widget, GridPlot, slider and "display" button.
  • Pass an iterator to populate list widget
  • create a callback function that takes in selected items in list and decides what/how to plot
    • for example def multi_select(), click a "display" button, auto-generates square GridPlot with the imaging data, useful for looking at CNMF outputs
    • mcorr example, def mcorr_evaluate(), auto-generates 1x3 GridPlot just to show side-by-side raw, mcorr movie, and down-sampled movie

Use ipywidgets or ipyvuetify ?

latest API proposal

I think we finally have enough from mesmerize-core and fastplotlib, as well as user experience, to make an elegant API for mesmerize-viz (which I can probably write in a weekend).

Basically, we make DataFrame and Series extensions for visualizations, to get rid of the common plotting code that is all over the current mesmerize-core demo nbs.

For example for mcorr:

df.iloc[i].mcorr.viz(["raw", "mcorr"], image_widget_kwargs={...})
# plots a ImageWidget of raw and mcorr

Similar for cnmf:

row.cnmf.viz([["mcorr", "rcm", "temporal_stack"], ["temporal"]],... kwargs)

plots this (and the RCM too): https://user-images.githubusercontent.com/9403332/210027199-6e4ac193-6096-4d18-80d5-a41591ea4d4f.gif

Contours are automatically put on the rcm plot etc. and all contours and temporal are linked by default. Use "good" contours by default.
There are checkboxes allowing to choose if you want to view good, bad, or no contours. If they want to change colors or other stuff they have to access the graphics themselves.

We can also allow these to be called at the DataFrame level.
This makes a grid of all items in the dataframe:

df.mcorr.viz("mcorr")

# or a subset of items
df.iloc[5:10].mcorr.viz("mcorr")

# if a list is provied we could do image widget gridplot where each gridplot row is from one dataframe row?
df.iloc[5:10].mcorr.viz(["mcorr", "mean"])
# providing mcorr and mean would also require changes to ImageWidget to allow non-matching dims

This plots only one item but provides a dataframe widget to go through multiple batch items:

df.mcorr.viz(["raw", "mcorr"]) 

Potential dataframe widgets:
widgetti/ipyvuetify#71
https://github.com/mwouts/itables

❗ Need to determine if these also work with vaex and/or polars if we migrate away from pandas.

Signatures:

row.mcorr.viz

• data
  • list of str, "raw" | "mcorr"
• input_movie_kwargs: dict
  • passed to caiman.get_input_movie()
• image_widget_kwargs: dict
  • passed to fastplotlib.ImageWidget

row.cnmf.viz

• data
  • list of str, or list of lists of str
    • each str must belong to: "contours", "mcorr", "rcm", "rcb", "residuals", "temporal", "temporal-stack", "temporal-hm" (temporal heatmap)
    • If it's a list of list, multiple gridplots are made which are then VBox stacked.
    • perhaps also allow showing things like "rcm" or "temporal" multiple times with different component_ixs args (see below), could be done like "rcm.0", "rcm.1".
• component_ixs: str, np.ndarray, or dict
  • if str or np.ndarray, used for get_temporal(), get_contours(), get_rcm(), etc.
  • if dict, must be like {"rcm": "all", "contours": "good"}. If there are multiple of the same data, can do {"rcm.0": "good", "rcm.1": "bad"}
• input_movie_kwargs: dict
  • passed to caiman.get_input_movie()
• image_widget_kwargs: dict
  • passed to fastplotlib.ImageWidget

Architecture

Allow the user to create an [m x n] grid, kinda like how in Google docs you can insert a table. This then generates a table of dropdown menus in the shape of [m x n] with options for mcorr or cnmf relevant plots. These dropdown menus are in a tab in the parent widget.

Example options:
raw movie
Mcorr movie
Downsampled avg movie
Correlation image
Shifts

Cnmf input movie
Movie with contours
Reconstructed
Residuals
Heatmap
Traces
Etc.

On the code side of things, the base viz widget is identical and various viz subplots can be shown in its GridPlot. These subplots are shown in the [m x n] Grid Plot which is in another tab in the parent widget. These Subplot classes are not the fastplotlib subplots.

Each Subplot can be managed by a class with a few methods (again diff from fastplotlib Subplot, maybe I should think of` a better name). Using the analogy of Qt signals and slots:

slot_batch_item_changed() 
slot_frame_index_changed()
slot_component_selection_changed()
slot_components_changed()

This class must inherit from an abstract base class to keep things sane. Must be a way to entire all subclasses implement all abstract methods, possibly __init_subclass__()

The individual subplots can send signals to the parent widget (the one that manages the batch list, frame slider etc), for example when component has been mouse hovered. The parent widget then signals all the Subplots. If a Subplot emits a signal, that emitter function has to be decorated with a signal blocker to prevent infinite recursion.

Lastly, a decorator so custom Subplot viz classes can be made and registered so that the parent widget shows it as an option when creating a grid plot. These custom viz classes must also inherit the ABC.

This architecture will prevent me from having to create tons of classes for each type of grid plot.

And for multi selection in the batch manager, we can have a grid of dropdown menus in another tab to choose what items to plot in that grid location.

release

make a release and publish to pypi once fastplotlib/fastplotlib#478 is done

pin fastplotlib version and set min python version as 3.10