zoccoler/napari-flim-phasor-plotter

napari-flim-phasor-plotter

Napari-flim-phasor-plotter is a napari plugin to interactively load and show raw fluorescence lifetime imaging microscopy (FLIM) single images and series and generate phasor plots. These are Fourier transforms of the decay data being visualized using the napari-clusters-plotter plotter, adapted to suit the FLIM context. This allows qualitative and quantitative downstream analysis of FLIM images.

---

Quick demo

Contents

• Installation
• Usage
  • Loading Raw FLIM Data
    1. Input Data Types
    2. Opening a Raw FLIM Image
    3. Loading Stacks
    4. Sample Data
  • Phasor Analysis
    1. Generating Phasor Plots
    2. Phasor Plot Navigation
    3. Phasor Plot Selection
    4. Integrating Phasor Analysis into a Workflow
       • Clustering
       • Post-processing
  • Saving Results
  • Data Conversion
• Limitations
• Contributing
• License
• Issues

Installation

We recommend installing napari-flim-phasor-plotter with mamba after having Miniforge installed in your computer. Follow these steps from a terminal.

Click here to choose the right download option for your OS.

Create a conda environment:

mamba create -n napari-flim-phasor-env python=3.9 napari pyqt git

Activate the environment:

mamba activate napari-flim-phasor-env

Install napari-flim-phasor-plotter plugin with:

pip install napari-flim-phasor-plotter

Alternatively, clone this repository and install the latest plugin development version with:

pip install git+https://github.com/zoccoler/napari-flim-phasor-plotter.git

Usage

Loading Raw FLIM Data

1. Input Data Types

This plugin can read the following FLIM file types:

• .ptu
• .sdt
• .tif
• .zarr

This plugin works with the following data shapes:

• 2D FLIM images (actually 3D data where FLIM counts are in the first axis).
• 3D FLIM images (actually 4D data where FLIM counts are in the first axis).
• 3D timelapse FLIM images (actually 5D data where FLIM counts are in the first axis).
• Multichannel '.tif' or '.zarr' data may need to be loaded separately.

If you read your files using this plugin as a reader, it returns and works with the data axes in the following order (data from multiple detectors are displayed as distinct napari layers):

(flim_counts, time, z, y, x)

Even if the data is 2D, the plugin will add a unitary time and a z axis. It also provides the standard intensity image in another layer by summing the flim_counts dimension.

2. Opening a Raw FLIM Image

Drag and drop a compatible file format (check supported file formats here below) to open a FLIM image. It gets displayed in two layer: a 'raw FLIM image series' (a sequence of intensity images each corresponding to an individual time point of the FLIM 'micro-time'), and a timely summed up image (usually just known as the 'intensity' image). Scrolling through the FLIM time series provides a first glimpse of lifetimes across image regions.

3. Loading Stacks

If you have multiple slices or time-points as separated files, you can choose a folder containing the files. In order for the plugin to properly build a stack, the file names must contain some indication about which slice or time-point they represent, i.e., each file name should contain a _t and/or _z followed by a number. This number should start from 1 and increase by 1 for each new slice or time-point.

Here are a few example templates:

• timelapse:
  • image_t001.ptu
  • image_t002.ptu
• z-stack:
  • image_z01.sdt
  • image_z02.sdt
• 3D timelapse:
  • image_t001_z001.tif
  • image_t001_z002.tif
  • ...
  • image_t002_z001.tif

4. Sample Data

The plugin comes with a few sample FLIM raw images:

• '2D' raw FLIM images:
  • Hazelnut (originally a '.ptu' file)
  • Seminal Receptacle (originally a '.sdt' file)
• '3D' raw FLIM image stack (Hazelnut 3D)
  • Hazelnut 3D (originally a series of '.ptu' files)
• '2D' synthetic FLIM image
  • Lifetime Cat

To load it, go to File > Open Sample -> FLIM phasor plotter.

Phasor Analysis

1. Generating Phasor Plots

Call the plugin from the menu Plugins > FLIM phasor plotter > Calculate Phasors to generate a phasor plot by pixel-wise Fourier transformation of the decay data. Hereby, select the FLIM image to be used (it should be the layer with the raw data), specify the laser pulse frequency (if information is present in the file metadata, this field will be updated after phasor calculation). Choose a harmonic for optimal visualization, define an intensity threshold (here in absoluete values) to exclude pixels of low photon counts, and optionally apply a number of iterations n of a 3x3 median filter. Run creates the phasor plot and an additional labels layer in the layer list.

2. Phasor Plot Navigation

Use the toolbar on top of the plot to navigate through the plot. For example, by activating the zoom tool button (magnifying glass icon), you can zoom in (with left click) or out (with right click), just remember to disbale the zoom tool after using it by clicking on the icon once again.

Change the colormap of the phasor plot from various Colormaps by clicking on the pulldown Expand for advanced options. There, you can also choose to display the color range in log scale by checking the Log scale checkbox. Optionally, add tau lines to the plot by specifying a range of lifetimes to be displayed (write them separated by commas) in the field Tau lines anc click on Show/hide to visualize them on top of the phasor plot.

3. Phasor Plot Selection

Manually encircle a region of interest in the phasor plot to highlight the corresponding pixels in the newly created image layer. Hold ‘Shift’ to select and visualize several clusters as a way to investigate image regions of similar FLIM patterns.

4. Integrating Phasor Analysis into a Workflow

Clustering

This plugin integrates with napari-clusters-plotter plugin. Thus, you can use the clustering widget provided by the napari-clusters-plotter plugin to segment the phasor plot automatically and then visualize the segmentation results in the original FLIM image. Access it via Tools -> Measurement post-processing -> Clustering (ncp). Below is a demonstration:

Post-processing

After cluster selection, it is common to have different labels (colors) for selected clusters. Within each label, it is also common to have disconnected regions and even isolated pixel in the segmentation. To address this, we offer a few basic post-processing functions.

A common step is to select a single cluster of interest for further processing. By selecting the Labels layer (usually named cluster_ids_in_space) and checking the show selected checkbox, we can identify our cluster/label os interest by continuously increasing the label number until we find the desired cluster. After that, we can extract the chosen label as a mask via Plugins -> FLIM phasor plotter -> Manual Label Extraction. This will create a new layer with the mask of the selected cluster.

To connect small isolated regions and remove small holes within the mask, we can use the smooth_cluster_mask function. This can be accessed via Plugins -> FLIM phasor plotter -> Smooth Cluster Mask. This will remove holes with an area smaller than the specified fill area px in total number of pixels and connect regions within a given smooth radius. Don't forget to select the layer containing the mask before running the function, because this function expects a layer with a single label (like a binary mask).

Beyond this point, we can use other plugins, like the napari-segment-blobs-and-things-with-membranes and napari-skimage-regionprops plugins, to further process the mask. For example, we can perform instance segmentation on the mask via Tools -> Segmentation / labeling -> Connectec component labeling (scikit-image, nsbatwm). We can also extract features from the objects via Tools -> Measurement tables -> Objetct Features/Properties (scikit-image, nsr).

Below is a demonstration of the post-processing steps:

Also, this example workflow can be reproduced by running this jupyter notebook: Example_workflow.ipynb.

Saving Results

Save your segmentation results by selecting (clicking on) the corresponding Labels layer (usually named cluster_ids_in_space) and then going to File -> Save Selected Layer. This can save the layer as a .tif file. To save a screenshot of the phasor plot, click on the Save button on the toolbar. To save the phasor plot as a .csv file, go to Tools -> Measurement -> Show table (nsr) and a new widget will show up. From the labels layer dropdown, choose the layer that contains the table, whose name starts with Labelled_pixels_from_... and then click on the Run button. This should bring the table with G and S values for each pixel. Click on the Save as csv... button to save the table as a .csv file.

Data Conversion

If a collection of raw (uncompressed) images are larger than 4GB, we recommend converting them to .zarr. This can be done via Plugins > napari-flim-phasor-plotter > Convert to zarr.

Warning: In the current version, lazy loading with .zarr is available, but processing may still load all data into memory, so keep track of your memory usage.

Limitations

The plugin does not yet support:

• Phasor calibration
• Round cluster selection or cursor selection (only free-hand selection is available)
• Pseudo-channel generation from selected clusters in the phasor plot
• FRET analysis
• Tile processing
• Fitting of decay curves

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the BSD-3 license, "napari-flim-phasor-plotter" is free and open source software.

If you use this plugin in a publication, please cite us: https://doi.org/10.5281/zenodo.12620956

Issues

If you encounter any problems, please file an issue along with a detailed description.

napari-flim-phasor-plotter's Issues

Open tif error

I installed napari-flim-phasor as the steps, and type napari in the command line in this env, and try to open a tif image or stack in napari, there's error:
TypeError: 'NoneType' object is not subscriptable
No matter the tif is 2d or 3d, it both don't work. It's because of 'tif.shaped_metadata = None'. It seems like it can't read my tif's shape properly. What should I do?
Also, when I try to drag in ptu file, the napari break down, the window closed itself.

Add IRF

This current implementation masks data in time starting from the whole image histogram peak.
We should find a way to use the IRF to obtain this information instead.

Avoid using `Labels.color`

With napari 0.4.19, this warnings pops up after manually selecting a cluster

FutureWarning: Labels.color is deprecated since 0.4.19 and will be removed in 0.5.0, please set Labels.colormap directly with an instance of napari.utils.colormaps.DirectLabelColormap instead.!

Reader function is unable to process the url

Calling the plug-in reader function on url results in error message instead of a file download:

File ~/Documents/GitHub/napari-flim-phasor-plotter/src/napari_flim_phasor_plotter/_reader.py:25, in napari_get_reader(path='https://zenodo.org/record/7656540/files/2a_FLIM_single_image.ptu')
     12 """A basic implementation of a Reader contribution.
     13 
     14 Parameters
   (...)
     23     same path or list of paths, and returns a list of layer data tuples.
     24 """
---> 25 file_extension = get_most_frequent_file_extension(path)
        path = 'https://zenodo.org/record/7656540/files/2a_FLIM_single_image.ptu'
     26 # If we recognize the format, we return the actual reader function

File ~/Documents/GitHub/napari-flim-phasor-plotter/src/napari_flim_phasor_plotter/_reader.py:126, in get_most_frequent_file_extension(path=PosixPath('https:/zenodo.org/record/7656540/files/2a_FLIM_single_image.ptu'))
    125 # Get most frequent file entension in path
--> 126 most_frequent_file_type = max(set(suffixes), key=suffixes.count)
    127 return most_frequent_file_type

UnboundLocalError: local variable 'suffixes' referenced before assignment

Output layer scale is not preserved

In some widgets (binning, split N largest and manual_label_extract), the output labels layer loses the scale information (along with other layer properties).

Change installation instructions from conda to mamba

filter data for 'improved' clusters in phasor plot

applying filters creates 'denser' clusters in the phasor plot that are easier to distinguish (at least visually), let's find out how.

Median filter loads all data into memory

The current implementation of the median filter makes a full copy of the array.
So, if the image is very large or if it is a dask array, it will load all data into memory!

An independent implementation of the median filter with dask is necessary.
This may require rechunking the array and using dask_image.ndfilters.median_filter. Data may need to be rechunked again after median filter for proper fft calculation.

plugin unable to open sdt file

I am trying to open an sdt file retreived from a Becker Hickl FLIM instrument, but I am getting the following error message:
ValueError: could not broadcast input array from shape (1024,1024,1024) into shape (1024,1024)

Is there any way for me to convert the 3D array into 2D? Or is there some other troubleshooting needed?

implement wavelet filtering

Besides median filter, wavelets are also used to filter data.
We should have such an implementation for a denser visualization of clusters in the plot.

Overlay phasor cluster id layer with summed intensity image

hi @zoccoler & @sviaro,

the plugin nicely matures and the option for phasor cluster selection is great. 🚀

We would increase the level of information if we made the summed intensity image the top layer and overlayed it (e.g. with opacity 50%, blending additive as a default setting) with the cluster_ids_in_space (opacity 100%) as default upon cluster selection. This combines the phasor cluster classification with pixel intensities and provides overall more detailed info on the image. Fine tuning can be done manually then.

How about also renaming the cluster_ids_in_space layer to '(image name +) phasor cluster' in the layer list? May be more intuitive for users.

cheers,
Conni

Divide by zero Error in phasor calculation if DC average close to 0

During phasor calculation, pixels where DC is zero are replaced by DC average to avoid division by zero without interfering with the phasors. But if DC average is close to 0, it gets converted to integer and, thus, gets set to 0 again.

Fix median filter implementation

Median filter is being applied to the original image, but according to the literature, it should be applied to the g and s images.

3D hazelnut sample fails to be loaded

dear @zoccoler and @sviaro,

great to have sample data included and the synthetic cat data is amazing.

the only sample that cannot be loaded is the 3D hazelnut dataset, with following error:

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
File ~\AppData\Local\miniconda3\envs\flim-phasor-plotter-230817\lib\site-packages\napari\_qt\menus\file_menu.py:219, in FileMenu._rebuild_samples_menu.<locals>._add_sample(plg='napari-flim-phasor-plotter', smp='hazelnut_z_stack', *args=(False,))
    217 def _add_sample(*args, plg=plugin_name, smp=samp_name):
    218     try:
--> 219         self._win._qt_viewer.viewer.open_sample(plg, smp)
        plg = 'napari-flim-phasor-plotter'
        smp = 'hazelnut_z_stack'
        self._win = <napari._qt.qt_main_window.Window object at 0x000001764A6A42E0>
        self = <napari._qt.menus.file_menu.FileMenu object at 0x0000017658F5CD30>
    220     except MultipleReaderError as e:
    221         handle_gui_reading(
    222             e.paths,
    223             self._win._qt_viewer,
    224             plugin_name=plugin_name,
    225             stack=False,
    226         )

File ~\AppData\Local\miniconda3\envs\flim-phasor-plotter-230817\lib\site-packages\napari\components\viewer_model.py:912, in ViewerModel.open_sample(self=Viewer(axes=Axes(visible=False, labels=True, col...._transform_active_layer at 0x0000017656FF7820>}), plugin='napari-flim-phasor-plotter', sample='hazelnut_z_stack', reader_plugin=None, **kwargs={})
    910 if callable(data):
    911     added = []
--> 912     for datum in data(**kwargs):
        data = <bound method SampleDataGenerator.open of SampleDataGenerator(command='napari-flim-phasor-plotter.load_hazelnut_z_stack', key='hazelnut_z_stack', display_name='Hazelnut (3D Raw FLIM)')>
        kwargs = {}
    913         added.extend(self._add_layer_from_data(*datum))
    914     return added

File ~\AppData\Local\miniconda3\envs\flim-phasor-plotter-230817\lib\site-packages\npe2\manifest\contributions\_sample_data.py:44, in SampleDataGenerator.open(self=SampleDataGenerator(command='napari-flim-phasor-..._z_stack', display_name='Hazelnut (3D Raw FLIM)'), _registry=None, *args=(), **kwargs={})
     41 def open(
     42     self, *args, _registry: Optional["CommandRegistry"] = None, **kwargs
     43 ) -> List[LayerData]:
---> 44     return self.exec(args, kwargs, _registry=_registry)
        args = ()
        kwargs = {}
        self = SampleDataGenerator(command='napari-flim-phasor-plotter.load_hazelnut_z_stack', key='hazelnut_z_stack', display_name='Hazelnut (3D Raw FLIM)')
        _registry = None

File ~\AppData\Local\miniconda3\envs\flim-phasor-plotter-230817\lib\site-packages\npe2\manifest\utils.py:63, in Executable.exec(self=SampleDataGenerator(command='napari-flim-phasor-..._z_stack', display_name='Hazelnut (3D Raw FLIM)'), args=(), kwargs={}, _registry=None)
     61     kwargs = {}
     62 reg = _registry or kwargs.pop("_registry", None)
---> 63 return self.get_callable(reg)(*args, **kwargs)
        reg = None
        kwargs = {}
        args = ()
        self = SampleDataGenerator(command='napari-flim-phasor-plotter.load_hazelnut_z_stack', key='hazelnut_z_stack', display_name='Hazelnut (3D Raw FLIM)')

File ~\AppData\Local\miniconda3\envs\flim-phasor-plotter-230817\lib\site-packages\napari_flim_phasor_plotter\_sample_data.py:90, in load_hazelnut_z_stack()
     87 from napari_flim_phasor_plotter._reader import read_stack
     89 folder_path = DATA_ROOT / "hazelnut_FLIM_z_stack"
---> 90 image, metadata = read_stack(folder_path)
        folder_path = WindowsPath('C:/Users/cblei/AppData/Local/miniconda3/envs/flim-phasor-plotter-230817/lib/site-packages/napari_flim_phasor_plotter/data/hazelnut_FLIM_z_stack')
     91 image, metadata = image[0], metadata[0]  # Use first channel, second detector is empty
     92 return [(image, {'name': 'hazelnut raw FLIM z-stack',
     93                  'metadata': metadata,
     94                  'contrast_limits': (np.amin(image[image.shape[0] // 2, ...]),
   (...)
    101                                   }),
    102         ]

File ~\AppData\Local\miniconda3\envs\flim-phasor-plotter-230817\lib\site-packages\napari_flim_phasor_plotter\_reader.py:293, in read_stack(folder_path=WindowsPath('C:/Users/cblei/AppData/Local/minico..._flim_phasor_plotter/data/hazelnut_FLIM_z_stack'))
    291 from natsort import natsorted
    292 from napari.utils import notifications
--> 293 file_extension = get_most_frequent_file_extension(folder_path)
        folder_path = WindowsPath('C:/Users/cblei/AppData/Local/miniconda3/envs/flim-phasor-plotter-230817/lib/site-packages/napari_flim_phasor_plotter/data/hazelnut_FLIM_z_stack')
    294 if file_extension == '.zarr':
    295     file_paths = folder_path

File ~\AppData\Local\miniconda3\envs\flim-phasor-plotter-230817\lib\site-packages\napari_flim_phasor_plotter\_reader.py:193, in get_most_frequent_file_extension(path=WindowsPath('C:/Users/cblei/AppData/Local/minico..._flim_phasor_plotter/data/hazelnut_FLIM_z_stack'))
    191         suffixes = [path.suffix]
    192 # Get most frequent file entension in path
--> 193 most_frequent_file_type = max(set(suffixes), key=suffixes.count)
    194 return most_frequent_file_type

UnboundLocalError: local variable 'suffixes' referenced before assignment

could you have a look at it @zoccoler ?

Thanks!

Add browse button to zarr conversion

Implement Binning

The current implementation makes a label for each pixel. This may generate unnecessarily huge tables, which may hamper performance for large data.

Implementing a binning option could reduce these tables.

Error when running a median filter

When running a median filter on a sample FLIM image I receive the following error:

EmitLoopError: calling <psygnal._weak_callback._StrongFunction object at 0x7f87856bba40> with args=(False,) caused
RuntimeError: filter footprint array has incorrect shape..

Display intensity images when openning

Currently, images are opened as 'timelapse' (photon counting time), which is appropriate for phasor calculation.
However, user usually expect that the summed intensity image comes along.
It would be nice to have them added as additional layers.

Cannot open `.sdt` files anymore

import stacks of more dimensional data

import & process ptu files of e.g z stacks and/or time series for which I could provide examples

laser_frequency field may be unnecessary

Consider removing it from input fields if it is really unnecessary

Documentation needs update for `0.0.6`

Some menu names have changed and the gif in the readme is outdated.

Add Cancel button to convert to zarr widget

This may require multithreading.

Phasor Plot Error

I am encountering a major error in my phasor plot when I use the flim-phasor-plotter plugin. I used the napari-sdtfile plugin to upload my sdt file into napari, then used the phasor plotter plugin. Here is the plot I got:

I have tried uploading the file in .tif format as well, and I get a very similar looking phasor plot. Any ideas about how to troubleshoot this error?

Laser frequency field should be updated along with dropdown selection

If available in metadata

Implement binning for dask arrays

Overlay FLIM phasors of different datasets in plotter

Overlaying 2 or more datasets in the plotter (visualized with different transparency/colors) would advance the options to comparatively analyse FLIM datasets. This would have to be implemented in the napari-clusters-plotter, issue opened there.

Replace ptu library

There are a few python libraries now capable of reading ptu files.
The ptu reader code here should be replaced by https://pypi.org/project/ptufile/ for example

Add Heatmap to Plotter

Phasor Plots are usually displayed as a heatmap or 2D Histogram.
This would be a good enhancement to the Plotter. This may be implemented in the napari-cluster-plotter plugin, there is an issue there for this.

Increase tests coverage

Rename repository/plugin

The name of this plugin is not ideal and does not depict what it actually can do.

Sample lifetime cat layer names must be more clear

Right now, the layer names don't allow knowing which one is the raw data and which one is the intensity image.
This may lead the user to have the wrong one selected in the dropdown fields.

Add sample data

The plugin should have some small sample data:

• one '.ptu' image file
• one '.sdt' image file
• a small folder with files to convert to '.zarr'
• one small 4D data, probably the same '.zarr' file from above

Optionally plot tau lines

One or multiple tau lines would be important to have displayed for phasor plot calibration

Plotter updates time from wrong slider

Here we use the convention that the first axis contains the photon counting information, aka, microtime (ut), while the napari-clusters-plotter consider the first axis the time.
Thus, changing the time slider for FLIM data (second axis here) does not update the plot while changing the microtime does.