juglab/labkit-ui

Advanced Tool for Labeling And Segmentation

Labkit (UI)

An advanced Fiji plugin for image segmentation.

Tutorials, installation instructions and documentation

>> Can be found on: https://imagej.net/plugins/labkit.

How to cite

Please cite our paper: LABKIT: Labeling and Segmentation Toolkit for Big Image Data

@article{arzt2022labkit
  author={Arzt, Matthias and Deschamps, Joran and Schmied, Christopher and Pietzsch, Tobias and Schmidt, Deborah and Tomancak, Pavel and Haase, Robert and Jug, Florian},   
  title={LABKIT: Labeling and Segmentation Toolkit for Big Image Data},      
  journal={Frontiers in Computer Science},      
  volume={4},      
  year={2022},      
  url={https://www.frontiersin.org/article/10.3389/fcomp.2022.777728},       
  doi={10.3389/fcomp.2022.777728},      
  issn={2624-9898},   
}

Project Structure

• This github repository contains the Labkit user interface / Fiji plugin.
• https://github.com/juglab/labkit-pixel-classification contains the implementation of the pixel classification algorithm.
• https://github.com/juglab/labkit-command-line contains a command line interface, that can be used to execute the Labkit pixel classification algorithm on the command line and HPC clusters.

Software Developement

If you developed a segmentation algorithm, but you don't want to develope your own UI. You might just use the Labkit UI, see this example:

CustomSegmenterDemo.java

(and don't hesitate to contact us ;)

labkit-ui's Issues

Add progress bar

When training a classifier, the user can not notice that something happens. A progress bar or something like this would be nice during training.

Fix grayscale images

High Performance Datastructures for Labeling

When creating dense labels on a big image 300x300x300 pixels:

• drawing becomes slow
• too much memory is used (12 bytes per labeled pixel)

We need better data structures to deal with these problems, possible approaches:

• SparseCellImg
  • Special cell image for BitType, which needs less memory when a cell is completely black / white.
  • It has a cursor that skips the complete black cell or pixels, such that iterating over white pixels is fast.
• Use OctTrees / NTrees
• Caches
  • Deal with the high memory usage by applying disk caches.
• Compression
  • Implement a compressed BitType. Compression maybe like LabelRegion, but with randomaccess and the possibility to change the data.

Throws error when selecting Classifier Settings

Hi,

I got below error when selecting Segmentation > Classifier Settings.

Maybe because one has to train the classifier at least once?

Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
	at net.imglib2.labkit.actions.ClassifierSettingsAction.lambda$new$0(ClassifierSettingsAction.java:19)
	at net.imglib2.labkit.menu.MenuFactory$Entry.lambda$asAction$0(MenuFactory.java:103)
	at org.scijava.ui.behaviour.util.RunnableAction.actionPerformed(RunnableAction.java:47)
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
	at javax.swing.AbstractButton.doClick(AbstractButton.java:376)
	at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:842)
	at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:886)
	at java.awt.Component.processMouseEvent(Component.java:6539)
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
	at java.awt.Component.processEvent(Component.java:6304)
	at java.awt.Container.processEvent(Container.java:2239)
	at java.awt.Component.dispatchEventImpl(Component.java:4889)
	at java.awt.Container.dispatchEventImpl(Container.java:2297)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904)
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4535)
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4476)
	at java.awt.Container.dispatchEventImpl(Container.java:2283)
	at java.awt.Window.dispatchEventImpl(Window.java:2746)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
	at java.awt.EventQueue$4.run(EventQueue.java:733)
	at java.awt.EventQueue$4.run(EventQueue.java:731)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

View changes resolution while mouse is down

Hi @maarzt ,

when I press the mouse button to annotate something, it appears that the resolution in the viewer is changing. When I release the key, it returns back to normal. Is it possible to turn this feature off?

Thanks!

Cheers,
Robert

Bug: Progress dialog gains focus all the time

Hi @maarzt ,

I was just trying to export a segmented stack. This popup show up:

Unfortunately, this dialog gains the focus all the time repeatedly. I can't use my computer anymore. Can this focus-thingy be disabled?

Thanks!

Cheers,
Robert

Change behaviour of parent eye icon in the labeling panel

Hello, I was wondering if it may be possible to slightly change behavior of the Parent Labeling Eye icon in the Labeling panel, so that clicking it either hides all labels or makes all labels visible (?)
For example,

Initial State:

After I click on the eye icon, it hides all labels:

After I reclick on the eye icon, I would expect it should reset visibility of all labels to on, but that doesn't happen currently and it reverts back to its previous state:

The reason for this suggestion is that when there are multiple labels, it is not easy to find which labels are currently not being displayed, so a reset visibility of labels solution might be helpful! Thank you.

Color button on mac

The button to change the labels color on mac is very ugly. Make it nicer

Display labels in an alphabetically-sorted fashion

Hello,

I noticed that when i load previously annotated labels in labkit, they are not displayed alphabetically in the Labeling panel on the left. Could an option to sort them so be made available? Thank you!

Keyboard short-cuts not functional

Hey @maarzt ,

I just discovered, that BDVs keyboard short-cuts are partly not functional in Labkit. I did not manage to open the help window by hitting F1. Furthermore, the brightness-contrast window was not opening when hitting S, depending on where one clicked before. By clicking in the BDV panel, this shortcut worked.

Last but not least: would it be possible to make these things (brightness/contrast, help, ....) accessible from the menu? That would be awesome :-)

Thanks!

Cheers,
Robert

Add 3d views with scenery

IllegalAccessError: tried to access method net.imglib2.util.StopWatch.<init>

Hi @maarzt ,

I just tried Labkit for the first time. It came via update site to my Fiji on Windows 10 64 bit with Java 1.8. I opened an image and clicked the menu Plugins > Segmentation > Labkit. An error came up:

Exception in thread "PainterThread" java.lang.IllegalAccessError: tried to access method net.imglib2.util.StopWatch.()V from class net.imglib2.cache.iotiming.IoStatistics
at net.imglib2.cache.iotiming.IoStatistics.(IoStatistics.java:21)
at net.imglib2.cache.iotiming.CacheIoTiming.getIoStatistics(CacheIoTiming.java:30)
at net.imglib2.cache.iotiming.CacheIoTiming.getIoStatistics(CacheIoTiming.java:17)
at net.imglib2.cache.iotiming.CacheIoTiming.getIoTimeBudget(CacheIoTiming.java:40)
at bdv.viewer.render.MultiResolutionRenderer.createProjector(MultiResolutionRenderer.java:658)
at bdv.viewer.render.MultiResolutionRenderer.paint(MultiResolutionRenderer.java:496)
at bdv.viewer.ViewerPanel.paint(ViewerPanel.java:485)
at net.imglib2.ui.PainterThread.run(PainterThread.java:86)

Furthermore, the viewer is empty:

Is there maybe something wrong with the installation via update site? A dependency might be missing? Any hint would be helpful!

Cheers,
Robert

Hide button in Progress dialog doesn't work

Hi @maarzt ,

I'm just playing with labkit (awesome tool btw!) and when exporting the final segmentation result to ImageJ, the progress dialog doesn't disappear when I click on Hide. Furthermore, it appears to be shown repeatedly. Thus, if I work in another program, this dialog is sent to front again and again.

I work on Win10 64 Bit and a recently updated Fiji and Java 8 172. If I can help debugging this, please let me know.

Cheers,
Robert

Increase usability for label editing

Currently Labkit's concept for label editing is very much like a very simple version of Paintbrush.
The usability provided by this concept is quite low.

I can think of many things, that would simplify label editing:

• Splitting a labels into connected components.
• Selecting connected components (to remove / copy / ... etc).
• Joining two labels into one.
• Split a label into multiple layers like known from Gimp / Photoshop.

Flood fill depending on different labels

We would like to use one label to mark the boundary of something and another label to mark the area inside by flood filling the first label.

Add prompt to indicate when the same label is re-used

Hello,

I have a feature request: It would be great if there is a prompt when a user accidentally types in the same label while annotating. For example, here labkit could suggest that nucleus_149 already exists.
Thank you for your time!

Button "Labkit Pixel Classification"

Hi,

I started to use Labkit on Fiji and I saw a video where you used the button "Labkit Pixel Classification" as classifier. How to configurate Labkit and Fiji to see the button "Labkit Pixel Classification" pls ?

Thank you for your help.

Scripting

Hi @maarzt,

I just trained a model successfully (it looks really great!) Is there a way for executing this model on 1000 images? Idealy via ImageJ macro as I have pre- and post-processing steps I would like to call.

Thanks a lot!

Cheers,
Robert

Multi view data isn't handled properly

1. Labkit only works with the first few but shows all.
2. If the affine transformation for every time frame, then there's an offset between image and segmentation.

Load czi resolution pyramid

Better error message for empty labeling when starting segmentation

Labkit runnning in the background

Hi @maarzt ,

I just hit an issue with labkit continuing to run in the background after closing it. Is it possible, that clicking "Cancel" while training doesn't work? Is it possible to kill the training thread when Labkit is closed?

Thanks!

Cheers,
Robert

Orthogonal view

Labeling persists through all slices

Hi,
First of all, thank you for creating and sharing such a useful tool.
I am trying to create segmentations on a 3D nuclear preparation and for some reason every time I label a nuclei, the paint remains through the whole stack even when the body I intended to label has already disappeared, leading to a bad segmentation. Besides that, I also find that once the classifier's trained it gives back a completely blank screen.
I have tried it with other preparations that are much more uniform and for some reason LabKit seems to understand where a label belongs and consequently, disappears along with the nuclei, like it should do.

Thank you!

Filter features per channel

Hi,

First of all Labkit is awesome.

Here's my issue : when I segment a big dataset with 4 channels, I export it to a xml/hdf5 format, thus I get all the channels in the dataset. However, I would like to make a classifier taking into account only a subset of the channels (say 1 3). Would this be possible to select the channels allowed to be used in the classifier ?

Also, since there a possibility to use several classifiers - this would be awesome to allow for a complete modularity for the different classifiers (different features / different labels / different resolution). Is this doable ?

Thanks

Nicolas

Wizard for training a classifier

The user has many options when training a classifier:

• Select a classification algorithms
• Select pixel features
• Select which labels to use for training
  Currently it's hard for the user to spot this options and to understand them.
  A better GUI is would be very useful/

After training the resulting segmentation could be used as a label.
Currently this functionality is very much hidden in the menu.

Reset orientation after rotating the image

Exporting binary images

Hi @maarzt ,

when exporting a binary segmentation result from LabKit, one cannot use the resulting image as binary image. One has to threshold it again in order to do so. Would it be possible to add a menu to LabKit "Show segmentation result as binary image in ImageJ"? I think all that is necessary is setting pixels to 0 or 255 and storing it as 8-bit unsigned byte image.

Thanks!

Cheers,
Robert

Use an elliptic brush

Currently there are two problems with the sphere brush:

• When drawing labels it's often annoying, that the brush goes to deep in the "z"-dimension. A brush that more flat would be much more useful.
• For anisotropic images the sphere brush produces elliptic results.

Add all the features provided by trainable segmentation

When exported label map is added to 3D ROI Manager, overlapping areas are counted as new labels

Hi,
Firstly, thank you for this resource, it is working really well for me. I am using it to generate ground-truth images for use with the StarDist Segmentation, and it is easy to use and has great functionality. I have come across a problem however, but I'm not sure if it is because I am doing something wrong.

I colour each nucleus in an image with a separate label, and some of my labels overlap in lab kit. In lab kit this is fine, the overlapping area belongs to both labels as it should. However when I export the label map into imageJ and add the image to the 3D Roi Manager, these overlapping areas are counted as new labels, which is not ideal. From 200 labels on lab kit, I have 258 in the Roi Manager. I'm worried that, as well as skewing the 3D Measure values, this may also be leading to errors in the training of my segmentation model with StarDist, if it too is erroneously attributing nuclear identity to these overlapping areas.

I was wondering how I could prevent this.

Thanks again, and look forward to hearing from you,

Michael Schwimmer

Make GPU selectable

Hi @maarzt ,

I'm working on a computer with 3 GPUs and test the GPU-preview. Is it possible to see and/or change which GPU is labkit running on?

Thanks!

Cheers,
Robert

Keep label colors when renaming labels

Importing label image with high label ids freezes labkit

Hi,

When using "Import Labeling" to import a label tif image that contains even a single object with a high object id (e.g. 5000), all intermediate ids (1...5000) will get allocated leading labkit freezing and Fiji becoming unresponsive. This situation happens frequently when curating segmentation results...

Example: mask_labkit_1000.tif.zip

Anisotroopic voxel size - sigma configuration

Hi @maarzt ,

I'm working with a dataset with voxel size 0.4x0.4x1.4 microns. Anisotropic voxels are quite common. Is it possible to configure sigmas for filters individually for XY and Z? Alternatively: Would it be possible to run filters in 2D slice by slice? That might make the whole processing much faster.

Thanks!

Cheers,
Robert

Example scripts/macros

Hey @maarzt ,

are there example scripts/macros available demonstrating how to apply a model to an image file? If yes, might it make sense to link it from the docs?

Thanks!

Cheers,
Robert

brush size in 3D

Hi @maarzt ,

I just hit an issue with a collaborator using labkit for annotating objects. Can you tell us if it is possible to change brush size in x/y and z independently? In some situations we would like to annotated 3D in other situations, we would like to annotated slice-by-slice.

Thanks!

Cheers,
Robert

Concepts for labeling in 3d

If you think of software like Blender it becomes clear, that editing a 3d volume is a very difficult task,
which requires versatile tools. But which functional, that we can provide, would be most useful for labeling tasks in biology?

Multiresolution performance issue

While drawing on high zoom level, the background images changes resolution and moves.

Export individual segmented time points

Hey @maarzt,

I'm working with a 4GB timelapse dataset. Annotation workes nicely. If I wanted to export just a single time point 3D stack to ImageJ, e.g. to test another plugin, would that be possible with labkit?

Thanks!

Cheers,
Robert

Slider for BDV Depth Axis

Overlapping labeling results in new class

When drawing a new label (e.g. class = 2) on top of an already labeled region (e.g. with class = 1) and saving the resulting labeling as tiff file, the overlapping part will appear with a new label (class = 3), which is surprising and hard to notice. Is this intended behaviour? I would expect that the new label should take precedence, i.e. the overlapping region should have class = 2.

PS: Thanks for the tool - it's really helpful!

Feature request: undo command

Hi everyone! First of all, I have recently started using LabKit and I LOVE it. Thank you for making such a useful robust tool. Would you consider implementing a Command+Z or other "Undo" button? I'm curating lots and lots of segmentations with multiple time points and it's painful to have to start from the last save when I make a mistake. I'm happy to user test if you decide this is something you want to pursue.

Default tool: Annotation

Hi @maarzt ,

would it be possible that the default tool when starting labkit is annotation? I think it would be very intuitive if a user could start annotating in Labkit right away.

Furthermore, could "Foreground" be selected by default? Me - the user opens a tool annotate "something" not "something else" ;-)

Thanks!

Cheers,
Robert

Mouse wheel in 2D: Zoom instead of rotation?

Hi @maarzt ,

in 2D mode, the mouse wheel drives the rotation of the image, could this be changed to zoom in/out? That would be awesome!

Thanks!

Cheers,
Robert

czi file: multiple layer support

• detect layers
• provide choice to chose one at the beginning
• solve labeling e.g. provide one labeling file for each layer

Provide a "Labkit" update site

Indicator for file saving process

It would be nice to have an indicator (e.g. popup window or progress bar) telling the user that the labeling is being saved to a file. Users are accidentally interrupting the saving process by force-quitting ImageJ because they don't know the saving process is still active.

ERror when starting labkit

Hi @maarzt ,

I just tested Labkit in Fiji (not the GPU-preview, the official version). When starting it from the menu, Labkit doesn't open up and shows this error instead:

[ERROR] Module threw error
java.lang.NoSuchMethodError: bdv.viewer.ViewerPanel.getDisplay()Lnet/imglib2/ui/InteractiveDisplayCanvasComponent;
	at net.imglib2.labkit.brush.LabelBrushController.<init>(LabelBrushController.java:64)
	at net.imglib2.labkit.BasicLabelingComponent.initBrushLayer(BasicLabelingComponent.java:97)
	at net.imglib2.labkit.BasicLabelingComponent.<init>(BasicLabelingComponent.java:53)
	at net.imglib2.labkit.SegmentationComponent.<init>(SegmentationComponent.java:50)
	at net.imglib2.labkit.LabkitFrame.initSegmentationComponent(LabkitFrame.java:62)
	at net.imglib2.labkit.LabkitFrame.<init>(LabkitFrame.java:52)
	at net.imglib2.labkit.LabkitFrame.show(LabkitFrame.java:46)
	at net.imglib2.labkit.LabkitFrame.showForImage(LabkitFrame.java:40)
	at net.imglib2.labkit.plugin.LabkitPlugin.run(LabkitPlugin.java:27)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Might there any other update site needed for running labkit?

Thanks for any hint!

Cheers,
Robert

Jump to label feature

It might be hard to find the labels, if only small regions in a huge image are labeled.

A functionality jump to the labeled region should be implemented.