ideonate/jupyter-innotater

Inline data annotator for Jupyter notebooks

jupyter-innotater's Issues

Changing the format of the tool.

Is there any option to format the view of the tool. Like to shift the label boxes below the image viewer. Since using the default format gives very little space for the image.

Support pandas DataFrame

Pandas is a common library used for data management. It would be very useful to support dataframes on top of numpy arrays.

JupyterLab support

I assume, based on the install instructions, that this project does not support JupyterLab? (also, trying to test the first example from the README doesn't yield any useful output; see below). Do you plan to introduce such support or is that outside of the scope of this project?

Running this code

import os
import numpy as np
import matplotlib.pyplot as plt
from jupyter_innotater import Innotater, ImageInnotation, BoundingBoxInnotation

img = plt.imread("roadrunner.png")  # any test image
targets = np.zeros((1, 4))
Innotater(ImageInnotation([img]), BoundingBoxInnotation(targets))

gives me the following

Innotater(children=(HBox(children=(VBox(children=(ImagePad(value=b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00…

but no widgets or images displayed. I just tested in a classic Notebook, and it seems to work fine there.

Javascript error box appears instead of an image

Hi! I'm trying to produce a few simple bounding boxes for a set of images. Unfortunately, I run into the following error using the basic example from the project readme:

Clicking the box expands the following message:

[Open Browser Console for more detailed log - Double click to close this message]
Failed to load model class 'InnotaterModel' from module 'jupyter-innotater'
Error: No version of module jupyter-innotater is registered
    at f.loadClass (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.083e6b37f2f7b2f04b5e.js?v=083e6b37f2f7b2f04b5e:1:74855)
    at f.loadModelClass (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:10721)
    at f._make_model (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:7517)
    at f.new_model (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:5137)
    at f.handle_comm_open (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:3894)
    at _handleCommOpen (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.083e6b37f2f7b2f04b5e.js?v=083e6b37f2f7b2f04b5e:1:73392)
    at b._handleCommOpen (http://localhost:8888/static/lab/jlab_core.da7da3bf6a92a54183c1.js?v=da7da3bf6a92a54183c1:2:994423)
    at async b._handleMessage (http://localhost:8888/static/lab/jlab_core.da7da3bf6a92a54183c1.js?v=da7da3bf6a92a54183c1:2:996413)

I'm trying to run jupyter-innotater inside Jupyter Lab, having installed it via conda install -c conda-forge jupyter_innotater.

Here's the output of jupyter --version, just in case. If more input is helpful to debug this issue, please let me know.

IPython          : 8.4.0
ipykernel        : 6.15.1
ipywidgets       : 8.0.1
jupyter_client   : 7.3.5
jupyter_core     : 4.11.1
jupyter_server   : 1.18.1
jupyterlab       : 3.4.5
nbclient         : 0.6.7
nbconvert        : 7.0.0
nbformat         : 5.4.0
notebook         : 6.4.12
qtconsole        : not installed
traitlets        : 5.3.0

Error displaying widget: model not found

Hi!

First of all, thanks for what seems like a great plugin!

I am having some issues with getting the innotater to work.

I am running a fresh local installation of JupyterLab, with only the jupyter-innotater installed, and when trying to run the example I get the output:

Error displaying widget: model not found

On further inspection of the JS-console I see the following errors:

manager.js:319 Uncaught (in promise) Error: Module jupyter-innotater, semver range ~0.2.1 is not registered as a widget module
    at C.loadClass (manager.js:319)
    at C.<anonymous> (manager-base.js:264)
    at l (manager-base.js:45)
    at Object.next (manager-base.js:26)
    at manager-base.js:20
    at new Promise (<anonymous>)
    at Rtm6.S (manager-base.js:16)
    at C.e._make_model (manager-base.js:258)
    at C.<anonymous> (manager-base.js:247)
    at l (manager-base.js:45)

utils.js:119 Error: Could not create a model.
    at n (utils.js:119)
    at async C._handleCommOpen (manager.js:62)
    at async g._handleCommOpen (default.js:985)
    at async g._handleMessage (default.js:1138)

What am I doing wrong? Thanks in advance for any help.

Format options for drawing bounding boxes

Depending on the image, it may be hard to see the bounding box. It would be nice to be able to set basic parameters for how the box is drawn (line width and color, and possible style (e.g. dashed, dotted, solid), should suffice). Here's an example where the bounding box around the roadrunner (in red) is hard to see.

Default multi-class bounding box selection doesn't update label on save

Use Case: A widget for annotating multiple Bounding Boxes within a single image, each of which can belong to one of many classes.

Problem: If you draw a bounding box, but leave the dropdown on its default value, the one-hot labelling of the bounding box will be all 0 (e.g. no label assigned). If you change the assigned class in the drop down to any other class and reselect the default (first) item, then the labelling will be recorded properly.

Example Innotater Call

# setup all the other variables, e.g. image list etc before this.
annotation_labels = ["default", "a", "b"]

innotater_args = { 
            "inputs": [
                ImageInnotation(image_files, path=image_path, width=image_width), 
                TextInnotation(image_files, multiline=False)
            ],
            "targets": [
                RepeatInnotation(
                    (BoundingBoxInnotation, inno_bboxes),
                    (MultiClassInnotation, inno_labels,
                        {
                            "name": "ROI Type",
                            "classes": annotation_labels,
                            "dropdown": True
                        }
                    ),
                    max_repeats=50,
                    min_repats=1
                )
            ]
}

Innotater(**innotater_args)

"Error: Module jupyter-innotater, semver range ~0.2.2 is not registered as a widget module"

I'm getting the above error when trying to do the example in the README.md. I'm using a jupyter notebook in a binder with the versions for ipywidgets, notebook, widgetsnbextension, jupyter_innotater and pillow from the requirements.txt (other package versions may differ because of additional package requirements, but those probably shouldn't cause the error...?)

Any ideas what might cause this?

Enable enlarge image in ImageInnotater

Currently the width and height property in the ImageInnotation subclass are effective only if the canvas size is smaller than source image.
I would like to enlarge source image if there are no problems.

In imagewidget.js

			if (wantwidth !== undefined && wantwidth > 0) {
				self.usewidth = wantwidth;
				self.canvas.setAttribute('width', wantwidth.toString());

                                // Disabling zoom if canvas width is larger than source image width.
                                // Therefore only shrinking is possible.
				if (wantwidth > self.imgel.width) {
					self.usewidth = self.imgel.width;
					self.zoom = 1.0;
				}
				else {
					self.usewidth = wantwidth;
					self.zoom = wantwidth / self.imgel.width;
				}
			}

not working in jupyter notebook

i tried using display, and without, @interact is working, and other wigdets

is it possible to select and label data in a text?

I am looking for a tool to use in jupyterlab where I could select and label text within a document. Is it possible with this tool or is there any intention to do so soon? thanks.
like this:

Import Failed

Limitation of the current Class inheritance

When adding the DataMixin class (PR: #14), the inheritence of most of the classes in the repo have not been set properly.
An failing example:

import ipywidgets
from jupyter_innotater.data import Innotation, TextInnotation, BoundingBoxInnotation

class ButtonInnotationExtended(Innotation):
    requires_data = False

    def __init__(self, icon, description, on_click=None, **kwargs):
        self.icon = icon
        self.description = description
        self.on_click_callback = on_click
        super().__init__(**kwargs)

    def on_click(self, callback, remove=False):
        self.button.on_click(callback, remove)

    def _create_widget(self):
        if not hasattr(self, "button"):
            self.button = ipywidgets.Button(description=self.description, layout=self.layout, icon=self.icon)
            if self.on_click_callback is not None:
                self.button.on_click(self.on_click_callback)
            return self.button
        return self.button

    def update_ui(self, uindex):
        pass

    def update_data(self, uindex):
        pass

Calling ButtonInnotationExtended(description='save', on_click=None, icon='save') would fail with the exception TypeError: object.__init__() takes exactly one argument (the instance to initialize).

After reading this very good blog posts: Python’s super() considered super! and Cooperative multiple-inheritance paradigm in python, it seems that there is a better way to propagate unknown arguments through the class inheritance tree.

RBG-formatted images?

In the documentation, it says that images passed directly as numpy arrays should be in RGB format. However, this doesn't seem to be the case. I've loaded an image in RGB format using matplotlib's imread and displayed it using imshow (which uses RGB format for 3-channel images), and here's the comparison with the image shown by Innotator:

Matplotlib

Innotator

import os
import numpy as np
import matplotlib.pyplot as plt
from jupyter_innotater import Innotater, ImageInnotation, BoundingBoxInnotation

# image is RGBA; drop alpha channel
img = plt.imread("roadrunner.png")[..., :-1]
plt.figure(figsize=(8,8))
plt.imshow(img)
plt.axis("off")

targets = np.zeros((1, 4))
Innotater(ImageInnotation([img]), BoundingBoxInnotation(targets))

Reset RepeatInnotation number of visual elements to min_repeats

When labelling one image to n > min_repeats elements, the following images to be labelled keep the same n rows (instead of displaying min_repeats).

Shortcut to trigger a magnifying glass when drawing a bounding box

For high quality images (4K), it is not always possible to display the full image and still see the features the user would like to find.
Something similar to this image:

Vague Error message

Exception: Do not call update_data on an input-only class

The Class of the object and its location (target) would make the error message simpler to read.

Change readme to indicate that for python 3.6+

or potentially add support for python 3.5
Looking into see if this is possible

mat image support, workaround

a HxW image didn't work for me, but a HxWx1 did. Its no big deal really.

this is what I'm using on my end to get around it:

if len(img.shape) == 2:
    img = img[..., None]

Possibility to move bbox edges/corners

In BoundingBoxInnotation once a bbox is created it's not possible to be edited. It would be nice if the edges and corners could be active and be moved for fine adjustment of existing bboxes.

Annotating time series?

Hi, I am wondering if functionality for annotating time series or signal data might be something that could be implemented in this project? The ability to iterate over a bunch of numpy arrays (ex. each array represents a snippet of ECG data containing N different rhythms to be labelled) would be especially helpful for my projects. Thanks for the great work!

Callback on other widget with the same repeat_index

Is it possible to build a reset row button which would revert every widget to its default value ?
Example:
From

To

pypng >= 0.0.20 Removed 3D support from_array

I found the error while I was running the bellow like code with jupyter-innotater==0.2.2.

import io

import png
import numpy as np

print(png.__version__, np.__version__)

npim = np.zeros((256, 256, 3), dtype=np.uint8)
npim[:, :, 0] = 255
npim[:, ::8, 1] = 255
npim[:, 1::8, 1] = 255
npim[:, 2::8, 1] = 255

pngbytes = io.BytesIO()
pngmode = 'L' # Greyscale
if len(npim.shape) == 3:
    if npim.shape[2] > 4:
        raise Exception("Image numpy array appears to have more than 4 channels")
    pngmode = ('L','LA','RGB','RGBA')[npim.shape[2]-1]
else:
    npim = np.expand_dims(npim, axis=-1) # Need a third axis for channel
   

pngim = png.from_array(npim, mode=pngmode) # Don't have BGR available so flipped to RGB above
pngim.write(pngbytes) if hasattr(pngim, 'write') else pngim.save(pngbytes) # PyPNG API due to change after v0.0.19
pixel_value = pngbytes.getvalue()
pngbytes.close()

png.__version__='0.20220715.0' np.__version__='1.23.4'

---------------------------------------------------------------------------
ProtocolError                             Traceback (most recent call last)
Cell In [11], line 20
     16 # h, w, ch = npim.shape
     17 # npim = npim.reshape(h, w * ch)
     19 pngim = png.from_array(npim, mode=pngmode) # Don't have BGR available so flipped to RGB above
---> 20 pngim.write(pngbytes) if hasattr(pngim, 'write') else pngim.save(pngbytes) # PyPNG API due to change after v0.0.19
     21 pixel_value = pngbytes.getvalue()
     22 pngbytes.close()

File /app/venv/lib/python3.9/site-packages/png.py:1320, in Image.write(self, file)
   1309 """Write the image to the open file object.
   1310 
   1311 See `.save()` if you have a filename.
   (...)
   1316 cannot be streamed again.
   1317 """
   1319 w = Writer(**self.info)
-> 1320 w.write(file, self.rows)

File /app/venv/lib/python3.9/site-packages/png.py:668, in Writer.write(self, outfile, rows)
    665     a = array(fmt, itertools.chain(*check_rows(rows)))
    666     return self.write_array(outfile, a)
--> 668 nrows = self.write_passes(outfile, check_rows(rows))
    669 if nrows != self.height:
    670     raise ProtocolError(
    671         "rows supplied (%d) does not match height (%d)" %
    672         (nrows, self.height))

File /app/venv/lib/python3.9/site-packages/png.py:703, in Writer.write_passes(self, outfile, rows)
    700 elif self.bitdepth == 16:
    701     rows = unpack_rows(rows)
--> 703 return self.write_packed(outfile, rows)

File /app/venv/lib/python3.9/site-packages/png.py:738, in Writer.write_packed(self, outfile, rows)
    735 # raise i scope out of the for loop. set to -1, because the for loop
    736 # sets i to 0 on the first pass
    737 i = -1
--> 738 for i, row in enumerate(rows):
    739     # Add "None" filter type.
    740     # Currently, it's essential that this filter type be used
    741     # for every scanline as
    742     # we do not mark the first row of a reduced pass image;
    743     # that means we could accidentally compute
    744     # the wrong filtered scanline if we used
    745     # "up", "average", or "paeth" on such a line.
    746     data.append(0)
    747     data.extend(row)

File /app/venv/lib/python3.9/site-packages/png.py:658, in Writer.write.<locals>.check_rows(rows)
    655     wrong_length = False
    656 if wrong_length:
    657     # Note: row numbers start at 0.
--> 658     raise ProtocolError(
    659         "Expected %d values but got %d values, in row %d" %
    660         (vpr, len(row), i))
    661 yield row

ProtocolError: ProtocolError: Expected 255 values but got 256 values, in row 0

I checked that latest pypng.from_array is unsupported 3D ndarray.

https://gitlab.com/drj11/pypng/#release-0020

So I made a simple patch code and check that it work.

print(f"{png.__version__=} {np.__version__=}")

npim = np.zeros((256, 256, 3), dtype=np.uint8)
npim[:, :, 0] = 255
npim[:, ::8, 1] = 255
npim[:, 1::8, 1] = 255
npim[:, 2::8, 1] = 255

pngbytes = io.BytesIO()
pngmode = 'L' # Greyscale
if len(npim.shape) == 3:
    if npim.shape[2] > 4:
        raise Exception("Image numpy array appears to have more than 4 channels")
    pngmode = ('L','LA','RGB','RGBA')[npim.shape[2]-1]
else:
    npim = np.expand_dims(npim, axis=-1) # Need a third axis for channel
    
h, w, ch = npim.shape
npim = npim.reshape(h, w * ch)
npim = np.ascontiguousarray(npim)

pngim = png.from_array(npim, mode=pngmode) # Don't have BGR available so flipped to RGB above
pngim.write(pngbytes) if hasattr(pngim, 'write') else pngim.save(pngbytes) # PyPNG API due to change after v0.0.19
pixel_value = pngbytes.getvalue()
pngbytes.close()

This error only occurs when opencv-python was not installed.

If you want to fix the error, I think that you need to fix the pypng version to 0.0.19 or fix the jupyter_innotater/data.py .

support voila for creating stand alone app

Support Voila

Thanks for releasing this inspiring project!

I'm trying to use voila[0] to turn jupyter-innotater into a stand alone app. Currently I'm facing a range of error messages in the browser console that indicate that innotater is currently
not compatible with voila.

I believe fixing this has huge benefits especially when non technical people are involved into the labeling process.

Voila [0] is popular project that allows to turn interactive jupyter notebooks into standalone apps just by serving them with voila example.ipynb.

Reproducing

[tool.poetry.dependencies]
python = "^3.7"
jupyter_innotater = { path = "./jupyter-innotater/jupyter-innotater" }
jupyter = "^1.0.0"
jupyterlab = "^1.1.4"
nbconvert = ">=5.5"
ipywidgets = "^7.5.1"
pandas = "^0.25.1"
opencv-python = "^4.1.1"
voila = "^0.1.17"
[tool.poetry.dev-dependencies]

Ensure that current setup is properly configured by running example from
voila repo.

voila notebooks/interactive.ipynb

Running the innotator example with the same setup

voila jupyter-innotater/Example/FlatImages.ipynb

doesn't work:

browser

The following voila discussions might be helpful to track down why innotator isn't compatible with
voila.

• voila-dashboards/voila#46
• voila-dashboards/voila#85

[0] https://github.com/voila-dashboards/voila

Done button

It would be nice to have a Done button widget that could be included in the layout and pressed to, e.g., call IPython.display.clear_output(). This would just make the notebook look nicer after you're finished with annotations.

Any tips on making it work on Google Colab?

Hi there,

I really value your tool, thanks for making it. I've used it on jupyter notebook, and it works perfectly for me.
A few students I work with use the colab notebook, and for them the innotator widgets don't display (at all).

The chrome developer console shows an error:
"Error has occurred while trying to update output."
"Error: Widgets module jupyter-innotater is not supported"

I'm not sure if you can think of an easy work around for this, it would be helpful.
Other ipywidgets do work fine.

I copied one of your examples into a colab notebook to reproduce the issue in case it helps.
https://colab.research.google.com/drive/1ANFSGUyX3K_6DaZJCgZWPHGYyGOnOb0T?usp=sharing

On a potentially related note, I noticed when I installed jupyter_innotator on the notebook that I need to install the package both on the kernel env and on the notebook env (I use a single notebook for multiple kernels, so they are actually in different python environments). This makes some sense, because there is both browser and python logic involved. Before I did that, I found a similar error when I only installed the package on the kernel environment, and not on the notebook one.

Difficulty in changing code to annotate points

Hello,

I have been using your innotator for a while and it is great for me to annotate some data on my remote server.
I have changed some code in one of the .js files but I didn't see any changes in the widget.
Could you point me to a better direction?

Thanks

Any intent to create an annotator for image segmentation? Polygonal or Point-based

I tried another library called ipyannotations but it's currently broken. I saw this lib, jupyter-innotater, but it doesn't seem to support segmentation annotation.