willnilges/inkpath

A plugin for transcribing hand-drawn notes, diagrams, and tables onto xournalpp documents

Inkpath

If you're anything like me, you're a huge nerd engineering major who enjoys working out problems both on whiteboards and digitally. You might also have six million photos of your whiteboard work trapped on your phone or in your Google Drive with no good way to easily group them in with your other notes.

Inkpath is a project designed to crunch those whiteboard photos into easily editable Xournal++ note files so that you can drop your whiteboard scrawlings directly into your lecture notes. Convenient!

The project consists of a lua script and a shared object library written in C++ statically linked with some OpenCV utils. The project now uses OpenCV in place of Autotrace to perform an Otsu Threshold. The thresholded image is then inverted, and skeletonized, producing the centerline of each individual object in the image (which at this point should be just markings). The resulting image is then scanned for contours. These contours are pushed onto the lua stack, and passed to the Xournal++ API. Unlike the previous implementation, this operates purely on rasters.

Installation and Usage

Inkpath is packaged as a statically-linked .so file coupled with a Lua script, so you should be able to download the release from the releases page and have it Just Work™.

As of 2022-09-24, the API changes in Xournalpp are merged, but have not made it into the package managers. You will probably need to build xournalpp from source in order to get them.

Inkpath is coming to a package manager near you soon™!

Manual Installation

Arch

# Install dependencies
pacman -S \
cmake gtk3 base-devel libxml2 portaudio libsndfile \
poppler-glib texlive-bin texlive-pictures gettext libzip lua53 lua53-lgi \
gtksourceview4 wget unzip git tmux

# Build openCV
./HACKING/build-opencv.sh

Debian

# Install dependencies
apt -y install \
make liblua5.3-dev build-essential pkg-config libglib2.0-dev libpng-dev \
cmake libgtk-3-dev libpoppler-glib-dev portaudio19-dev libsndfile-dev \
dvipng texlive libxml2-dev libzip-dev librsvg2-dev gettext lua-lgi \
libgtksourceview-4-dev git gdb x11-apps wget unzip
./HACKING/build-opencv.sh $NUMBER_OF_CORES_YOU_HAVE # You'll need sudo to install this
./HACKING/build-xopp.sh $NUMBER_OF_CORES_YOU_HAVE # You'll probably need sudo to install this too
make install

inkpath's Issues

Bring back the command line utility

I want to bring back the CLI for this project, so that you don't need the Xournalpp API to use it, instead it'll just write to a file. It won't be great, but it'll probably help someone (even if that someone is me while I work on my CMPE-755 project)

Lua Version skew crashes Inkpath (and by extension Xournalpp)

Looks like skewing between Lua 5.4 and Lua 5.3 crashes the plugin. I noticed this on Arch, but it probably applies everywhere. You must build Xournalpp and Inkpath with the same version of Lua.

So, target Lua 5.4 on Arch. I'm also going to try targeting Lua 5.4 on Debian and Ubuntu, but god knows if I can do it successfully.

Extension can't find the module ipcvobj

Steps to reproduce

1. Install Xournal++ from Arch's official repository or build it from the aur (Xournalpp-git) package.
2. Download the latest release of this extension from the releases.
3. Extract the extension to /usr/share/xournalpp/plugins/.
4. Start the extension from the tabs or via the keybind.

Screenshot

Extra information
I've tried to copy the extension to the .config directory and also I've tried copying the module itself to /usr/lib/lua/5.4/ but I also encountered an error from that.

Thanks for the great extension.

Add compilation instructions for other platforms

Currently, there's only compilation instructions for Debian. How do I compile this on Fedora, Arch, MacOS.., Windoze?

Add Preprocessing

I'm seeing hugely better results if I apply preprocessing with OpenCV before passing images to Inkpath. It's definitely not out of scope to add some CV libs and see if we can't have that implemented directly in the plugin.

TODO: Test on my 9-year-old T430s 😂

Xournalpp can't find `inkpath.so`

For some reason, the way I've written the main.lua file means that installing the .so file into the plugin directory doesn't work. There's probably a way to fix this....

What should inkpath call a "stroke?"

Inkpath has the ability to separate splines into individual strokes. Would doing that be better or worse than trying to stay "true" to what the humans think each stroke is?

Bitmaps don't work

Getting the same issue.

Thread 1 "xournalpp" received signal SIGFPE, Arithmetic exception.
0x00007f0ce031058f in compute_color_rgb (icolor=<optimized out>, boxp=<optimized out>,
    histogram=0x7f0cc68a2010, quantobj=<optimized out>) at src/median.c:349

Fuller stacktrace

#0  0x00007f0ce031058f in compute_color_rgb
    (icolor=<optimized out>, boxp=<optimized out>, histogram=0x7f0cc68a2010, quantobj=<optimized out>) at src/median.c:349
#1  select_colors_rgb (quantobj=<optimized out>, histogram=0x7f0cc68a2010)
    at src/median.c:381
#2  0x00007f0ce0310af6 in median_cut_pass1_rgb
    (ignoreColor=0x564956172dd0, image=0x5649561861d0, quantobj=0x56495624d440)
    at src/median.c:720
#3  quantize
    (image=0x5649561861d0, ncolors=<optimized out>, bgColor=0x564956172dd0, iQuant=0x7fff8ea648e8, exp=<optimized out>) at src/median.c:844
#4  0x00007f0ce030a6c4 in at_splines_new_full
    (bitmap=0x5649561861d0, opts=0x56495618d360, msg_func=<optimized out>, msg_data=<optimized out>, notify_progress=notify_progress@entry=0x0, progress_data=progress_data@entry=0x0, test_cancel=0x0, testcancel_data=0x0) at src/autotrace.c:262
#5  0x00007f0ce030a7d3 in at_splines_new
    (bitmap=<optimized out>, opts=<optimized out>, msg_func=<optimized out>, msg_data=<optimized out>) at src/autotrace.c:228
#6  0x00007f0ce18893a0 in transcribe_image (L=0x564955b29808) at src/lua_util.c:40

https://github.com/autotrace/autotrace/blob/4bb4d9e2e97932ea65edb7f2002a8146794a36eb/src/median.c#L349

Originally posted by @WillNilges in autotrace/autotrace#48 (comment)

Linking and installation issues

The app is currently dynamically linked, which makes it a total pain to install. I'd like to avoid having to tell people to install autotrace and friends, mostly because autotrace is currently a mess.

This app is also a mess. I have lua 5.4 installed... -_-

not working (windows)

I probably don't know what I'm doing, sorry, but it doesn't work (windows)

v1.0 Roadmap and beyond

v0.2

• Allow user to adjust scaling factor
• Insert strokes and select them, allowing user to drag and resize as needed (Also helps with strokes going off page?)
  • Default to upper left corner of current view (Can we find the visible coordinates in a window?)
• Pull in code from #5
• Open Xournalpp PR to extend API
• Write documentation: https://github.com/xournalpp/xournalpp.github.io/edit/site/docs/guide/plugins/plugins.md
• Come up with installation script for plugin? (Maybe ask Roland about this)

v1.0

• Add options for tweaking line thickness/bezier precision?
• Add save detection
• Loading dialog
  • Don't block when transcribing?
• Skirt bezierCurve math and pass splines directly to Xournalpp
• Further optimize data transfer code

v1.1

• Preprocessing
• Fix broken file formats (Such as JPEG and BMP)

v1.2

• Add color support

v1.3

• Multithread autotrace and/or spline parsing
• Additional contributions to Autotrace

XML Parser error: Wrong count of points (2)

It's possible to save documents that have Inkpath objects on them, but loading does not work. It fails with XML Parser error: Wrong count of points (2). No idea why.

Going to cut open one of my saves and see if I can figure it out.

Double check the error handling

For instance, what happens when we pass in an empty coordinate table? Does xournal crash?

README is wrong

This uses autotrace to translate whiteboard markings into splines. From there, it applies a bezier curve to approximate strokes as a series of points, then passes them to the Xournal++ Lua API for rendering.

That's not how it works anymore. It passes its bezier data directly to Xournal++, and then Xournal++ does the rasterization.

Add Tmux to Dockerfile

Determine stroke width through outline tracing

An important next step is to decide how the strokes and splines should be passed from Lua to the API function. I'm not quite happy how it is done now. Keep in mind that the API should be used by everyone who wants to add strokes and splines to a Xournal++ document from a Lua plugin. In particular assuming that points have no pressure and that the stroke width is 1.5 is unacceptable.

Suggestion for drawStroke (in final form):

stroke = {
x = {[1] = number, [2] = number, ...., [N] = number}
y = {[1] = number, [2] = number, ...., [N] = number}
pressure = {[1] = number, [2] = number, ...., [N] = number}
width = number
color = integer
fill = integer
tool = string
lineStyle = string
}

I'm not 100% sure whether it is better to have the x-coordinates, y-coordinates and pressure values separated, or to create a table of points, where each point is a table consisting of x, y-coordinates and pressure-value. The first approach is the one used by the Google IME API and other handwriting recognition software.

It is not necessary to implement all of that at once, but at least everything that is assigned explicitly (like the stroke width) should be implemented.

Originally posted by @rolandlo in xournalpp/xournalpp#3688 (comment)

This just gave me a (completely unrelated to the conversation) idea, for how to maybe use Potrace in order to deduce stroke width. It's pretty good at finding the outline of shapes, so I'm wondering if it'd be possible to somehow deduce the distance between opposing sides of a stroke to set the stroke width, then somehow trace the centerline. There's some OpenCV math out there that might make this possible. Further investigation is (obviously) required.

Skeletonization look-ahead multiple iterations

Note to self: The skeletonized images currently suck. I need to find a way to prevent it from eroding away connections between letters. It would be expensive to do contour detection multiple times on the skeletonized images. Maybe I can just “look ahead” N iterations or whatever. Currently, it checks if the NEXT iteration would result in a blank image. Maybe I should check if, like, the next three iterations would result in a blank image. That might improve the quality.