willnilges/inkpath

A plugin for transcribing hand-drawn notes, diagrams, and tables onto xournalpp documents

inkpath's Issues

Skeletonization look-ahead multiple iterations

Note to self: The skeletonized images currently suck. I need to find a way to prevent it from eroding away connections between letters. It would be expensive to do contour detection multiple times on the skeletonized images. Maybe I can just “look ahead” N iterations or whatever. Currently, it checks if the NEXT iteration would result in a blank image. Maybe I should check if, like, the next three iterations would result in a blank image. That might improve the quality.

What should inkpath call a "stroke?"

Inkpath has the ability to separate splines into individual strokes. Would doing that be better or worse than trying to stay "true" to what the humans think each stroke is?

Add compilation instructions for other platforms

Currently, there's only compilation instructions for Debian. How do I compile this on Fedora, Arch, MacOS.., Windoze?

Add Preprocessing

I'm seeing hugely better results if I apply preprocessing with OpenCV before passing images to Inkpath. It's definitely not out of scope to add some CV libs and see if we can't have that implemented directly in the plugin.

TODO: Test on my 9-year-old T430s 😂

XML Parser error: Wrong count of points (2)

It's possible to save documents that have Inkpath objects on them, but loading does not work. It fails with XML Parser error: Wrong count of points (2). No idea why.

Going to cut open one of my saves and see if I can figure it out.

Add Tmux to Dockerfile

Extension can't find the module ipcvobj

Steps to reproduce

1. Install Xournal++ from Arch's official repository or build it from the aur (Xournalpp-git) package.
2. Download the latest release of this extension from the releases.
3. Extract the extension to /usr/share/xournalpp/plugins/.
4. Start the extension from the tabs or via the keybind.

Screenshot

Extra information
I've tried to copy the extension to the .config directory and also I've tried copying the module itself to /usr/lib/lua/5.4/ but I also encountered an error from that.

Thanks for the great extension.

Xournalpp can't find `inkpath.so`

For some reason, the way I've written the main.lua file means that installing the .so file into the plugin directory doesn't work. There's probably a way to fix this....

Bring back the command line utility

I want to bring back the CLI for this project, so that you don't need the Xournalpp API to use it, instead it'll just write to a file. It won't be great, but it'll probably help someone (even if that someone is me while I work on my CMPE-755 project)

Linking and installation issues

The app is currently dynamically linked, which makes it a total pain to install. I'd like to avoid having to tell people to install autotrace and friends, mostly because autotrace is currently a mess.

This app is also a mess. I have lua 5.4 installed... -_-

v1.0 Roadmap and beyond

v0.2

• Allow user to adjust scaling factor
• Insert strokes and select them, allowing user to drag and resize as needed (Also helps with strokes going off page?)
  • Default to upper left corner of current view (Can we find the visible coordinates in a window?)
• Pull in code from #5
• Open Xournalpp PR to extend API
• Write documentation: https://github.com/xournalpp/xournalpp.github.io/edit/site/docs/guide/plugins/plugins.md
• Come up with installation script for plugin? (Maybe ask Roland about this)

v1.0

• Add options for tweaking line thickness/bezier precision?
• Add save detection
• Loading dialog
  • Don't block when transcribing?
• Skirt bezierCurve math and pass splines directly to Xournalpp
• Further optimize data transfer code

v1.1

• Preprocessing
• Fix broken file formats (Such as JPEG and BMP)

v1.2

• Add color support

v1.3

• Multithread autotrace and/or spline parsing
• Additional contributions to Autotrace

Lua Version skew crashes Inkpath (and by extension Xournalpp)

Looks like skewing between Lua 5.4 and Lua 5.3 crashes the plugin. I noticed this on Arch, but it probably applies everywhere. You must build Xournalpp and Inkpath with the same version of Lua.

So, target Lua 5.4 on Arch. I'm also going to try targeting Lua 5.4 on Debian and Ubuntu, but god knows if I can do it successfully.

Determine stroke width through outline tracing

An important next step is to decide how the strokes and splines should be passed from Lua to the API function. I'm not quite happy how it is done now. Keep in mind that the API should be used by everyone who wants to add strokes and splines to a Xournal++ document from a Lua plugin. In particular assuming that points have no pressure and that the stroke width is 1.5 is unacceptable.

Suggestion for drawStroke (in final form):

stroke = {
x = {[1] = number, [2] = number, ...., [N] = number}
y = {[1] = number, [2] = number, ...., [N] = number}
pressure = {[1] = number, [2] = number, ...., [N] = number}
width = number
color = integer
fill = integer
tool = string
lineStyle = string
}

I'm not 100% sure whether it is better to have the x-coordinates, y-coordinates and pressure values separated, or to create a table of points, where each point is a table consisting of x, y-coordinates and pressure-value. The first approach is the one used by the Google IME API and other handwriting recognition software.

It is not necessary to implement all of that at once, but at least everything that is assigned explicitly (like the stroke width) should be implemented.

Originally posted by @rolandlo in xournalpp/xournalpp#3688 (comment)

This just gave me a (completely unrelated to the conversation) idea, for how to maybe use Potrace in order to deduce stroke width. It's pretty good at finding the outline of shapes, so I'm wondering if it'd be possible to somehow deduce the distance between opposing sides of a stroke to set the stroke width, then somehow trace the centerline. There's some OpenCV math out there that might make this possible. Further investigation is (obviously) required.

README is wrong

This uses autotrace to translate whiteboard markings into splines. From there, it applies a bezier curve to approximate strokes as a series of points, then passes them to the Xournal++ Lua API for rendering.

That's not how it works anymore. It passes its bezier data directly to Xournal++, and then Xournal++ does the rasterization.

not working (windows)

I probably don't know what I'm doing, sorry, but it doesn't work (windows)

Double check the error handling

For instance, what happens when we pass in an empty coordinate table? Does xournal crash?

Bitmaps don't work

Getting the same issue.

Thread 1 "xournalpp" received signal SIGFPE, Arithmetic exception.
0x00007f0ce031058f in compute_color_rgb (icolor=<optimized out>, boxp=<optimized out>,
    histogram=0x7f0cc68a2010, quantobj=<optimized out>) at src/median.c:349

Fuller stacktrace

#0  0x00007f0ce031058f in compute_color_rgb
    (icolor=<optimized out>, boxp=<optimized out>, histogram=0x7f0cc68a2010, quantobj=<optimized out>) at src/median.c:349
#1  select_colors_rgb (quantobj=<optimized out>, histogram=0x7f0cc68a2010)
    at src/median.c:381
#2  0x00007f0ce0310af6 in median_cut_pass1_rgb
    (ignoreColor=0x564956172dd0, image=0x5649561861d0, quantobj=0x56495624d440)
    at src/median.c:720
#3  quantize
    (image=0x5649561861d0, ncolors=<optimized out>, bgColor=0x564956172dd0, iQuant=0x7fff8ea648e8, exp=<optimized out>) at src/median.c:844
#4  0x00007f0ce030a6c4 in at_splines_new_full
    (bitmap=0x5649561861d0, opts=0x56495618d360, msg_func=<optimized out>, msg_data=<optimized out>, notify_progress=notify_progress@entry=0x0, progress_data=progress_data@entry=0x0, test_cancel=0x0, testcancel_data=0x0) at src/autotrace.c:262
#5  0x00007f0ce030a7d3 in at_splines_new
    (bitmap=<optimized out>, opts=<optimized out>, msg_func=<optimized out>, msg_data=<optimized out>) at src/autotrace.c:228
#6  0x00007f0ce18893a0 in transcribe_image (L=0x564955b29808) at src/lua_util.c:40

https://github.com/autotrace/autotrace/blob/4bb4d9e2e97932ea65edb7f2002a8146794a36eb/src/median.c#L349

Originally posted by @WillNilges in autotrace/autotrace#48 (comment)