shermp/kobo-uncaged

UNCaGED, for Kobo devices

kobo-uncaged's Issues

Initial testing

Starting an issue thread to gather early testing results.

@NiLuJe if you have any issues or comments, leave 'em here. Probably better than continuing the discussions in PR's

Web UI and NickelMenu early feedback request

Over the past week or so, I've been working on a UI overhaul of KU. Namely, I've ripped out all the fbink "UI", and replaced it with a new web based UI that is actually interactive.

I'm also ditching kfmon in favor of NickelMenu (sorry @NiLuJe , but I don't think this whole thing would work with kfmon).

And I'm ditching the whole "do stuff behind Nickel's back" approach. I think it adds a lot of complexity for little gain. Yes that means using the DB while Nickel is also using it, but it seems looking at sqlite docs, with the right connection settings, it should be pretty safe, although one might want to backup their DB before testing, just in case. FWIW, I haven't seen any DB corruption with the new code.

I think I'm about ready for some early feedback if anyone is willing to give it a go. NickelMenu is required to launch it. I'm currently using the following NickelMenu config:

menu_item :main :Kobo UNCaGED :cmd_spawn :/mnt/onboard/.adds/kobo-uncaged/nm-start-ku.sh
chain  :nickel_extras :web_browser
menu_item :main :Book Scan :nickel_misc :rescan_books_full
chain :nickel_misc :rescan_books_full

Although this could use some refinement. @geek1011 do you know if the rescan_books actions are synchronous or not? I'm not really sure. Sometimes tapping "Book Scan" once works, other times I need to tap it a second time. Also, the non-full variant doesn't appear to populate series?

Once the browser is launched, you will need to navigate to localhost:8181. I would recommend setting this to your home page, or add it as a favorite.

Please be aware, there is still plenty of jank. Lots of jank. First, the web UI is a retrofit. Second, I'm not experienced with writing web apps. Third, the Kobo browser is truly awful. Fourth, I suck at web design, which makes point three even worse.

Pinging @tmjwid as well, as this might solve your Wifi issues when it's release worthy. Mainly because I'm letting Nickel handle all the wifi stuff via the web browser.

My main areas of concern are it's still possible for KU to get "stuck", requiring killing the process(es). Also, there's currently nothing stopping multiple launches. Probably need to deal with that in the startup script, any ideas on that @NiLuJe ? Other than that, I want to try and reduce the jank. I won't say no to any help if there's any interest.

The new code is in the webui-nm branch. I'm afraid there's no fancy CI/CD setup with build artifacts 😭 On a sidenote, anyone interested in dockerizing koxtoolchain? I could use NickelTC I suppose, but KU doesn't need such an old toolchain...

EDIT: A couple of features of the new Web UI.

• Change settings without needing to edit the config file.
• Enter password via browser UI, if required instead if hardcoding them in the config file. Passwords are also now associated with their respective Calibre library UUID, instead of using a dumb list.

Cover image/thumbnail discussions

Opening this to follow on discussions from #3 & #9 regarding cover images and thumbnails.

Final testing

Kobo Elipsa

Hi @shermp , is this supposed to work with the Kobo Elipsa ? I extracted the zip file onto the root of Kobo directory but there's no option to Start Kobo Uncaged in Nickel Menu

Doesn't connect to Calibre

I successfully installed KU v0.5.0 with NickelMenu, and at startup the browser opens, but when I press the button 'Start' the 'Finished' dialog box show up and KU closes.
I also tried to enable 'Force Wifi On' in dev mode options.
I have Kobo Aura Edition 2, version 4.25.15875 and use Calibre 5.7.2 (but I have tried also with Calibre 4.X)

What could be the problem?

Building with Docker (and therefore GitHub actions

So, I really want to get some github actions going, but I kind of need a docker image to do it with.

The easiest option looks to be golang-crossbuild. My main concern is that this won't have any Kobo specific tweaks to the build system. How critical are the Kobo tweaks @NiLuJe ?

Another option might be to extend the koreader/kokobo image with Go, if I can get it working.

Or I could build my own koxtoolchain + Go docker image.

Another option may be NickelTC, although I would prefer newer compiler versions for building Go programs.

Any thoughts @NiLuJe or @pgaskin ?

Feature Request: Collections support

This is a fantastic piece of software. I would like to encourage you to add Collections support. I understand that you've previously been asked about that (on the Mobileread forum) and replied that it is a very low priority, so I just wanted to add my voice to those requesting the feature.

I considered trying to add it myself, but I don't know Go. However, if there is anything else that would help you add this feature (for instance, a description of the Kobo collections schema or the calibre/Kobo extended driver interface for assigning collections), please let me know and I will help however I can.

Not working?

Kobo-UNCaGED v0.3.0
Kobo Aura HD
Calibre 4.2

Just installed KFMon yesterday per these instructions https://www.mobileread.com/forums/showthread.php?t=314220 using this Plato-0.7.7_KOReader-v2019.09.3.zip

Getting this error:

failed to read metadata file: error reading metadata.calibre JSON: json: cannot unmarshal string into Go struct field KoboMetadata.application_id of type int

Error during initial connection

I managed to successfully send a file from Calibre (1.48, the last I can use), but then I had an error - I think getting metadata from my device.

Here is what Calibre said:

'NoneType' object has no attribute 'set'

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 86, in run
File "site-packages\calibre\gui2\device.py", line 490, in _books
File "site-packages\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.py", line 1288, in books
AttributeError: 'NoneType' object has no attribute 'set'

Feature request: allow loading apps wirelessly with Kobo UNCaGED

I was wondering if it would be possible to use Kobo UNCaGED's wireless connection to load/update apps on my Kobo?

If mu (superficial) understanding is correct, Kobo UNCaGED creates a (simulated) USB connection to the Kobo and then ferries files over by finding and connecting to Calibre? Would that mean that it would be possible to have Calibre (possibly with a new plugin) send over application files?

I personally have NickelMenu, KOReader, Plato, and Kobo UNCaGED (and thus NickelDBus) installed. For most of those it seems like updates have comprised of unzipping their releases into their .adds/ directory (and the on-time setup of their configuration file in .adds/nm), but perhaps NickelMenu is more complex with its use of a KoboRoot.tgz?

I have a decent amount of Linux development experience (albeit none on Kobo) and would be interested in helping out if this seems reasonable?

I would love to never have to find my USB cables and adapters again!

Kobo Libra 2: no UI, and log says "failed to get kobo info: unknown device"

Dear @shermp

Thanks for a very promising tool! I just bought a Kobo Libra 2, and sadly the UI is not showing. I get the message "Running precautionary library rescan", then just a window saying "Finished", and that's it.

I am using Kobo LIbra 2, Model N418, FCC ID: NOIKBN418, IC: 8857A-KBN418, with Firmware 4.21.19086 (2adb3f45ad, 1/20/22).
I am using Kobo-UNCaGED release 0.5.2.

I checked ku.log , attached.

It looks like

Kobo-UNCaGED/kobo-uncaged/device/device.go

Line 84 in 46c2309

if err = k.getKoboInfo(); err != nil {

is failing and thus exiting.
How can I help? I was looking for a debug mode that would print the return of kobo.DeviceByID obtained in

Kobo-UNCaGED/kobo-uncaged/device/device.go

Line 308 in 46c2309

if dev, ok := kobo.DeviceByID(id); ok {

, but no dice.

Happy to help debug this any way I can!

wpa_supplicant error on wifi enable.

Hi there,

I'm running latest KU with latest kfmon on a Glo HD with latest f/w (4.20.14622).
Whenever it autoclicks connect and initiates wifi enable, I get at wpa_supplicant failed to connect followed by WiFi did not enable (1). Aborting!

Here's the syslog.
Mar 15 19:14:32 UNCaGED: Mounting onboard
Mar 15 19:14:33 UNCaGED: Mounting SD card
Mar 15 19:14:33 UNCaGED: Enabling WiFi
Mar 15 19:14:35 wpa_supplicant[2894]: Successfully initialized wpa_supplicant
Mar 15 19:14:35 wpa_supplicant[2894]: rfkill: Cannot get wiphy information
Mar 15 19:14:41 UNCaGED: wpa_supplicant failed to connect
Mar 15 19:14:41 wpa_supplicant[2895]: eth0: CTRL-EVENT-TERMINATING
Mar 15 19:14:42 UNCaGED: WiFi did not enable (1). Aborting!

My kobo has the google analytics patch installed (redirect hostfile) and kfmon and that's it. Any other information I'll gladly supply.

Clara 2E support?

Hi all:

I'm a relatively new e-reader owner and trying to find my way concerning content management self-hosting etc... I've run into this info about the KU project and it sounds like an exciting fit for me. I've installed NickelMenu, NickelDBus and the latest KU and believe I've confirmed from ku.log that the Clara 2E is unsupported.

I don't have the skills to contribute code, but I do have the skills to test. Curious whether there's any planned support for the Clara 2E?

Thanks!

Error dialling Calibre

I am getting the error shown in the attached picture when trying to connect to Calibre using the "Auto Discover" option. I've also tried adding a manual connection (I wasn't sure if something specific has to go in the "Name" field when creating the connection—I just put "calibre" for the name). When I do that and tap the "Start" button, the screen flashes but then nothing happens.

I'm able to connect to the Calibre content server through the browser on this and other devices.

Device: Kobo Aura H2O Edition 2
Software version: 4.34.20097

Any suggestions?

Initial untesting

Untests #13

Musings around book addition

KU currently follows the conservative Calibre approach of letting Nickel import books into the DB, and then modifying metadata afterwards. This approach, while it works, is time consuming, and a battery hog.

I'm currently mulling over the idea of creating the book record(s) myself, bypassing Nickel's "Importing Content" stage. This would have the big advantage of doing everything in one step, removing a partition mount/dismount cycle, removing the need to use FBInk to attempt to detect the end of the import process etc.

The main downside would be the potential to "stuff it up", and would probably introduce more firmware compatibility constraints.

@NiLuJe and @geek1011, do you have any comments and/or potential concerns over taking this approach? I would probably add it as an alternative path, rather than a replacement.

Calibre-web connection?

I have both calibre on a desktop and calibre-web (via docker).
After initially not being able to connect to calibre-web, I started the content server on calibre.....and then the wireless service. It now seems to connect.

Is there an equivalent on calibre-web for this wireless service? what does it do?

Error Message after ejecting

Thank you for this very useful package.

I am using a Forma on FW4.21.15015
Error came up on reinstall after factory reset.
Manual install of KFMon failed so I installed KFMon using one click KOReader package which bundles KFMon & NickelMenu (OCP-KOReader-v2020.06.zip)

After reboot unzipped KoboUncaged-v0.4.1-full.zip to drive root.
Connection to calibre seems ok.
Transfer of files and Nickel post processing of books seems ok.

But Calibre emitted the error message below after ejecting kobo.
No error message using cable.

BTW I just noticed that nickel processes books differently when using this package.
It slots books into author named directories with kobo forma driver.
Where as with the wireless device driver the books are all in the drive directory.
Is this the expected behavior?

==================

calibre, version 4.19.0
ERROR: Unhandled exception: ControlError:Device responded with incorrect information

calibre 4.19 [64bit] embedded-python: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.16+
Windows: ('7', '6.1.7601', 'SP1', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 32, 0) && DeDRM (6, 7, 0) && Package KFX (from KFX Input) (1, 32, 0) && KePub Input (3, 2, 10) && KindleUnpack - The Plugin (0, 82, 1) && Kindle Collections (1, 7, 26) && KePub Metadata Writer (3, 2, 12) && KoboTouchExtended (3, 2, 12) && KePub Metadata Reader (3, 2, 9) && KePub Output (3, 2, 12) && KFX metadata reader (from KFX Input) (1, 32, 0) && KFX Input (1, 32, 0)
Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 350, in umount_device
File "site-packages\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.py", line 1459, in eject
File "site-packages\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.py", line 643, in _call_client
File "site-packages\calibre\devices\smart_device_app\driver.py", line 685, in _receive_from_client
ControlError: Device responded with incorrect information

Support for Kobo Sage and Libra 2

I posted on the MR forums, posting here as well just to cover all angles. Running KU on a Sage or Libra 2 results in the following log being generated. KU does not get to the point of checking for wifi or opening the browser to check for Calibre.

Apr 08 21:25:44 [Kobo-UNCaGED] Enabled loopback interface
Apr 08 21:25:44 [Kobo-UNCaGED] Starting Kobo UNCaGED
[Kobo-UNCaGED] 2022/04/08 21:25:44 Started Kobo-UNCaGED
[Kobo-UNCaGED] 2022/04/08 21:25:44 Creating KU object
[Kobo-UNCaGED] 2022/04/08 21:25:44 Getting Kobo Info
[Kobo-UNCaGED] 2022/04/08 21:25:44 New: failed to get kobo info: New: unknown device
[Kobo-UNCaGED] 2022/04/08 21:25:44 New: failed to get kobo info: New: unknown device
Apr 08 21:25:44 [Kobo-UNCaGED] Running precautionary library rescan
Apr 08 21:25:47 [Kobo-UNCaGED] Disabled loopback interface

Strategies for large libraries

The user bigwoof on MobileRead has run into issues using KU with a large book library, and it's brought to light that KU as released really is not very memory efficient. And that even when one tries to improve memory usage efficiency, holding the entire calibre metadata set in memory can be problematic.

I've been trying to think of strategies to deal with this, and these are the ideas I've come up with so far:

• Don't bother with calibre metadata. Just send Calibre whatever we have available in Nickel's DB. Simple to implement, probably the most efficient. Downside is not keeping the metadata.calibre file in sync with the calibre kobo driver.
• Store the metadata from calibre in some sort of file-based kv store. And maybe sync that store with metadata.calibre?
• Similar to above, but use an SQLite DB with proper columns to store metadata.
• Find a way of indexing/accessing JSON directly from file

I'm really open to all ideas.

Paging @NiLuJe and @pgaskin and @pazos for ideas.