Import failure when torrent name contains non-ascii character

[laharah]

Import failure when torrent name contains non-ascii character

[Laharah]

hmm, is there any way you could find out the name of the torrent file it fails on? It seems to have a non-standard encoding in the filename. If you don't think you can find out, I can get you a debug version of the plugin to try and capture it.

[Laharah]

I've added some error handling and logging, so at least the rest of your other torrents should keep importing. It'll also try opening the file without checking the encoding at all, which may still run into issues, but it might work.

You can download the new version here:
https://github.com/Laharah/deluge-uTorrentImport/releases/latest

At any rate, if you get a line that says "Unknown encoding in filename..." send the line to me and I might be able to work out the kink.

[wangqr]

The new version still does not load other torrents.
Log:

(deluge:8144): libglade-WARNING **: unknown attribute `swapped' for <signal>.

(deluge:8144): libglade-WARNING **: unknown attribute `swapped' for <signal>.
C:\Program Files (x86)\Deluge\library.zip\twisted\internet\_glibbase.py:375: GtkWarning: gdkdrawable-win32.c:2013 drawable is not a pixmap or window
Unhandled error in Deferred:


Traceback (most recent call last):
  File "twisted/internet/defer.py", line 434, in errback

  File "twisted/internet/defer.py", line 501, in _startRunCallbacks

  File "twisted/internet/defer.py", line 587, in _runCallbacks

  File "twisted/internet/defer.py", line 1241, in gotResult

--- <exception caught here> ---
  File "twisted/internet/defer.py", line 1183, in _inlineCallbacks

  File "twisted/python/failure.py", line 389, in throwExceptionIntoGenerator

  File "c:\users\wangqr\appdata\roaming\deluge\plugins\utorrentimport-2.3.3-py2.7.egg\utorrentimport\gtkui.py", line 123, in on_import_button_clicked

  File "twisted/internet/defer.py", line 1185, in _inlineCallbacks

  File "c:\users\wangqr\appdata\roaming\deluge\plugins\utorrentimport-2.3.3-py2.7.egg\utorrentimport\core.py", line 280, in begin_import

  File "c:\users\wangqr\appdata\roaming\deluge\plugins\utorrentimport-2.3.3-py2.7.egg\utorrentimport\core.py", line 316, in _import_torrent

  File "c:\users\wangqr\appdata\roaming\deluge\plugins\utorrentimport-2.3.3-py2.7.egg\utorrentimport\common.py", line 70, in error

exceptions.UnicodeDecodeError: 'utf8' codec can't decode bytes in position 118-119: invalid continuation byte

[Laharah]

Sorry about that, without knowing exactly what encoding is messing everything up I'm firing a bit blind. Give the newest release a try:
https://github.com/Laharah/deluge-uTorrentImport/releases/latest

[wangqr]

I haven't tried the 2.3.4 version yet. But I added some print lines in your 2.3.3 release:

diff --git a/utorrentimport/core.py b/utorrentimport/core.py
index 105c9ec..8f7cb87 100644
--- a/utorrentimport/core.py
+++ b/utorrentimport/core.py
@@ -298,6 +298,8 @@ class Core(CorePluginBase):
         """handles importing of a single torrent. Same arguments as `begin_import`"""

         try:
+            print(repr(torrent))
+            print(torrent)
             with open(unicode(torrent, 'utf-8'), 'rb') as f:
                 filedump = base64.encodestring(f.read())
         except IOError:

The last file name it printed out before the traceback is:

'C:\\Users\\wangqr\\AppData\\Roaming\\uTorrent\\[VCB-Studio] Phantom \xef\xbd\x9eRequiem for the Phantom\xef\xbd?[Ma10p_1080p].1.torrent'
C:\Users\wangqr\AppData\Roaming\uTorrent\[VCB-Studio] Phantom ???Requiem for the Phantom???[Ma10p_1080p].1.torrent

The original torrent is downloaded from:
https://dl.dmhy.org/2016/12/18/8cfc5a6e9cd1577396305991c5625f3f096fc3ba.torrent

[Laharah]

yeah, that looks like the unicode full tilda, but it seems to be malformed at the end for some reason. It's possible that uTorrent has some way of truncating strange encodings I'm not aware of that's messing up the way it's recorded in the resume data. Add to that window's weird handling of non latin-1 filenames and it gets really hard to find the problem. My patch should output the raw hex of the filename, which may or may not be helpful.

[wangqr]

Sorry but the log of the new version seems to contain some filenames which I would not like to make public. I have sent you an email with the log.

[Laharah]

got em, I'll look into a workaround for these torrents. Did eveything else import cleanly at least?

[wangqr]

Other torrents seems to be fine so far. The hash checking process is still running.

[wangqr]

There is one torrent failed the hash checking. I have moved the file and relocated it in uTorrent before. I think the renaming error line in the log may be relevant. It isn't a big issue since I can manually relocate it again in deluge. Maybe I will post more info later.

[Laharah]

Well, that's good at least, the rename error could be the same kind of error in another location.

Looking at those filenames, they seem to all be slightly broken utf-8 kanji and other asian codepoint filenames, however they do all decode cleanly into Latin-1 encoding, though they look more like gibberish that way. I suspect that utorrent coerces them into latin-1 at some point and drops the ordinals that can't be translated.

I'm not sure if windows will be able to find the files if I submit the filenames in the latin-1 gibberish, but it might be worth a shot. There's a beta release up that you could try just to see if windows will load them:
https://github.com/Laharah/deluge-uTorrentImport/releases/tag/2.3.5

[wangqr]

ver 2.3.5 is still skipping those files.

I opened the resume.dat in a hex editor and it seems that file names are correct in this file. Maybe something went wrong during parsing this file?
(b'\x9e\x20' becomes '0x3f' in the log)

[Laharah]

Weird, and certainly possible, I'll look through the code more deeply when I get home.

[Laharah]

well, this is a pretty weird problem. My code doesn't touch those filenames after it gets them straight from the decoding. The issue would have to lie somewhere in deluge's bdecode function.

looking at that hex dump, the decoding also eats the space after the 0x9E and changes them to a single 0x3f. Looking at deluge's bdecode string function, I can't see anything that should be causing that to happen. Would you by any chance be willing to email me your resume.dat file? I'm interested to see where exactly these strings are getting mangled.

[wangqr]

The file name goes wrong after this line:

deluge-uTorrentImport/utorrentimport/core.py

Line 276 in 443d7cc

torrent = os.path.abspath(

os.path.join worked well but os.path.abspath changed some non-ascii chars to question mark.

Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> a = os.path.join('C:\\', '\xef\xbd\x9e.dat')
>>> a
'C:\\\xef\xbd\x9e.dat'
>>> b = os.path.abspath(a)
>>> b
'C:\\\xef\xbd?dat'

I found a python issue here: https://bugs.python.org/issue13247

[Laharah]

well, I can't replicate this bug at all on my windows 10 machine, the example you gave doesn't mangle the final codepoint. However, looking at that python issue, it seems that if the path is already in unicode before it hits the os module, it should be okay. I've updated the code again. Would you be willing to give it a test for me?

https://github.com/Laharah/deluge-uTorrentImport/releases/tag/2.3.6

Also, what version of windows are you on? I might be able to spin up a vm for better testing.

[wangqr]

Thanks for the fix. Ver 2.3.6 works fine with no UnicodeDecodeError any more. Should I close this issue here?

I am using Windows 10 Education Build 15063 with English as system display language. The original install image (iso file) is a Simplified Chinese version of Win10 Anniversary.

---

The following error still exist. But this torrent's name contains only ascii chars so I think this is another issue.

[ERROR   ] 00:24:12 torrent:1058 Attempting to rename a folder with an invalid folder name:

[Laharah]

yeah, that error seems to be coming from deluge itself. I imagine it's because of a strange folder structure coming from utorrent. Deluge doesn't like torrents with multiple files that don't share a folder, and it looks like the plugin is trying to get rid of the root folder of the torrent. Feel free to open another issue with some more info about the file structure of the torrent that's failing, I'll close this one in a minute when I land these changes in the master branch.

Glad the main problem has been fixed though. Thanks so much for your work debugging this, without being able to replicate the issue, I'm not sure I would have found that abspath bug.