from ninja6o4: Traceback (most recent call last): File “dupRepor

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Program crashing when processing emails.,about handyguysoftware/dupreport

Comments (34)

ninja6o4 commented on May 25, 2024

OK grabbed latest beta:
Traceback (most recent call last):
File "dupReport.py", line 213, in
globs.inServer.processMessage(nxtMsg)
File "C:\dupReport-2.1.0_Beta\dremail.py", line 307, in processMessage
decode1 = email.header.decode_header(msg['Message-Id'])[0][0]
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36-32\lib\email\header.py", line 80, in decode_header
if not ecre.search(header):
TypeError: expected string or bytes-like object

Attached is pruned log, last entry (same email, it looks like.)
dupReport.log

from dupreport.

HandyGuySoftware commented on May 25, 2024

OK, as I suspected, for some reason dR is not getting a message ID from the email, so everything else is crashing behind it. Highly unusual. The easy fix is to add something like

if (no header)
    skip message

However, before I do that, would it be possible to post the entire email that's causing the error, including the headers? I want to see if there is a message format I need to code for. Also, did you say this worked on other mail servers or was that the original error you were talking about?

from dupreport.

ninja6o4 commented on May 25, 2024

I did also try it on a cPanel-based mail server and it worked without any issues (although it's an empty mailbox right now.)

I will look for that specific email later tonight and pm you privately the contents.

from dupreport.

HandyGuySoftware commented on May 25, 2024

OK, thanks. I've coded up a potential fix. I can post that now and see if it solves the problem, or wait for your email and do some additional research to see if more changes are needed. Your choice.

from dupreport.

HandyGuySoftware commented on May 25, 2024

Uploaded a possible fix to the Beta branch. Please have a look and see if it addresses your issue.

from dupreport.

dcurrey commented on May 25, 2024

Not sure if this is related to above but I was able to reproduce the error above by having an email message with a blank subject.

This was done with the most current code.

  File "./dupReport.py", line 214, in <module>
    globs.inServer.processMessage(nxtMsg)
  File "/root/dupReport-2.1.0_Beta/dremail.py", line 337, in processMessage
    decode = email.header.decode_header(msg['Subject'])[0]
  File "/usr/lib/python3.5/email/header.py", line 80, in decode_header
    if not ecre.search(header):

from dupreport.

HandyGuySoftware commented on May 25, 2024

@dcurrey , interesting, because a message with a blank subject should be simply rejected as "not a message of interest." Can you post the error message you're getting and the last 20 lines of the log file?

from dupreport.

dcurrey commented on May 25, 2024

Think I am on to something here looking at the log and the messageid it looks like dupreport is trying to add the blank subject message.

Database.searchForMessage(<20171228112123.Horde.p77A5h25wJZ9ESmi-G1jyFY@hordetest.mydomain.com>)
Database.execSqlStmt(): Executing SQL command.
Message ID [<20171228112123.Horde.p77A5h25wJZ9ESmi-G1jyFY@hordetest.mydomain.com>] does not exist. Adding to DB

As you stated it shouldn't

Here is the full message header notice something missing

Return-Path: <[email protected]>
Delivered-To: [email protected]
Received: from localhost (localhost.localdomain [127.0.0.1])
	by beta.mydomain.com (Postfix) with ESMTP id 92ED84543EEB
	for <[email protected]>; Thu, 28 Dec 2017 11:21:23 -0500 (EST)
X-Virus-Scanned: Debian amavisd-new at beta.mydomain.com
X-Spam-Flag: NO
X-Spam-Score: 6.956
X-Spam-Level: ******
X-Spam-Status: No, score=6.956 tagged_above=3 required=999
	tests=[HELO_DYNAMIC_IPADDR=3.243, MISSING_SUBJECT=1.767,
	RDNS_DYNAMIC=0.363, SPF_FAIL=0.919, TO_EQ_FM_DIRECT_MX=2.499,
	TO_EQ_FM_DOM_SPF_FAIL=0.001, TO_EQ_FM_SPF_FAIL=0.001, TXREP=-1.837]
	autolearn=no autolearn_force=no
Received: from beta.mydomain.com ([127.0.0.1])
	by localhost (beta.mydomain.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id oF2DDRqrgkR2 for <[email protected]>;
	Thu, 28 Dec 2017 11:21:23 -0500 (EST)
Received: by beta.mydomain.com (Postfix, from userid 5002)
	id 10BEF4543ED3; Thu, 28 Dec 2017 11:21:23 -0500 (EST)
Received: from mh-esr1-xxx-xxx-xxx-xxx.,myisp.net (mh-esr1-xxx-xxx-xxx-xxx.myisp.net
 [xxx.xxx.xxx.xxx]) by hordetest.mydomain.com (Horde Framework) with HTTPS;
 Thu, 28 Dec 2017 11:21:23 -0500
Date: Thu, 28 Dec 2017 11:21:23 -0500
Message-ID: <20171228112123.Horde.p77A5h25wJZ9ESmi-G1jyFY@hordetest.mydomain.com>
From: [email protected]
To: [email protected]
User-Agent: Horde Application Framework 5
Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes
MIME-Version: 1.0
Content-Disposition: inline

some text

from dupreport.

ninja6o4 commented on May 25, 2024

OK updated.. it runs for a long time (log is too large to read in n++), but it ends with this error:

Traceback (most recent call last):
File "dupReport.py", line 214, in
globs.inServer.processMessage(nxtMsg)
File "C:\dupReport-2.1.0_Beta\dremail.py", line 443, in processMessage
globs.log.write(3, 'endTimeStamp=[{}] beginTimeStamp=[{}]'.format(drdatetime.fromTimestamp(dateParts['endTimestamp']), drdatetime.fromTimestamp(dateParts['beginTimestamp'])))
File "C:\dupReport-2.1.0_Beta\drdatetime.py", line 134, in fromTimestamp
dt = datetime.datetime.fromtimestamp(float(ts))
TypeError: float() argument must be a string or a number, not 'NoneType'

My .rc file is unchanged from default other than the server access/credentials:

[main]
version = 2.1.0
dbpath = C:\dupReport-2.1.0_Beta
logpath = C:\dupReport-2.1.0_Beta
verbose = 1
logappend = false
subjectregex = ^Duplicati Backup report for
srcregex = \w*
destregex = \w*
srcdestdelimiter = -
dateformat = MM/DD/YYYY
timeformat = HH:MM:SS
warnoncollect = false
applyutcoffset = false
show24hourtime = true
purgedb = false

Another question - I tried running it again without verbose logging so I could read the .log, and I noticed that more often than not, it says "Message ID [[email protected]] does not exist. Adding to DB"

However, since I've already run it once before, should the majority (if not all) of these existing messages now be in the DB, even if it crashes out?

from dupreport.

dcurrey commented on May 25, 2024

Quick and dirty fix for no subject problem

Line 336 dremail.py

        # get Subject
        if not msg['Subject']:
            msg['Subject'] = '[No Subject]'
        decode = email.header.decode_header(msg['Subject'])[0]

from dupreport.

HandyGuySoftware commented on May 25, 2024

@ninja6o4 , the program doesn't commit the database until all the emails are parsed, so if it crashed in the middle it won't register those emails in the db. They will be scanned again on the next run.

@dcurrey , I'm thinking an easier fix is to just skip the message entirely if no subject, since it won't be from a Duplicati backup session. dR won't be able to do anything with it anyway. Continuing to process, even with a bogus subject will just lead to problems down the line. Thoughts?

from dupreport.

dcurrey commented on May 25, 2024

Hence the quick and dirty fix. :)

It does look like the check is just a few lines down in the dremail.py

So just do a return at the if statement instead of bogus subject

if not msg['Subject']:
     globs.log.write(1, 'Message [{}] is not a Message of Interest, No Subject.'.format(msgParts['messageId']))
     return None, None    # Not a message of Interest

But I am good on whatever works.

from dupreport.

HandyGuySoftware commented on May 25, 2024

On the road today. Will take a look at it tonight an work up a permanent fix. Will also look at the ts parsing problem, which I suspect is related.

from dupreport.

HandyGuySoftware commented on May 25, 2024

Just posted potential fix code to the Beta branch. I basically punted the problem by checking for essential message elements and skipping the message if any of them are missing:

        if msg['Message-Id'] is None or msg['Message-Id'] == '':
            globs.log.write(3,'No message-Id. Abandoning processMessage()')
            return None, None
        if msg['Subject'] is None or msg['Subject'] == '':
            globs.log.write(3,'No Subject. Abandoning processMessage()')
            return None, None
        if msg['Date'] is None or msg['Date'] == '':
            globs.log.write(3,'No Date. Abandoning processMessage()')
            return None, None

I also removed the debug lines that logged multiple copies of the email. That'll cut down on the log size a bit.

@ninja6o4 , I'll revise my earlier explanation for the multiple "does not exist in DB" messages. It's been a while since I've been in this section of the code. "Upon further review" it looks like dR only stores the data for emails that it successfully parses as Duplicati result messages. If it can't match an acceptable subject line (as defined in [main] subjectregex= in the .rc file) it just outputs an error message ("... not a message of interest...") and skips to the next message. As a result, if you have a lot of non-Duplicati messages in your inbox they will be reviewed every time the program runs. This was a time/space trade-off. I sacrificed some processing time to prevent storing lots of irrelevant message data in the database.

@mr-flibble, you may want to download and test this fix. You seem to have emails that like to break dR (:-)). Since this fix altered the way email headers are processed and matched it would be good to have you run through it as well.

Let me know how it goes.

from dupreport.

ninja6o4 commented on May 25, 2024

My current log folder is ~4000 messages, so it does take a few minutes to run, but probably not a big deal once i get this working and automated. My main concern is that it looks like it is literally downloading the entire folder every time it runs, which is ~200MB each time. I don't know if it would overly burden the db to log all message IDs and a flag as to if it is relevant or not. Maybe an option for the user as to whether or not to save it?

In any event, still getting the odd date format error. Log shows:

, Failed: .*, 24, 1
options.getRcSectionDateTimeFmt(, )
drDateTime.toTimestamp(, MM/DD/YYYY, HH:MM:SS, -28800)
drDateTime.toTimestamp(, None, None, -28800)
drdatetime.fromTimestamp(None, MM/DD/YYYY, HH:MM:SS)

from dupreport.

HandyGuySoftware commented on May 25, 2024

@ninja6o4, can you post the 20 or so lines around the format error message so I can see what's going on there?

from dupreport.

ninja6o4 commented on May 25, 2024

, Errors: [.*^], 24, 1
EmailServer.searchMesagePart(RGVsZXRlZEZpbGVzOiAwDQpEZWxldGVkRm9sZGVyczogMA0KTW9kaWZpZWRGaWxlczogMA0KRXhh

bWluZWRGaWxlczogMzY4MQ0KT3BlbmVkRmlsZXM6IDANCkFkZGVkRmlsZXM6IDANClNpemVPZk1v

ZGlmaWVkRmlsZXM6IDANClNpemVPZkFkZGVkRmlsZXM6IDANClNpemVPZkV4YW1pbmVkRmlsZXM6

IDI4ODEwNTc1OTQNClNpemVPZk9wZW5lZEZpbGVzOiAwDQpOb3RQcm9jZXNzZWRGaWxlczogMA0K

QWRkZWRGb2xkZXJzOiAwDQpUb29MYXJnZUZpbGVzOiAwDQpGaWxlc1dpdGhFcnJvcjogMA0KTW9k

aWZpZWRGb2xkZXJzOiAwDQpNb2RpZmllZFN5bWxpbmtzOiAwDQpBZGRlZFN5bWxpbmtzOiAwDQpE

ZWxldGVkU3ltbGlua3M6IDANClBhcnRpYWxCYWNrdXA6IEZhbHNlDQpEcnlydW46IEZhbHNlDQpN

YWluT3BlcmF0aW9uOiBCYWNrdXANClBhcnNlZFJlc3VsdDogU3VjY2Vzcw0KVmVyYm9zZU91dHB1

dDogRmFsc2UNClZlcmJvc2VFcnJvcnM6IEZhbHNlDQpFbmRUaW1lOiAxMi8yNy8yMDE3IDEwOjAw

OjMyIEFNDQpCZWdpblRpbWU6IDEyLzI3LzIwMTcgMTA6MDA6MDAgQU0NCkR1cmF0aW9uOiAwMDow

MDozMi42MDg3NjAzDQpNZXNzYWdlczogWw0KICAgIE5vIHJlbW90ZSBmaWxlc2V0cyB3ZXJlIGRl

bGV0ZWQsDQogICAgcmVtb3ZpbmcgZmlsZSBsaXN0ZWQgYXMgVGVtcG9yYXJ5OiBkdXBsaWNhdGkt

YmRiZWZjMzA2ZDRmYzQxYjc5MjQxN2EzZWRhZjU4MTk1LmRibG9jay56aXAuYWVzLA0KICAgIHJl

bW92aW5nIGZpbGUgbGlzdGVkIGFzIFRlbXBvcmFyeTogZHVwbGljYXRpLWliY2MxOGM0NjcyMWY0

MTBlOTc2ZTUxMDFhZWMyNDc1ZS5kaW5kZXguemlwLmFlcw0KXQ0KV2FybmluZ3M6IFtdDQpFcnJv

cnM6IFtdDQoNCg==

, Details: .*, 24, 1
EmailServer.searchMesagePart(RGVsZXRlZEZpbGVzOiAwDQpEZWxldGVkRm9sZGVyczogMA0KTW9kaWZpZWRGaWxlczogMA0KRXhh