handyguysoftware/dupreport

Email summary reporting tool for use with Duplicati backup system

WELCOME TO dupReport

dupReport is a Python-based email collection and reporting system for Duplicati. It will gather all your Duplicati backup status emails and produce a summary report on what Duplicati backup jobs were run and their success or failure.

Here is a list of some of dupReport's most important features:

• Collects all your Duplicati result emails and produces easy-to-understand status reports

• Runs on multiple operating systems. dupReport has been tested on Linux (Debian 8 & 9) and Windows 10, but users have reported it working on a wide variety of operating systems

• A Guided Setup for new users. If there is no configuration (.rc) file when the program runs, the Guided Setup will take the user through the most common configurable options.

• Support for IMAP and POP3 email services (IMAP is recommended for better results)

• Support for using multiple inbound (IMAP/POP3) and outbound (SMTP) email servers.

• Supports both text and JSON formatted status emails from Duplicati

• Supports SSL/TLS support for incoming/outgoing email transmissions.

• Output report in HTML, Text, CSV, and JSON formats

• Send results to email or local files (or both)

• User-defined reporting formats with configurable column and organization options

• Syslog-style logging format for easier searching and organization.

• Ability to send log output to an external syslog server or log aggregator.

• Support for the Apprise push notification service (https://github.com/caronc/apprise)

Please see the (new, updated, and reorganized) documentation on these and all the dupReport features.

---

Available Code Branches

Beginning with release 2.1, the branch structure of the dupReport repository has changed. We have moved to a more organized structure based on this article by Vincent Driessen (with some modifications). (Thanks to @DocFraggle for suggesting this structure.)

There are usually only two branches in the dupReport repository:

Branch Name	Current Version	Purpose
main	3.0.10	This is the Release branch, which should contain completely stable code. If you want the latest and greatest release version, get it here. If you are looking for an earlier release, tags in this branch with the name "Release_x.x.x" will point you there.
pre_prod	<None>	The Pre-Production branch. This is a late-stage beta branch where code should be mostly-stable, but no guarantees. Once final testing of code in this branch is complete it will be moved to master and released to the world. If you want to get a peek at what's coming up in the next release, get the code from here. If you don't see a pre_prod branch in the repository, that means there isn't any beta code available for testing.

If you see any additional branches in the repository, they are there for early-stage development or bug fix testing purposes. Code in such branches should be considered highly unstable. Swim here at your own risk. Void where prohibited. Batteries not included. Freshest if eaten before date on carton. For official use only. Use only in a well-ventilated area. Keep away from fire or flames. May contain peanuts. Keep away from pets and small children.

Bug reports and feature requests can be made on GitHub in the dupReport Issues Section. Please do not issue pull requests before discussing any problems or suggestions as an Issue.

The discussion group for dupReport is on the Duplicati Forum in this thread.

The program is released under an MIT license. Please see the LICENSE file for more details.

Enjoy!

---

Documentation

What is dupReport?

System Requirements

Getting Started (Quickly)

Understanding Source-Destination Pairs and Identifying Emails of Interest

Running dupReport

• Installation

• Command Line Options

dupReport.rc File Configuration

• Program Management: The [main] Section
• Handling Specific Backup Jobs: The [<source-destination>] Sections
• Email Management
• Push Notifications Using Apprise: The [apprise] Section

Reporting

• The [report] Section
• Custom Report Formats

dupreport's Issues

2.1.0 DB out of date error

Hello, after upgrade to 2.1 beta, I get this error message.

Nothing in log file
Config is automatically updated

*** Remote Interpreter Reinitialized  ***
>>> 
Database file B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1/dupReport.db is out of date. Needs update to latest version.
Traceback (most recent call last):
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1\dupReport.py", line 138, in <module>
    canContinue = initOptions()
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1\dupReport.py", line 91, in initOptions
    globs.log.out(1, 'Need to upgrade database {} from version {} to version {}{}{}'.format(globs.opts['dbpath'], currDbVersion, globs.dbVersion[0], globs.dbVersion[1], globs.dbVersion[2]))
TypeError: out() takes exactly 2 positional arguments (3 given)
>>> 

Workaround for this message is dupReport.py -i - after the error is no longer displayed

dupReport throws a strange error

Brand new Debian 9 with nothing uncecessary installed,

wolf@dupReport:~/dupReport-master$ ./dupReport.py
Traceback (most recent call last):
File "./dupReport.py", line 868, in
create_summary_report()
File "./dupReport.py", line 646, in create_summary_report
d0 = datetime.date(int(then[0]),int(then[1]), int(then[2]))
ValueError: month must be in 1..12

dupReport.rc as :

[main]
dbpath = /home/adeo/dupReport-master
logpath = /home/adeo/dupReport-master
verbose = 1
logappend = true
sizereduce = giga
subjectregex = ^Duplicati Backup report for
summarysubject = Duplicati Backup Summary Report
srcregex = \w*
destregex = \w*
srcdestdelimiter = -
border = 1
padding = 5

[incoming]
transport = imap
server = ssl0.ovh.net
port = 993
encryption = tls
account = dupreport@*.com
password = ****************
folder = INBOX

[outgoing]
server = ssl0.ovh.net
port = 587
encryption = tls
account = dupreport@*.com
password = ****************
sender = dupreport@*.com
receiver = support@*.com

Inactive backup warning!

Not sure if this would be a needed feature or not for most.

I have noticed some duplicati users myself included that run some backupsets on a manual basis.
Most likely the destination server is not always online.

In order be warned about backups that haven't been completed in X amount of days may be useful

Most of the time one would figure this out in the main reports. If they bother to read them completely

Need to handle different date/time formats

It looks like the date and time formats in the Duplicati email reports vary depending on locale. I have the following formats:

This one is OK:
EndTime: 11/4/2017 12:03:26 PM
BeginTime: 11/4/2017 11:58:16 AM

This one fails with exception:
EndTime: 04/11/2017 14:33:58
BeginTime: 04/11/2017 14:31:00

Probably many ways to handle that. The attached version works for me this moment.

dupReport.zip

-c option broken

When running dupreport with the -c option I get the following: This happens with warnoncollect set to either true of false. The warning email does get sent before crash.

Traceback (most recent call last):
  File "./dupReport.py", line 217, in <module>
    globs.outServer.sendEmail(msgHtml, msgText)
NameError: name 'msgHtml' is not defined

Temporary fix was add the "and not globs.opts['collect']:" on line 210 and 214.

    # Do we need to send output to file(s)?
    if globs.opts['file'] and not globs.opts['collect']:
        report.sendReportToFile(msgHtml, msgText, msgCsv)

    # Are we forbidden from sending report to email?
    if not globs.opts['nomail'] and not globs.opts['collect']:
        globs.outServer.sendEmail(msgHtml, msgText)

This corrects the error with both email and file creation.

TypeError: can't use a string pattern on a bytes-like object

Received the above error when trying to run dupReport.py.

Stack trace and logifile are below.

The subject line of the email message that caused it to blow up was automatically generated by a FOSCAM IP camera.

I am not sure what the "b" is on the front of the subject line that reads,

Subject=[b'SonomaGrandroom:IPCamera SMTP test mail'].

I suspect that something funky is encoded into the byte stream of the subject line and perhaps needs to be filtered out / converted into a processable byte stream?

Any help would be greatly appreciated!

Marc

Stack trace

./dupReport.py
Traceback (most recent call last):
  File "./dupReport.py", line 861, in <module>
    process_mailbox_imap(mailBox)
  File "./dupReport.py", line 739, in process_mailbox_imap
    mParts = process_message(msg)                # Process message into parts
  File "./dupReport.py", line 470, in process_message
    if re.search(options['subjectregex'], msgParts['subject']) == None:
  File "/usr/lib/python3.4/re.py", line 170, in search
    return _compile(pattern, flags).search(string)
TypeError: can't use a string pattern on a bytes-like object

Logfile

Processing mailbox...
Message ID [<[email protected]>] does not exist. Adding to DB
Subject=[Marc, welcome to your new Google Account].
Message [<[email protected]>] is not a Message of Interest
Message ID [<[email protected]>] does not exist. Adding to DB
Subject=[This is a test].
Message [<[email protected]>] is not a Message of Interest
Message ID [<[email protected]>] does not exist. Adding to DB
Subject=[b'SonomaGrandroom:IPCamera SMTP test mail'].

Option to initialize the DB then stop

The -i option inits the database then continues processing. A new -I option will init the database then exit. Would be useful for testing program to re-create specific conditions without having to re-run the entire data set.

Add background color to "No New Activity" message in reports

When checking it with normal report mode could the line that reads "No new activity. Last activity on 2017-11-17 at 20:00:16 (30 days ago)" be changed to red when the nobackupwarn is hit? Just something to make it pop out like the errors and warnings do?.

Change default report to show MB

Enhancement request: Change default report to show MB.

The announcement page shows "Size(MB)" and "+/-(MB)", but that isn't showing up in my output version.
Original posting picture.

My output with version 2.0.3 using defaults shows:

If using bytes, it should show "bytes".

Also some add'l thoughts, let me know if you want me to open a separate issue for this one:
Could I suggest the following columns (separated by commands for clarity only:
Date;Time;Protected Files Count; Files Count changing during Job ; Protected Size (MB); Size change during job; Add files; Deleted files; Modified files; Errors; Job Result

Also, if you can use a two layer table:
Overall Stats versus Job Stats would be interesting.

--Dave

Produce Report Grouped by Source or Destination

Marc_Aronson

I have a suggestion for an alternative way of formatting reports. The basic idea is to create a visual grouping when the same source is backed up to multiple destinations.

A picture is worth a thousand words – this google sheet is a mockup of what I am thinking.

Food for thought…

Marc

Figure out UTC offset from email header

Recognize UTC offset from incoming email header and apply that offset to EndDate field in report.

DB not updating

dubReport is running properly, and I'm getting the emails with the reports, but they're empty.

The log file is being updated, but the DB is not.

Way to be warned of failures between reports

My plan is to run the dupreport.py -r /etc/dupreport -c on a nightly basis. Only problem I see with this is I am not alerted to any errors in between the -c and when I run dupreport with the -t option. Having it run daily I would just end up ignoring the email altogether.

A way to parse the error and warning is needed. Given my limited skills I am testing the following

This very poorly done hack was attempted on the 2.0.3 issue 18 branch

On line 438 I added the following to def build_email_sql_statement

    # If errors found send backup warn email!
    if ('collect'in options) and ((sParts['warnings'] != '' or sParts['errors'] != '')):
         send_email_backup_warn( mParts['sourceComp'], mParts['destComp'], mParts['emailDate'], mParts['emailTime'], sParts['messages'], sParts['warnings'], sParts['errors'])

Then have the following code added. I am sure you can clean this up better than me if you decide to add something like this.

def send_email_backup_warn(source, destination, lastDate, lastTime, messages, warning, error):

    write_log_entry(2, 'Send_email()')
    msgText=''
    msgHtml='<html><head></head><body><table border={} cellpadding="{}">'.format(options['border'], options['padding'])
    msgHtml = msgHtml + '<tr>'  # New table row
    msgText = msgText + '{}@{}\n'.format(source,destination)
    msgHtml = msgHtml + '<td align="center" colspan = "11"><b>{}@{}</b></td>'.format(source,destination)
    if (warning != ''):
         msgText = msgText + '{}\n\n'.format(warning)
         msgHtml = msgHtml + '<tr><td><align="center" colspan = "11"><i>{}</i></td>'.format(warning)
    if (error != ''):
         msgText = msgText + '{}\n\n'.format(error)
         msgHtml = msgHtml + '<tr><td align="center" colspan = "11"><i>{}</i></td>'.format(error)
    msgText = msgText + '\n'
    msgHtml = msgHtml + '</tr></body><html>'

    # Build email message
    msg = MIMEMultipart('alternative')
    msg['Subject'] = "!!! Backup Warning for {}@{} !!!".format(source, destination)
    msg['From'] = options['outsender']
    msg['To'] = options['outreceiver']

    # Send the message via local SMTP server.
    server = smtplib.SMTP('{}:{}'.format(options['outserver'], options['outport']))
    write_log_entry(2, 'SMTP Server=[{}]'.format(server))
    if options['outencryption'] == 'tls':   # Do we need to use SSL/TLS?
        server.starttls()
    server.login(options['outaccount'], options['outpassword'])
    server.sendmail(options['outsender'], options['outreceiver'], msg.as_string())
    server.quit()

Sorry for my bad programming ability :(

Hopefully someone can test this. I didn't keep my bad backup emails from duplicati.

edit: fixed the output email to better match Handyguysoftware

Background on table cell

Hello HandyGuy, I made a small change, see if you find it interesting: a highlight in the lines with the names of the jobs. When you have a lot of small backups, I found the report look a bit confusing, so I put a background in the line with the names. Look:

I just put the color tag here (line 690):

msgHtml = msgHtml + '<td bgcolor="#E6E6E6" align="center" colspan = "11"><b>{}</b></td>'.format(txt[0])

( BTW: Do you prefer this type of post here or in the Duplciati forum? )

Option to send output to a file or stdout instead of email

Scheduled for 2.0.4

Logging doesn't flush to file

Brand new to dupreport, and i noticed that it doesn't flush to the log file until the end of the job.

I'm running it on CentOS7, installed Python3.4, did the initial config, and am now running it.

It ended up timing out with an SMTP error, but in the 5 minutes while I waited, nothing was written to the log.

Unfortunately, while I can sort of read python, I can't write it. I can google, and the following may help:
https://stackoverflow.com/questions/39396393/logging-how-to-control-the-times-in-which-flush-to-log-file

Customize the Sub-heading in the Report

Marc_Aronson

Another idea for enabling report customization that I think would pair well with what you described: Allow the user to format the sub-header row that currently contains “”*** source-name to destination-name ***" with a template that allows keyword substitution. ie, Something like %SOURCENAME% would be substituted with the actual source name.

A new sub-head row would need to be generated anytime one of the keywords used in the template changes.

Marc

allow -r & -d options to specify full file path

-r & -d options only take a path specification. Change behavior to specify:

<path> to use /dupReport.rc or /dupReport.db

or

<full-path-to-file> to use full path spec as filepath

Convert .rc and database files to new format for 2.1 release

Revert database to a specific date

The -i option clears the database to a clean state. For testing purposes, add the ability to revert (rollback) the database to a specific date. This will allow re-running the program as of that date.

Use <details> tag for better readability of error message

Hello, I thought it would be useful use https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details for collapsing error messages in report.

I only tried it with html body fo email, not with dupReport code.

Here is how it looks like without and with details tag (collapsed).

And expanded

Long Column Titles and Summary Stats

Also some add'l thoughts, let me know if you want me to open a separate issue for this one:
Could I suggest the following columns (separated by commands for clarity only:
Date;Time;Protected Files Count; Files Count changing during Job ; Protected Size (MB); Size change during job; Add files; Deleted files; Modified files; Errors; Job Result

Also, if you can use a two layer table:
Overall Stats versus Job Stats would be interesting.

--Dave

Update README and Changelog file for 2.1 release.

Research and create /docs directory and supporting files to move README to a web-based document.

dbpath, rcpath, and logpath should check for ending '/' to avoid duplicate path character specification

If dbpath, rcpath, or logpath have a trailing '/' charactewr, remove this, since the program will add it back during path processing. Leaving it in causes duplicate '/' and causes an error.

Can't parse the '@' sign in subject emails

dcurrey8h
So far the only issues I have seen with very small test is I can’t change the srcdestdelimiter setting. I have tried _ @ # all cause errors. In fact if it tries to parse any message with the @ in the subject errors are produced.

Since this was my own mailserver I just filtered the mails from Duplicati into its own folder and had dupreport use that.

This is Ubuntu 16.04 testing using dovecot and postfix

Had a chance to look at the @ in subject again. I think it might be unique to some mail clients. Sogo mail was creating the issue but it would accept same message from horde and roundcube webmail clients.

For example message with the subject of "this is a message with @ in subject’’

Sogo headers would look like.

User-Agent: SOGoMail 3.2.10
MIME-Version: 1.0
Date: Mon, 23 Oct 2017 15:04:46 -0400
Subject: this is a message with =?utf-8?q?=40?= in subject
Message-ID: 476-59ee3d80-b-1881d5c0@267997335
The other two clients leave it alone

Subject: this is a message with @ in subject
Errors that happen

Traceback (most recent call last):
File "./dupReport.py", line 856, in
process_mailbox_imap(mailBox)
File "./dupReport.py", line 734, in process_mailbox_imap
mParts = process_message(msg) # Process message into parts
File "./dupReport.py", line 480, in process_message
msgParts['sourceComp'] = re.search(srcRegex, msgParts['subject']).group().split('-')[0]
AttributeError: 'NoneType' object has no attribute 'group'

Looking at the code it looks like the “-” is hard coded on line 478 and 479.

Shouldn’t that be something like

msgParts['sourceComp'] = re.search(srcRegex, msgParts['subject']).group().split(options['srcdestdelimiter'])[0]
msgParts['destComp'] = re.search(destRegex, msgParts['subject']).group().split(options['srcdestdelimiter'])[1]

Tmp log file not being removed after copy to permanent log file.

Remove emails from database that are no longer on server.

Need a way to purge the old emails out the database that are no longer available on the email server.

This is what I was thinking short of having dupreport remove seen emails itself after collect. Not sure I would like that option myself.

Here is what I am currently testing.

Added another column to the emails database table for an emailseen flag. Simple 1 for seen 0 for not seen would do.

Have a new setting purgdbemail set to false added to options

At the start of email collect check you update the status to 0 for all emails.
When an email is first seen you just add it to the database with the emailseen = 1.
Email is already in database you update emailseen back to 1.
At end of run anything not added or seen at this point should still be 0.
If the purgedbemail is false nothing else happens. If true you go and delete all email entries in database where emailseen=0.

This should keep the database at minimum size and in sync with email folder.

Any thoughts on this or maybe a better way of doing it?

Incorrect 12/24 hour time conversion

EndTime: 10/27/2017 12:30:27 PM

Shown as:

2017/10/27 | 24:30:27 in dR summary email

Duplicati Backup report for G-B2 10/27/17 7:30 AM

Include links to file list for a backup

Is it possible for the report to include links such as to the web GUI command line for a specific job with parameters pre-populated to generate a file list for the specific job?

I realize it would only work if clicked while on the machine from which the backup ran, but it might still be useful…

Requested by JonMikelV in the Duplicati forums. Link to thread.

Customize date/time format specification by source/destination pair

Helpful if you have backups running in different locales

2.1.0 datime error

Hello, after upgrade I get this error message (no problem before upgrade)

Traceback (most recent call last):
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1\dupReport.py", line 182, in <module>
    globs.inServer.processMessage(nxtMsg)
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1\dremail.py", line 338, in processMessage
    dtTimStmp = drdatetime.toTimestamp(xDate, dfmt='YYYY/MM/DD', tfmt='HH:MM:SS')  # Convert the string into a timestamp
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1\drdatetime.py", line 106, in toTimestamp
    ts = datetime.datetime(year, month, day, hour, minute, second).timestamp()
AttributeError: 'datetime.datetime' object has no attribute 'timestamp'

My config:

dateformat = DD.MM.YYYY
timeformat = HH:MM:SS
warnoncollect = false
applyutcoffset = false
show24hourtime = true

Last messages in log:

]
Database.searchForMessage(<DRDILCEFW2U4.C5C3URJ3TJ903@xx>)
Database.execSqlStmt(): Executing SQL command.
Message ID [<DRDILCE4W2U4.C7C3URJ3TJ903@xx>] does not exist. Adding to DB
drDateTime.toTimestamp(2017/11/17 15:54:29, YYYY/MM/DD, HH:MM:SS, None)

Probably date and time in problematic email (which worked in 2.0.4)
Date: Fri, 17 Nov 2017 15:54:29 +0100

Select the fields that show up in the report.

Provide a mechanism for selecting the fields that show up in the email report.

Option to add purgedb into dupReport.rc missing

When updating beta to current version I get the error

  File "./dupReport.py", line 128, in <module>
    canContinue = initOptions()
  File "./dupReport.py", line 65, in initOptions
    if oMgr.readRcOptions() is True:  # Need to restart program (.rc file needs editing)
  File "/home/dcurrey/dupReport-2.1.0_Beta/options.py", line 211, in readRcOptions
    self.options['purgedb'] = self.options['purgedb'].lower() in ('true')   # boolean
KeyError: 'purgedb'

Level 2 logs too verbose. Add a Level 3 with the really esoteric stuff.

Check for version numbers during upgrade process

Check for from/to versions during upgrade process to make future updates more manageable.

Create Temporary Storage files for pre-logging program output.

Due to the program structure, there is a lot of message output prior to opening up the dupreport.log file. This can be suppressed using the suppress() function call, but then all that debugging info is lost.

Create a temporary working space to hold pre-log program output, then copy that pre-log info to the official log file once it gets opened,

Bold column titles on reports

convert_time_date IndexError

Hello, this is my first try to run dupReport, so I'm sorry if this is my mistake :)

If I runs script, I get this error:

Traceback (most recent call last):
  File "D:\Portable Python 3.2.5.1\dupReport.py", line 941, in <module>
    process_mailbox_imap(mailBox)
  File "D:\Portable Python 3.2.5.1\dupReport.py", line 796, in process_mailbox_imap
    mParts = process_message(msg)                # Process message into parts
  File "D:\Portable Python 3.2.5.1\dupReport.py", line 542, in process_message
    dateParts['endSaveDate'] = convert_date_time(statusParts['endTimeStr'])[0]
  File "D:\Portable Python 3.2.5.1\dupReport.py", line 371, in convert_date_time
    endDate = "{:04d}/{:02d}/{:02d}".format(int(datePart[2]),int(datePart[0]),int(datePart[1]))
IndexError: list index out of range
>>> 

This is at the end of the log:

Warnings: []
Errors: []

]
db_search_message() for messageId=[<WN6AOV4ES2U4.04WRZNI25YDY@JC-PC>]
Message ID [<WN6AOV4ES2U4.04WRZNI25YDY@sumarum>] does not exist. Adding to DB
db_search_srcdest_pair(test, report)
Source/Destination pair [test/report] already in database.
convert_date_time(5. 11. 2017 10:38:48)

And this time/date is in email:

EndTime: 5. 11. 2017 10:38:48
BeginTime: 5. 11. 2017 10:38:28
Duration: 00:00:19.7302375

And this is my regional settings:

Is there something I can do to fix this? Thank you! :)

Repeat legend column for every job (enhancement)

Hello, sorry for bother with another "issue" :) This time it's an idea for a new feature

I run a report once every 5 days, so table is pretty long and I have a problem remembering which column is what. ..

What do you think about option to repeating the line containing "Data, Errors, Files ..." after the name of each backup job? I think it would certainly improve readability.

Option to order report by source or by destination

Currently "order by source" is the default.

2.1 Beta no go

Tried 2.1 fails to create the database and .rc file on linux. Attempted clean install of dupreport.

Here is what I am seeing right after I run dupreport.py

Traceback (most recent call last):
  File "./dupReport.py", line 138, in <module>
    canContinue = initOptions()
  File "./dupReport.py", line 54, in initOptions
    needToUpgrade, currRcVersion = oMgr.checkRcFileVersion()
  File "/home/dcurrey/dupReport-2.1.0_dc/options.py", line 149, in checkRcFileVersion
    return needToUpgrade, currVerNum
UnboundLocalError: local variable 'currVerNum' referenced before assignment

The duplog shows

******** dupReport Log - Start: Fri Dec  8 10:49:40 2017
initOptions()
progPath=/home/dcurrey/dupReport-2.1.0_dc
Processing command line...
options.processCmdLineArgs()
Command line parsed. args=[Namespace(Version=False, append=False, collect=False, dbpath=None, file=None, initdb=False, logpath=None, nomail=False, rcpath=None, report=False, rollback=None, size=None, verbose=None)]
RC path not specified on command line. Using default.
RC path=[/home/dcurrey/dupReport-2.1.0_dc/dupReport.rc]
options.openRcFile(/home/dcurrey/dupReport-2.1.0_dc/dupReport.rc)
options.checkRcFileVersion()
Need to upgrade rc file? True

Problem with Czech characters in email message

Hello, sorry for new problem:)

I'm getting this error message (bellow) because one email with czech characters ("ř" or "í" )

Traceback (most recent call last):
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1_IS_config\dupReport.py", line 1053, in <module>
    send_email()
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1_IS_config\dupReport.py", line 728, in send_email
    server.sendmail(options['outsender'], options['outreceiver'], msg.as_string())
  File "B:\Dulezite_Programy_universal\dupReport_PortablePython3.2.5.1_IS_config\App\lib\smtplib.py", line 735, in sendmail
    msg = _fix_eols(msg).encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 909-910: ordinal not in range(128)

This is mail causing problems:

DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 26
ExaminedFiles: 34909
OpenedFiles: 28
AddedFiles: 0
SizeOfModifiedFiles: 17731452928
SizeOfAddedFiles: 0
SizeOfExaminedFiles: 88587088695
SizeOfOpenedFiles: 17731452928
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 2
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
ParsedResult: Error
VerboseOutput: False
VerboseErrors: False
EndTime: 23.11.2017 23:25:34
BeginTime: 23.11.2017 23:00:00
Duration: 00:25:33.6594762
Messages: [
    No remote filesets were deleted
]
Warnings: [
    Failed to process metadata for "x.ldf", storing empty metadata => (5) Přístup byl odepřen: [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18\x.ldf],
    Failed to process path: x => (5) Přístup byl odepřen: [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18\x.ldf],
    Failed to process metadata for "x.mdf", storing empty metadata => (5) Přístup byl odepřen: [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18\x.mdf],
    Failed to process path: x.mdf => (5) Přístup byl odepřen: [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18\x.mdf]
]
Errors: []

I found workaround - if i change:
old value msg = _fix_eols(msg).encode('ascii')
to msg = _fix_eols(msg).encode('utf-8')
in file ..\App\Lib\smtplib.py

then reporting is working ok*

But I don't know is this is best practice :-)

Program crashing when processing emails.

from ninja6o4:

Traceback (most recent call last):
File “dupReport.py”, line 213, in
globs.inServer.processMessage(nxtMsg)
File “C:\dupReport-2.1.0_Beta\dremail.py”, line 304, in processMessage
decode = email.header.decode_header(msg[‘Message-Id’])[0]
File “C:\Users\xxx\AppData\Local\Programs\Python\Python36-32\lib\email\header.py”, line 80, in decode_header
if not ecre.search(header):
TypeError: expected string or bytes-like object

I pruned the log, here is the last entry on it: ------=_NextPart_000_0025_01D1E182.8F2771A0-- ] messageId=<002401d1e1bd$3b8 - Pastebin.com1

rpt_bysource & rpt_bydest do not handle unseen backup sets properly

Backup sets without any activity do not display properly in these reports.

Add -I option: Initialize & keep processing

A shortcut to aid the initialize/run/initialize/run testing loop.

Program crashing when seeing new Source-Destination Pair

1. When adding a new S/D pair, program crashes in final sendEmail() call
2. Performing a rollback (-b) right after adding a new S/D pair causes a crash

Option to remove a source-destination pair from backupsets table

Handling failed backups

If dupreport tries to parse a failed upload it gives errors. In this case a simple aborted upload.

Temporary workaround was to add advance option in duplicatii editing they way mail subject is displayed.

--send-mail-subject=Duplicati %OPERATIONNAME% (%PARSEDRESULT%) report for %backup-name%

This create the subject of "Duplicati Backup (Success) report for somebackup-somedest"

Then edit the dupReport.rc file changing the subjectregex setting

subjectregex = ^Duplicati Backup \(Success\) report for

DupReport now should only process successful backups. You may to need to look at some kind of option to notify users when dupreport encounter a failed or warning status for %PARSEDRESULT% from the subject line.

Check how default subheading= is produced

Check how the subheading= line in .rc file is produced for each of the various reports.

Also, add documentation on allowable keywords for each report.

Recognize UTC timestamp in email fields

recognize "(timestamp)" field in EndDate and EndTime fields in report emails. This will be incorporated in future Duplicati releases.