tierpod/dmarc-report-converter

Convert dmarc reports from xml to human-readable formats

dmarc-report-converter's Issues

Created a Docker Compose wrapper

Hi,

Not sure how to contact you/y'all so here's an "issue".
I made a Docker Compose wrapper around dmarc-report-converter to make it easier for myself to generate and view reports on my local machine: https://github.com/nielsbom/dmarc_report_viewer

I've only tested this on my arm64 Apple hardware and using OrbStack so not sure if/how it works on other hardware, OS'es or container tech.

Feel free to close this issue or link to it from this repo's README.

And most importantly thank you for making this tool!

Add support for xml input files

Error without outputting json file when report contains empty record

Hi there

Thanks for this awesome tool, I've been experimenting with it for the last 2 weeks or so and it works very good!
However, today I ran into a problem, that only seems to happen when choosing json as output format.

When running the tool today, it suddenly stopped with this error:

2024/03/07 08:11:18 [ERROR] processFiles: json: error calling MarshalJSON for type dmarc.Report: json: unsupported value: NaN

It took me a while to figure out where the problem came from, but eventually I found an empty generated json file and I could trace back to a report that contained something like this:

    <record>
        <row>
            <source_ip/>
            <count>0</count>
            <policy_evaluated>
                <disposition/>
                <dkim/>
                <spf/>
            </policy_evaluated>
        </row>
        <identifiers>
            <header_from/>
        </identifiers>
        <auth_results>
            <spf>
                <domain/>
                <result/>
            </spf>
        </auth_results>
    </record>

Not sure why a reporter would send such an empty report, but it happened.

When choosing html as output format, this tool is able to generate a report without errors, but not with json output format. Would be great if this bug could be fixed somehow, either by fixing the json generating logic, or by just skip outputting a report since this one contains no useful info

(would also be nice if this tool had some verbose flag, that you can use to output debug info, for example in this case it would have been nice if I could find the problematic report's filename while running in verbose mode)

Thanks!

Add support fo zip input file

Improve UX of browsing through reports

Hi, thanks for the cool tool! I was looking for something like this, most other projects use way more elaborate dependencies I do not need. It fits my needs quite good :D

Currently it is pretty tedious to browse through all the reports that are put in a subfolder. I do like the idea of using the webserver built-in feature of open directories, but the UX isn't great at the moment. Let me know your thoughts on this!

Possible options:

• Merging all the pages to a single page.
• Rename the actual report file to 'index.html' so clicking on a sub-folder makes you enter the report directly. If there are multiple for a single day, they could be merged (maybe as a setting for people who do not want this)
• Not merge pages, but adding 'next' buttons to report pages so it is easier to browse
• Something else?

Add readme, usage and screenshots

Add config

Messages with inline MIME headers are not processed

I've received some emails that use inline MIME headers which are silently ignored.

Sample content:

MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: multipart/mixed; boundary="17122880850.dd5Fb0.406180"


--17122880850.dd5Fb0.406180
Date: Thu, 4 Apr 2024 23:34:45 -0400
MIME-Version: 1.0
Content-Type: text/plain; charset="US-ASCII"
Content-Disposition: inline


This is a DMARC aggregate report for example.com

8 records.
8 passed.
0 failed.

Submitted by Pobox by Fastmail Pty Ltd
Generated with Mail::DMARC 1.20240313


--17122880850.dd5Fb0.406180
Date: Thu, 4 Apr 2024 23:34:45 -0400
MIME-Version: 1.0
Content-Type: application/gzip;
 name="pobox.com!example.com!1712188800!1712275199!46972001.xml.gz"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="pobox.com!example.com!1712188800!1712275199!46972001.xml.gz"

PUT/YOUR/BASE64/STUFF/HERE==

--17122880850.dd5Fb0.406180--

At the very least, it would help to log a message:

[WARN] extractAttachment: inline MIME headers are not currently supported, skip

The underlying issue is here:

dmarc-report-converter/cmd/dmarc-report-converter/imap.go

Lines 173 to 174 in abf80ca

	switch h := p.Header.(type) {
	case *mail.AttachmentHeader:

An offending message returns headers of type *mail.InlineHeader. The go-message API for inline vs attachment headers is very different. See emersion/go-message#138 for a discussion about the need for feature parity between the two types, but it doesn't appear that any work has been done.

Feature request: weekly or monthly report

Hi,
First of all, thank you for the great tool. I was able to install it and I love it!
It is great.
I was wondering if you could consider to add a weekly report or a monthly report for the same email.

Thanks

Limit number or time of reverse lookup when lookup_addr = True

When dmarc report contains many ip addresess it takes too long time to perform reverse lookup

Make AssetsPath available in external_template context

When building a custom template using the external_template format, the .AssetsPath value is not available to the template engine context.

Add html_static.template (without jquery)

Enable GitHub workflow on pull requests

Can the GH workflow be enabled on pull_request instead of just push? I don't think it would trigger a release since the workflow checks for tags before doing that.

I was thinking of changing the workflow to use:

on:
  push:
  pull_request:
    paths-ignore:
      - '**.md'

Add ability to delete source file after converting

Support importing .pst files

'Personal Storage Table' is a Microsoft-proprietary format, which is commonly used when exporting a mailbox from Microsoft Outlook. I used readpst from pst-utils to extract the .xml.gz files which could be imported here, but it would be nice if it was directly integrated here. Seems a Go implementation exists here, but I am not sure if it supports extracting the mail attachments to files in a directory: https://github.com/mooijtech/go-pst

Edit: seems like it supports attachments!

Improve log messages

Ignore eml files when processing extracted reports

Given input.delete = no

After eml files are processed and attachments have been extracted, the convert/parse phase attempts to reprocess the eml files as reports and logs an error:

[ERROR] files: mimetype text/plain; charset=utf-8 not supported in file dmarc_files/foo.eml, skip

We should ignore the eml files in this phase.

I plan to submit a PR shortly to resolve this issue.

Date field request

Hi,

You have created a ID and TodayID field, but I'm looking to only a date field when the report is running.
Is it possible to also create a Today field that only gives Y-M-D without any other information?

This way I can create from the config file a path like:
{{.Today}}/{{.PolicyPublished.Domain}}/{{.ReportMetadata.OrgName}}/index.html
That will result in:
2022-3-2/domain.com/google.com/index.html

Thanks.

Most dmarc reports from Comcast cannot be read

I've seen this problem with most (but not all) reports from Comcast:

[INFO] performs a reverse lookups, this may take some time
[INFO] imap: connecting to server imap.gmail.com:993
[INFO] imap: found 1 messages, 0 unseen
[INFO] imap: found attachment: comcast.net!xxx.org!1679443200!1679529600.xml.gz
[INFO] imap: save attachment to: /Users/xxx/Downloads/dmarc/comcast.net!xxx.org!1679443200!1679529600.xml.gz
[ERROR] imap: unexpected EOF, skip
[INFO] files: found 1 input files in /Users/xxx/Downloads/dmarc
[ERROR] files: unexpected EOF in file /Users/xxx/Downloads/dmarc/comcast.net!xxx.org!1679443200!1679529600.xml.gz, skip
[ERROR] processFiles: reports list is empty

The error is happening here:

imap.go:
				_, err = io.Copy(f, p.Body)
				if err != nil {
					log.Printf("[ERROR] imap: %v, skip", err)
					continue
				}

My first thought was to update everything in go.mod, in case emersion had fixed it:

module github.com/tierpod/dmarc-report-converter

go 1.20

require (
	github.com/emersion/go-imap v1.2.1
	github.com/emersion/go-message v0.16.0
	github.com/hashicorp/logutils v1.0.0
	gopkg.in/yaml.v2 v2.4.0
)

require (
	github.com/emersion/go-sasl v0.0.0-20220912192320-0145f2c60ead // indirect
	github.com/emersion/go-textwrapper v0.0.0-20200911093747-65d896831594 // indirect
	golang.org/x/text v0.8.0 // indirect
)

But now imap.go no longer compiles:

impossible type switch case: mail.AttachmentHeader
	p.Header (variable of type "github.com/emersion/go-message/mail".PartHeader) cannot have dynamic type "github.com/emersion/go-message/mail".AttachmentHeader (method Add has pointer receiver)

I am not a go developer, so I'm rather stuck at this point. Any thoughts?

Add tests

incorrect parsing Auth results DKIM

It happen for not parsing the correct DKIM Auth Result as the message may got multiple DKIM signatures.
Example:

<auth_results>
  <dkim>
    <domain>[domain1]</domain>
    <result>pass</result>
    <selector>select1</selector>
  </dkim>
  <dkim>
    <domain>[domain2]</domain>
    <result>fail</result>
    <selector>selector2</selector>
  </dkim>
  <spf>
    <domain>[domain1]</domain>
    <result>pass</result>
  </spf>
</auth_results>

I got this, and the parsed HTML show this entry as DKIM auth fail, SPF auth pass, but DMARC still pass for DKIM & SPF

Thank you!

Just wanted to say thanks. My day is better because of you. 👍

Add support for imap or eml input files

Setup instructions not working

I am trying to follow the instructions and I cannot get the first line to work:

download pre-builded archive from releases or build dmarc-report-converter executable:

go get -u github.com/tierpod/dmarc-report-converter
cd $HOME/go/src/github.com/tierpod/dmarc-report-converter
make bin/dmarc-report-converter
now you can run ./bin/dmarc-report-converter and test it

Configuration on nginx not working

I am wondering what I am doing wrong.
Do you actually need php, go installed if you don't build from source?
I just want to generate and serve the html files via nginx (daily reports) so I can view them via browser. It should be simple right?

The main thing I don't understand how do you actually serve report files from /usr/share/nginx/html.
I have run the ./dmarc-report-converter.sh and I see the files are generated in the /tmp/dmarc_files and /tmp/html folder.

A working sample nginx config and config.yaml would be super helpful to get up and running quickly... I am probably doing something really silly...

/etc/nginx/conf.d/dmarc.conf

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name dmarc.example.com;
    ssl_certificate /etc/ssl/certs/cert.crt;
    ssl_certificate_key /etc/ssl/certs/cert.key;

location /dmarc/ {
    root /usr/share/nginx/html;
    autoindex           on;
    autoindex_localtime on;
    }
}

/opt/dmarc-report-converter/config.yaml

input:
  delete: no
  dir: "/tmp/dmarc_files/"
  imap:
    server: "mail.example.com:993"
    username: "[email protected]"
    password: "pass"
    mailbox: "inbox"
    debug: no
    delete: no
output:
  file: "/tmp/html/{{ .ID }}.html"
  format: "html"
  assets_path: "/dmarc/assets"
lookup_addr: no
merge_reports: no
log_debug: no
log_datetime: no

from /var/log/nginx/error.log looks like my paths are just wrong...
do you need to configure index in nginx config too?

2022/11/14 21:40:53 [error] 2057#2057: *1 open() "/etc/nginx/html/dmarc" failed (2: No such file or directory), client: 1.2.3.4, server: dmarc.example.com, request: "GET /dmarc HTTP/2.0", host: "dmarc.example.com"

Only delete mails if attachements are succesfully saved to tmp location

Hey,

I have run into the following issue:

[INFO] imap: found 19 messages, 0 unseen
[INFO] imap: found attachment: <filename>.zip
[INFO] imap: save attachment to: /tmp/dmarc_files/<filename>.zip
[ERROR] imap: open /tmp/dmarc_files/<filename>.zip: permission denied, skip
[...]
[ERROR] processFiles: reports list is empty

I the description it says:

delete (bool): delete email messages from IMAP server if reports are fetched successfully

So the expected behaviour should be, that the email messages are only deleted, if they are saved succesfully.
However the observed behavious is, that all found emails are going to be deleted from the inbox, although the file could not be saved.

imap: can't read next part: unhandled charset "utf-ascii"

Some providers send attachments with utf-ascii charset, seems like vendored version of imap library doesn't support it:

Content-Type: application/gzip;
    charset="UTF-ASCII";

Unable to deploy

[eofs@cp bottega.eofs.dev]$ go get -u github.com/tierpod/dmarc-report-converter
go: go.mod file not found in current directory or any parent directory.
	'go get' is no longer supported outside a module.
	To build and install a command, use 'go install' with a version,
	like 'go install example.com/cmd@latest'
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.

Running latest. go

[eofs@cp bottega.eofs.dev]$ go version
go version go1.20.5 linux/amd64

Templates should just be part of the compiled binary, ideally.

Having the templates read at run-time means that the current working directory when launching the binary
must have a `./templates/...' with the appropriate content included. If this isn't the case you get:

$ /usr/local/bin/dmarc-report-converter  -config /etc/dmarc-reporting/config.yaml
panic: open ./templates/html_static.gotmpl: no such file or directory

goroutine 1 [running]:
main.loadTemplate(0x8c1c680, 0x1e, 0xc)
	/usr/local/bin/config.go:63 +0xf6
main.loadConfig(0xbff2ec2a, 0x43, 0x2, 0x3, 0x0)
	/usr/local/bin/config.go:90 +0x11c
main.main()
	/usr/local/bin/main.go:30 +0x159
$ 

how about discussion-over-code-review? :)

ReadParseGZIP: unsupported MimeTypeGZIP

Appeared this morning when processing files from the last week: files: ReadParseGZIP: supported mimetypes not found
Seems Yahoo reports are the only culprit at the moment, though some of their files use application/x-gzip, some make use of application/gzip now (spotted for yahoo.co.uk, yahoo.com, yahoo.in, ymail.com).

Not a Go programmer, there seems to be a single MIME type for each possible format (xml, gz, zip) only, so not easy for me to adjust for multi type matching... Thanks ;-)

Play with pico css

https://picocss.com

XML syntax error

DMARC reports from protection.outlook.com are send in .gz format. This extension is skipped while it is not .ZIP
Furthermore, If the extracted raw XML file is added as input it result in the following error:
[ERROR] files: XML syntax error on line 1: illegal character code U+001F, skip

Can this be fixed?
When the XML file is passed to e.g. https://us.dmarcian.com/xml-to-human-converter/ it can be read correctly.

Error invalid char ':' on first line of install

I get an error when running the first line of the installation procedure.

go get -u https://github.com/tierpod/dmarc-report-converter.git

package https:/github.com/tierpod/dmarc-report-converter.git: https:/github.com/tierpod/dmarc-report-converter.git: invalid import path: malformed import path "https:/github.com/tierpod/dmarc-report-converter.git": invalid char ':'

option to output logs to stderr instead of stdout

Hi,
first of all, I really like this converter, very helpful, and thank you for making it.
I just wanted to suggest a small enhancement.
I wanted to do the simplest thing possible, so e.g. run the converter monthly like so, and send the report to myself:

dmarc_report_converter | mailx --subject "DMARC report" [email protected]

so I thought the simplest thing to do would be using these options:

output.file: "stdout"
output.format: "txt"

However, stdout also includes all of the debug logs like:
"[INFO] imap: found 1 messages, 0 unseen"
which I generally don't want to see when reading a report.

I think it would make sense if logs went to stderr (or at least if there was an option for that), and only the actual report would go to stdout.

Of course, I can just save the report to a file, but I think this would be a bit simpler and nice to have.
Let me know what do you think.

Add "total pass/failed messages"

Now option not working

I tried the line specified above. This gives the error:

panic: template: filename:1: function "now" not defined

goroutine 1 [running]:
html/template.Must(...)
/opt/hostedtoolcache/go/1.13.15/x64/src/html/template/template.go:372
main.loadConfig(0x7ffd72018763, 0xd, 0x2, 0x2, 0x0)
/home/runner/work/dmarc-report-converter/dmarc-report-converter/cmd/dmarc-report-converter/config.go:103 +0x5fc
main.main()
/home/runner/work/dmarc-report-converter/dmarc-report-converter/cmd/dmarc-report-converter/main.go:30 +0x1a6[ERROR] loadConfig: yaml: line 20: did not find expected key

Originally posted by @jwnetwerk in #23 (comment)