Feature request: weekly or monthly report

[tierpod]

Feature request: weekly or monthly report

[tierpod]

Hi @planetahuevo ! You are welcome, I'm glad that my little project can be useful for somebody.
I'm thought about adding weekly/monthly reports, but still can't decide how to do it.

Each dmarc report contains data for one day. At the moment each dmarc report parses independently. So, example of my workflow is: crontab executes dmarc-report-converter every day and saves HTMLs without any aggregation. I've configured nginx to serve this files.

If we want to make weekly/monthly reports, we have two ways:

• Execute dmarc-report-converter every week/month (not days). In that case, it have to aggregate data from many dmarc reports in memory and make complex HTMLs.
• Use external database to store daily data. In that case, I have to add dependency on database, and write web application for visualize reports from this database. I think, it is quite complex way.

Also, dmarc-report-converter can render reports as json. So there is another way - save data to elasticsearch and use kibana for visualize data.

I will appreciate if you can give me ideas about it.

[planetahuevo]

Hi!
I have been thinking about this for a while, before I found your project, so this is what I think:

• Using a external database makes no sense. It complicates the project, the server, the backups, etc. We already have the data on the reports, so we do not need to store the data again.
• Monthly reports looks ok, but I think it won´t be necessary, a weekly report is good enough and that makes everything easy.

We can also take another approach, instead of using a date to select the reports, which will force your system to scan all of them, find the ones you want and discard the rest, it makes more sense to extend your tool to scan multiple reports at once, with a number of reports.
For example. If you do nothing, it will work as it is now, take the report and generate the html.
But you could add a flag (for example --multiple=7) that indicate to the system that it need to collect 7 reports and generate the html after that.
I am not sure how the system works now, but you could limit the number to the total of reports to process, or you could get all reports (as the system does now) and create one html after 7 reports.
If there are less than the number, you just process all of them.

Another alternative that I like less, is to just mark the flag without number, and then the system process all reports and generate 1 html with all the info.

One important thing here, is to make sure that we put together all reports from the same domain so we do not mix.
I am not a developer, so I have no idea how can you do this in GO, but I suppose that you can add them to memory and when you have 7 of the same domain, you generate the report for that domain.

You will also need to create another filename on the config for multiple reports, but appart from that, config can be the same.

About elasticsearch, there is already another github project that use that with Kibana, I found it much more complicated that this.

One more thing. There is a flag on the DMARC reports that allow to change the frequency of the reports. ri=
I have setup mine today to a 7 days report, so maybe we can get the 7 days report directly from the source, and that will solve the problem. :)
We will need to wait 7 days to see what happen.

I would love to be able to help you with the code, but I know nothing about go.

[tierpod]

Thank you for reply!

There is a flag on the DMARC reports that allow to change the frequency of the reports. ri=

As I understand from rfc, provider of reports can ignore this option :(

DMARC implementations MUST be able to provide daily reports and SHOULD be able to provide hourly reports when requested. However, anything other than a daily report is understood to be accommodated on a best-effort basis.

But you could add a flag (for example --multiple=7) that indicate to the system that it need to collect 7 reports and generate the html after that.

It is interesting idea 👍

I will think about that issue and will try to implement one of these methods.

[planetahuevo]

Great :)
On the meantime the daily ones are great.

[tierpod]

I am testing new version now. I had to change some internal logic to achieve this functionality. When I make sure that everything works as expected, I will release new version and will write upgrade notes.

[planetahuevo]

Amazing. I have been a little busy on the past months but I would love to help you testing this. Let me know if you need help with that.

[tierpod]

If you want to test it and can build new version from sources, have a look at issue#14-2 branch

Please read UPGRADING.md, you have to update templates and edit configuration file.

[tierpod]

The branch was merged, new release v0.5