teamnewpipe/crashreportimporter

NewPipe Crash Report Importer for Sentry

crashreportimporter's Issues

Stream processing instead of IMAP fetching

The way this importer works is very expensive. The mail server has to put mails into a mailbox which is then queried via IMAP. This not only takes time but generates high load on the mail host and involves saving many copies of the data.

This importer should ideally do some sort of "stream processing", that is, have mails forwarded from the MTA directly, bypassing the MDA. The usual checks, e.g., for spam, should still be run.

This could either be implemented by using some sort of "pipe alias", as some MTAs provide (i.e., calling a script if a mail, or maybe even easier, this project could host its own SMTP service and accept mails this way. This is also how many services are included in mail pipelines (accept mails via SMTP and return them to the main MTA to a special SMTP endpoint).

Going for our own SMTP service also makes integration a lot easier, as there's no more mailbox required, mails can simply be spam-checked and then forwarded to a special, non-public address.

Remove unused "ip_range" field from Sentry

It was discussed and removed over NewPipe, in the issue TeamNewPipe/NewPipe/issues/1148.

@TheAssassin You mentioned that you were going to remove them (they're still here), maybe this was forgotten?

Only accept current version(s)

We are getting a bunch of reports for old versions. Who cares about people reporting issues which happend versions ago? There is a check to sort out errors which occurred more than 30 days ago, but that does not help when people run outdated versions.
We should probably only accept the latest version or the current major versions (like all reports from 0.18.x)

CC @wb9688

Improve stack traces visualization in Sentry

For example, wrapped exceptions, very common in Java, are not displayed at the moment and all the traces are just glued together with not indentation. It makes identifying the causes and other things a bit more difficult/impossible since the message from the cause is not displayed.

A very straightforward example of this issue would be this event: issues/4156/events/62029, which the cause would very likely be a exception from the IOException family.

In fact, I bet a lot of the issues that are in Sentry right now are of this kind, a network issue that abruptly left the extractor in a bad state (which should be detected and handled correctly to avoid a full on app crash).

I think it's mainly caused by the way the custom stack trace parsing works, maybe this should be done on the client side, which then would send the information in a sane format and avoid regex in the first place?

Don't know much about Sentry, but would be willing to fix some time in the future if nobody takes care of it.

New access rights for new members

Hi @TheAssassin,

as we have some new Members at TeamNewPipe, namely @yausername @karyogamy @nv95 id like to suggest them to get access to sentry.schabi.org. What do you think, could you handle this?

Test this project

This project is completely untested and hackish, but is to some extent "critical infrastructure" of our project. We should introduce unit and/or system tests.