thermondo/log-reporter

heroku sentry reporter

motivation

In our Heroku apps we saw that developers tend to overlook some Heroku errors. Since we're using sentry for error reporting, we had the idea of writing a small service that creates sentry issues for each error.

We now open the codebase up, with the goal to evolve this service into something that is useful for more companies.

design

Heroku supports HTTPS log drains. On each of your Heroku application you can configure a new log drain pointing to this log-reporter service. This leads to us getting all log messages.

The service then parses the logs for specific errors and generates a sentry report out of it.

So the sentry error grouping works we try to replace some patterns in the path which we think represent identifiers.

errors

Currently we parse & report:

• H12 - request timeout
• R10 - boot timeout

deployment

Deployment works via heroku.yml and the linked Dockerfile.

We're currently running the service on one small dyno, and it's handling around 3k RPM without any issues.

configuration

Configuration can be set using environment variables, more details in the config module.

the service itself

• PORT (mandatory): normally set by Heroku, the port the webserver runs on
• SENTRY_DSN (optional): the sentry DSN where the service should send its own errors to. The sentry client library additional reads some other environment variables like SENTRY_ENVIRONMENT.
• SENTRY_DEBUG (optional): activates sentry debug logging

mappings for services

You need to add this service as a log-drain to each sender-service:

$ heroku drains:add https://xxx-log-reporter.herokuapp.com/
...
$ heroku drains
[this will print the _drain token_ you need]

Then you can add any mapping as a new environment variable SENTRY_MAPPING_XXX where XXX can be replaced with any service name.

The value for this mapping contains of 3 pieces, separated by |:

• the logplex token from above
• the destination sentry environment name
• the destination sentry dsn

Example value:

d.xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx|production|https://[email protected]/9999999

current limitations

This service is running in production at thermondo, but has some pending improvements, and also contains some internal thermondo specifics around the replacement patterns in paths.

You can use the service at your own risk, if these specifics are fine with you.

log-reporter's Issues

replace some more patterns in URLs

• WO-IDs (WO22144D)
• other reference-IDs? (BV12345...)
• salesforce-IDs (0WO1i000003COEnGAO), find regex pattern for 15-char and 18-char version
• offer-ids (0680435-01, 0680435-01-A)

router metrics, add `transaction` / `route` as tag

using the same logic as for timeout errors we can add a "route" tag.

report R10 (boot timeout) as sentry error

Example:

2023-04-25T12:40:28.796486+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
2023-04-25T12:40:28.822216+00:00 heroku[web.1]: Stopping process with SIGKILL

(optionally?) report R12 (exit timeout) to sentry

add request method to title & fingerprint

use separate threadpool for log-parsing & reporting

the currently used tokio blocking pool is not designed for CPU intensive work because it will just add new threads when needed, assuming most threads are just waiting for I/O (see also the spawn_blocking documentation).

We should change this to a separate threadpool (possibly rayon_core::ThreadPool) which is limited to the CPU count.

To keep in mind:

• graceful shutdown needs to be handled, we should try to wait for pending parsing & reporting when shutting down the server

dyno count metric should report last value between scaling events

Currently the metrics look like this:

We could just report the last value between scaling events.

improve SFID parser with _real_ definition

• https://www.radnip.com/has-salesforce-hit-an-architectural-limit/#comment-1818467455
• https://salesforce.stackexchange.com/questions/1653/what-are-salesforce-ids-composed-of

add sentry performance tracing

based on tracing-spans (instrument)

performance: find a way to stream the body into the parser

with ~150 lines of logs we "only" get around 2k RPS, with an empty body we get 50k RPS for the web-server component.

I'm not sure if this is possible with axum, but it would be good if we wouldn't need to convert the body to a byte buffer in the web handler.

So either pass the body to the background task and buffer it there, or even convert the parser into a streaming parser.

move log-parsing & error creation into thread

• the request doesn't have to wait for the parsing etc to be finished
• see also 25fbec5

to keep in mind:

• any error happening in the task should still be reported to sentry
• we need a graceful shutdown so on restarting the dynos we would try to wait for the existing tasks to be finished before shutting down the app. I remember tokio default behaviour for async tasks is to do nothing, so cancel them. I'm not sure what is true for the blocking/sync threadpool.

we could use:

• tokio::task::spawn_blocking, when we solve graceful shutdown somehow
• or use our own thread / threadpool. Handling graceful shutdown could be easier there.