Memory leak when running long, work intensive Oban background job

[michallepicki]

Memory leak when running long, work intensive Oban background job

[michallepicki]

Hi @unflxw

I believe the issue might lie in all those millions of Ecto queries being instrumented. Could you try running this job with the instrument_ecto: false configuration option, and seeing if the issue persists?

There's no memory leak when using the instrument_ecto: false configuration

If that does fix the issue, adding Appsignal.Tracer.ignore() at the top of the Oban worker's perform method should also fix it, without removing Ecto instrumentation for the rest of the application.

Thank you, this is a good workaround for us for now

[michallepicki]

Or maybe a better Appsignal behavior would be to detect an unusually high amount of spans under a root trace, and log a warning or error when it happens, and stop adding more after that to avoid crashing the server 🤔

[michallepicki]

This is on current Elixir (1.16.0) and Erlang (26.2.1)

[michallepicki]

The memory leak still happens if I try to ignore the worker with ignore_actions: ["Worker#perform"]

[unflxw]

Hi @michallepicki!

Ignoring through the ignore_actions configuration option takes place "after the fact" -- the action is instrumented, then ignored. So it makes sense that the configuration option wouldn't help with the memory costs of instrumenting it.

I believe the issue might lie in all those millions of Ecto queries being instrumented. Could you try running this job with the instrument_ecto: false configuration option, and seeing if the issue persists?

If that does fix the issue, adding Appsignal.Tracer.ignore() at the top of the Oban worker's perform method should also fix it, without removing Ecto instrumentation for the rest of the application.

[unflxw]

Perfect! Thank you for letting us know. Feel free to reach out here, or at [email protected], if you run into any other issues.

[michallepicki]

Sorry, I don't understand why you closed the issue? I think this is a valid usage of Oban and Ecto, and I'm surprised that it's leaking memory (this is a big background job, but ideally we'd like to have the job instrumented anyway)

[michallepicki]

And there's a second slightly overlapping issue here as well, that canceling an Oban job is not handled properly and the VM is stuck with extra memory usage leaked

[unflxw]

@michallepicki I have created #914 about the issue when the Oban job is cancelled via Oban Web, after the details you provided over email clarified the issue. Thank you!

As you described over email, AppSignal is not leaking memory: rather, it is using a lot of memory, and freeing it after the job is done, with an additional delay on the cancellation scenario. To instrument a transaction containing a million Ecto queries, we must temporarily keep information about a million Ecto queries in memory. We have some improvements in the works that will alleviate the memory load, but at the end of the day, monitoring necessarily trades system resources for operational visibility.

Certainly, this is a valid usage of Ecto, but that doesn't mean that it's meaningful to instrument it in this way. Our automatic instrumentation makes specific assumptions which have trade-offs. It works well for instrumenting endpoints that perform a small set of operations, and it doesn't work so well for instrumenting batch jobs that perform a large amount of operations.

Ultimately, AppSignal can't display millions of events in its event timeline anyway, so I would suggest instrumenting this differently. An option could be to disable the Ecto instrumentation, and instead using custom instrumentation to instrument the whole Ecto transaction as a single event. This allows you to keep track of how long the Ecto transaction is taking as a whole, without polluting the transaction with millions of repetitive events.

[michallepicki]

Thank you for writing this up, I hope future improvements will make it possible to use Appsignal for instrumenting batch jobs that perform a large amount of operations. I imagine more state would need to be tracked on your servers, but it may be worth it. In the meantime, do you think there could be some warnings added to the Elixir Appsignal documentation about this, mentioning the Appsignal.Tracer.ignore() workaround? Or maybe this behavior is documented somewhere already and I missed it?