Memray leaks native backtrace state

[tonybaloney]

Memray leaks native backtrace state

[tonybaloney]

So, yep, real issue

Is it wrong that I breathed a sigh of relief?

Thanks for figuring it out. I'll update and retest when there's a patch.

[godlygeek]

Who memory profiles the memory profilers?

Well, to my immense surprise, I can reproduce this. I'll need to spend some time figuring out what is going on. Our working set size shouldn't be nearly this large, so clearly more is being kept alive from one test to the next than we intended.

[godlygeek]

Some initial notes:

• It's not related to flaky; I see the same thing with pytest-repeat, and I'm confident we'd see the same with tests that are just parametrized to run many times
• Memray 1.5.0 is fairly old, but the issue reproduces even with the latest version, so it's not something we've fixed already
• While the process is chugging through memory, I've spotted 65k entries in /proc/$pid/maps, which is... a lot. We've got many copies of every shared library loaded in memory, presumably from either the unwinding or symbolifying that we need to do to support native traces...

[godlygeek]

So, yep, real issue. We're leaking the state associated with libbacktrace's unwinding for each test case. We have to leak it - the API of libbacktrace is designed so that's the only option, surprisingly:

memray/src/vendor/libbacktrace/backtrace.h

Lines 81 to 85 in 98032ae

	Calling this function allocates resources that cannot be freed.
	There is no backtrace_free_state function. The state is used to
	cache information that is expensive to recompute. Programs are
	expected to call this function at most once and to save the return
	value for all later calls to backtrace functions. */

But, we ought to be reusing it from one test case to the next, instead of leaking it between each test and creating a new one for the next test. This has been wrong for a long time, but we never noticed because only pytest-memray reads many different capture files during the lifetime of the program - it's not possible to trigger this misbehavior through the public Memray APIs or the memray CLI entry point, because they only ever operate on one single capture file.

It's easy enough to fix - we just need to move some state to be explicitly global rather than recreating it each time a new capture file is read, and wrap a lock around it. While looking at the surrounding code I spotted some other things that I don't like, though, so I'm gonna try to clean up a few different things here...

[pablogsal]

@tonybaloney the latest release, includes a bug fix for this. Hopefully it works this time. 😊