jemalloc probably won't work well on aarch64-linux

[facebook]

jemalloc probably won't work well on aarch64-linux

[thoughtpolice]

Please note that this isn't causing huge problems for me. Yet. But eventually I want to distribute aarch64-linux builds of my Nix package for buck2. So, this is mainly just to catalogue the issue since I suspect any movement towards an actual solution will require a bit of stakeholder input, and because someone else may eventually run into it.

[ndmitchell]

We have now disabled jemalloc everywhere apart from Mac/Linux, since it doesn't play well on other OS's like Illumos #120. We also disable jemalloc if you are doing a build of Buck2 with Buck2, mostly because we haven't setup Buck2 to build Jemalloc. Is that enough?

[thoughtpolice]

That's close, but an aarch64-linux package for e.g. NixOS (just as an example) won't be able to work cross-system unless we also turn off jemalloc there, too.

Would a patch to make jemalloc an optional Cargo feature be accepted? Then we could just turn it off, and it could default to on, to leave the current behavior. I could write this.

It might also be worth exploring if other allocators can boost performance while more gracefully handling these requirements.

[ndmitchell]

Internally we'll probably always use jemalloc, and it's been well tuned, so I am suspicious there is anything else out there with higher performance. But if you find something, we'd switch.

Happy for it to be an optional Cargo feature. Note that it is already gated on a few things. Alternatively have you tried asking upstream at jemalloc, in case they can have a fallback path for the NixOS example?

[thoughtpolice]

I believe Jason has stated multiple times that the page size being (effectively) part of the API isn't going to change because it would require a large rework; see jemalloc/jemalloc#467. That said, apparently jemalloc can support sizes smaller than the baked in page size (e.g. build for 16k, run on 4k using the --with-lg-page option) — and it looks like NixOS enables that feature! Which is nice, but...

It would probably be good to still add a flag for places that don't enable this, and also because packages like jemallocator tend to do things like build their own copy of jemalloc as part of their build.rs, which then thwarts usage of the NixOS version with the appropriate flags. While this can (and probably should) be fixed on our side, it's likely not the last time something like this will happen.

[ndmitchell]

A feature flag to disable jemalloc seems reasonable, and like it would solve all the issues here. Patch welcome.