Add support for custom allocators

[ebiggers]

Add support for custom allocators

[ebiggers]

Would an API like the following work, or do you need something that's per-compressor/decompressor?

LIBDEFLATEEXPORT void LIBDEFLATEAPI
libdeflate_set_memory_allocator(void *(*malloc_func)(size_t), 
                                void (*free_func)(void *));

As for stack allocation, the sizes are too big to portably allocate on the stack. E.g. the size of a libdeflate_compressor is currently over 1MB.

[RReverser]

E.g. the size of a libdeflate_compressor is currently over 1MB.

Ah, makes sense then.

Would an API like the following work, or do you need something that's per-compressor/decompressor?

Personally for me, a global one would work fine, although not sure if someone might have a different usecase.

An important thing though is to be able to build the library in a way that wouldn't try to link to stdlib malloc / free and instead use exclusively configured allocator.

zlib solves this by having a Z_SOLO define which removes dependency on libc altogether. Maybe adding something similar would work here?

[RReverser]

Btw, since you actually want aligned allocations, perhaps it's best to design API to require them instead? Then you wouldn't need to over-allocate in case host already supports aligned allocations natively.

[ebiggers]

I pushed out commits to the dev branch (https://github.com/ebiggers/libdeflate/commits/dev) that add support for libdeflate_set_memory_allocator() and building with make FREESTANDING=1. Can you check whether it works for you?

I didn't make the API aware of aligned allocations since only the compressor needs an aligned alocation, and it's 32 alignment bytes for a 1MB+ allocation, so an extra 31 bytes doesn't matter.

[RReverser]

Tried it out on Rust + Wasm integration; generally works well. One awkward bit is that Rust allocator APIs expect size and alignment as extra arguments for alloc and dealloc.

While for alignment it's fine to just use 1, since the libdeflate will take care of proper alignment anyway, the size part is a bit trickier because I don't have it at the moment I call dealloc, so I end up providing own allocator functions that allocate sizeof(usize) + actually requested size bytes duration allocation, write the allocated size to the beginning and return a pointer to the actual data. Then, in the dealloc wrapper, I subtract sizeof(usize) from the pointer, read the original size and call the actual dealloc.

Perhaps libdeflate could call the free callback with extra size argument to make this easier, although I suppose most other integrations wouldn't make use of it and it would become just wasteful...

Another unfortunate bit, unrelated to allocation specifically, is that Clang doesn't provide any sysroot for wasm32-unknown-unknown target, which I'm using, making it completely bare-bones, meaning that not only libc is missing but all the headers, too.

After your changes, the only header left that libdeflate depends on is inttypes.h.

While there are sysroots provided by Emscripten and WASI and some others, it seems reallly unfortunate to depend on one of them with all the extra APIs they bring, during the build process, when all I need is the integer types definitions.

I wish Clang would've made this easier, but I wonder if it would be acceptable to provide custom definitions for these types under #ifdef __WASM32__ here?

libdeflate/common/common_defs.h

Lines 53 to 60 in 73017f0

	typedef uint8_t u8;
	typedef uint16_t u16;
	typedef uint32_t u32;
	typedef uint64_t u64;
	typedef int8_t s8;
	typedef int16_t s16;
	typedef int32_t s32;
	typedef int64_t s64;

[ebiggers]

Actually, besides <inttypes.h>, libdeflate also includes <stddef.h> (for size_t), <stdbool.h> (for bool), <limits.h> (for UINT32_MAX), and sometimes other headers like <emmintrin.h> (for SIMD intrinsics). It's difficult to avoid including anything.

Is the issue that <inttypes.h> is not provided by your compiler, but the other headers listed above are? My system's clang (v9.0.1) actually has an inttypes.h. Is there a specific clang command I can run to build libdeflate in the way you're building it?

Note that would be possible to replace <inttypes.h> with <stdint.h> (for the library only, not the programs), if that would help.

[RReverser]

Hmm yeah actually stdint.h, stddef.h, stdbool.h all seem fine, which makes the fix even simpler: RReverser@7c7fed4

This doc seems to describe the difference well:

The <stdint.h> header is a subset of the <inttypes.h> header more suitable for use in freestanding environments, which might not support the formatted I/O functions.

As for

Is there a specific clang command I can run to build libdeflate in the way you're building it?

I've been compiling it via a local copy of Rust bindings modified to use freestanding support. These bindings just pass list of files to compile directly to clang, bypassing Makefile.

Now that I've tried with make and replaced inttypes.h with stdint.h, I found one more issue on the current branch: it still tries to link with -lc:

$ make FREESTANDING=1 CC=clang CFLAGS='-target wasm32'
Rebuilding library due to new compiler flags
  CC       lib/deflate_decompress.o
  CC       lib/utils.o
  CC       lib/arm/cpu_features.o
  CC       lib/x86/cpu_features.o
  CC       lib/deflate_compress.o
  CC       lib/adler32.o
  CC       lib/zlib_decompress.o
  CC       lib/zlib_compress.o
  CC       lib/crc32.o
  CC       lib/gzip_decompress.o
  CC       lib/gzip_compress.o
  AR       libdeflate.a
  CC       lib/deflate_decompress.shlib.o
  CC       lib/utils.shlib.o
  CC       lib/arm/cpu_features.shlib.o
  CC       lib/x86/cpu_features.shlib.o
  CC       lib/deflate_compress.shlib.o
  CC       lib/adler32.shlib.o
  CC       lib/zlib_decompress.shlib.o
  CC       lib/zlib_compress.shlib.o
  CC       lib/crc32.shlib.o
  CC       lib/gzip_decompress.shlib.o
  CC       lib/gzip_compress.shlib.o
  CCLD     libdeflate.so.0
clang: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
wasm-ld: error: unknown argument: -soname=libdeflate.so.0
wasm-ld: error: cannot open crt1.o: No such file or directory
wasm-ld: error: unable to find library -lc
wasm-ld: error: cannot open /usr/lib/llvm-9/lib/clang/9.0.0/lib/libclang_rt.builtins-wasm32.a: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:187: recipe for target 'libdeflate.so.0' failed
make: *** [libdeflate.so.0] Error 1

I've fixed that too by passing -nostdlib in freestanding mode: RReverser@5cec831

Feel free to just merge that branch (3 commits) into your dev.

From here, there are couple of issues with linker params (e.g. -soname / shared libs are not supported by wasm-ld as well as dependencies on stdio.h (probably fixed only by excluding programs from freestanding target).

These issues don't affect my particular workflow, since Rust bindings don't use Makefile anyway, but just thought I'd let you know in case you think it's important to fix them anyway.

---

By the way, I've used libdeflate in a PNG compressor that now powers Squoosh.app and wrote a bit about my experience. In case you might find it interesting: https://rreverser.com/bringing-oxipng-to-squoosh/

[ebiggers]

Thanks for testing. I've merged the custom allocator and freestanding support to the master branch now, along with some fixes including your suggestions for -nostdlib and using stdint.h.

Regarding passing the allocation size to free(): from my point of view it's better to keep the API simple and just require functions that behave like the standard malloc() and free().

[RReverser]

Sounds reasonable, thank you for your help!