Comments (9)
Would an API like the following work, or do you need something that's per-compressor/decompressor?
LIBDEFLATEEXPORT void LIBDEFLATEAPI
libdeflate_set_memory_allocator(void *(*malloc_func)(size_t),
void (*free_func)(void *));
As for stack allocation, the sizes are too big to portably allocate on the stack. E.g. the size of a libdeflate_compressor
is currently over 1MB.
from libdeflate.
E.g. the size of a libdeflate_compressor is currently over 1MB.
Ah, makes sense then.
Would an API like the following work, or do you need something that's per-compressor/decompressor?
Personally for me, a global one would work fine, although not sure if someone might have a different usecase.
An important thing though is to be able to build the library in a way that wouldn't try to link to stdlib malloc
/ free
and instead use exclusively configured allocator.
zlib solves this by having a Z_SOLO
define which removes dependency on libc altogether. Maybe adding something similar would work here?
from libdeflate.
Btw, since you actually want aligned allocations, perhaps it's best to design API to require them instead? Then you wouldn't need to over-allocate in case host already supports aligned allocations natively.
from libdeflate.
I pushed out commits to the dev
branch (https://github.com/ebiggers/libdeflate/commits/dev) that add support for libdeflate_set_memory_allocator()
and building with make FREESTANDING=1
. Can you check whether it works for you?
I didn't make the API aware of aligned allocations since only the compressor needs an aligned alocation, and it's 32 alignment bytes for a 1MB+ allocation, so an extra 31 bytes doesn't matter.
from libdeflate.
Tried it out on Rust + Wasm integration; generally works well. One awkward bit is that Rust allocator APIs expect size and alignment as extra arguments for alloc
and dealloc
.
While for alignment it's fine to just use 1
, since the libdeflate
will take care of proper alignment anyway, the size part is a bit trickier because I don't have it at the moment I call dealloc
, so I end up providing own allocator functions that allocate sizeof(usize) + actually requested size
bytes duration allocation, write the allocated size to the beginning and return a pointer to the actual data. Then, in the dealloc
wrapper, I subtract sizeof(usize)
from the pointer, read the original size and call the actual dealloc
.
Perhaps libdeflate
could call the free
callback with extra size argument to make this easier, although I suppose most other integrations wouldn't make use of it and it would become just wasteful...
Another unfortunate bit, unrelated to allocation specifically, is that Clang doesn't provide any sysroot for wasm32-unknown-unknown
target, which I'm using, making it completely bare-bones, meaning that not only libc is missing but all the headers, too.
After your changes, the only header left that libdeflate depends on is inttypes.h
.
While there are sysroots provided by Emscripten and WASI and some others, it seems reallly unfortunate to depend on one of them with all the extra APIs they bring, during the build process, when all I need is the integer types definitions.
I wish Clang would've made this easier, but I wonder if it would be acceptable to provide custom definitions for these types under #ifdef __WASM32__
here?
libdeflate/common/common_defs.h
Lines 53 to 60 in 73017f0
from libdeflate.
Actually, besides <inttypes.h>
, libdeflate also includes <stddef.h>
(for size_t
), <stdbool.h>
(for bool
), <limits.h>
(for UINT32_MAX
), and sometimes other headers like <emmintrin.h>
(for SIMD intrinsics). It's difficult to avoid including anything.
Is the issue that <inttypes.h>
is not provided by your compiler, but the other headers listed above are? My system's clang (v9.0.1) actually has an inttypes.h
. Is there a specific clang command I can run to build libdeflate in the way you're building it?
Note that would be possible to replace <inttypes.h>
with <stdint.h>
(for the library only, not the programs), if that would help.
from libdeflate.
Hmm yeah actually stdint.h
, stddef.h
, stdbool.h
all seem fine, which makes the fix even simpler: RReverser@7c7fed4
This doc seems to describe the difference well:
The
<stdint.h>
header is a subset of the<inttypes.h>
header more suitable for use in freestanding environments, which might not support the formatted I/O functions.
As for
Is there a specific clang command I can run to build libdeflate in the way you're building it?
I've been compiling it via a local copy of Rust bindings modified to use freestanding support. These bindings just pass list of files to compile directly to clang, bypassing Makefile.
Now that I've tried with make
and replaced inttypes.h
with stdint.h
, I found one more issue on the current branch: it still tries to link with -lc
:
$ make FREESTANDING=1 CC=clang CFLAGS='-target wasm32'
Rebuilding library due to new compiler flags
CC lib/deflate_decompress.o
CC lib/utils.o
CC lib/arm/cpu_features.o
CC lib/x86/cpu_features.o
CC lib/deflate_compress.o
CC lib/adler32.o
CC lib/zlib_decompress.o
CC lib/zlib_compress.o
CC lib/crc32.o
CC lib/gzip_decompress.o
CC lib/gzip_compress.o
AR libdeflate.a
CC lib/deflate_decompress.shlib.o
CC lib/utils.shlib.o
CC lib/arm/cpu_features.shlib.o
CC lib/x86/cpu_features.shlib.o
CC lib/deflate_compress.shlib.o
CC lib/adler32.shlib.o
CC lib/zlib_decompress.shlib.o
CC lib/zlib_compress.shlib.o
CC lib/crc32.shlib.o
CC lib/gzip_decompress.shlib.o
CC lib/gzip_compress.shlib.o
CCLD libdeflate.so.0
clang: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
wasm-ld: error: unknown argument: -soname=libdeflate.so.0
wasm-ld: error: cannot open crt1.o: No such file or directory
wasm-ld: error: unable to find library -lc
wasm-ld: error: cannot open /usr/lib/llvm-9/lib/clang/9.0.0/lib/libclang_rt.builtins-wasm32.a: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:187: recipe for target 'libdeflate.so.0' failed
make: *** [libdeflate.so.0] Error 1
I've fixed that too by passing -nostdlib
in freestanding mode: RReverser@5cec831
Feel free to just merge that branch (3 commits) into your dev
.
From here, there are couple of issues with linker params (e.g. -soname
/ shared libs are not supported by wasm-ld as well as dependencies on stdio.h
(probably fixed only by excluding programs from freestanding target).
These issues don't affect my particular workflow, since Rust bindings don't use Makefile anyway, but just thought I'd let you know in case you think it's important to fix them anyway.
By the way, I've used libdeflate in a PNG compressor that now powers Squoosh.app and wrote a bit about my experience. In case you might find it interesting: https://rreverser.com/bringing-oxipng-to-squoosh/
from libdeflate.
Thanks for testing. I've merged the custom allocator and freestanding support to the master branch now, along with some fixes including your suggestions for -nostdlib
and using stdint.h
.
Regarding passing the allocation size to free()
: from my point of view it's better to keep the API simple and just require functions that behave like the standard malloc()
and free()
.
from libdeflate.
Sounds reasonable, thank you for your help!
from libdeflate.
Related Issues (20)
- aarch64 build failure HOT 5
- arm build failure with soft float HOT 2
- Improve hc_matchfinder_longest_match() performance on Apple Silicon HOT 3
- Reject all DEFLATE streams that zlib rejects HOT 5
- CMake error with install target, with Visual Studio HOT 1
- Suggestion to publish your GPG keys more HOT 2
- Speed Benchmark: libdeflate vs BWTSATAN, BSC, 7zip, ZSTD, LZ4, LZSSE, BriefLZ, Satanichi HOT 2
- Build type does not default to Release with CMake 3.10 and earlier HOT 5
- Version `1.8` is able to decompress data that version `1.17` cannot, but only on Windows HOT 9
- Comparison to zlib and its forks HOT 2
- POSITION_INDEPENDENT_CODE ON for static compilations HOT 2
- v1.14 - v1.18 deflate compatibility HOT 5
- Decompressing Microsoft DOCX Files in Unreal Engine 5 with Libdeflate HOT 4
- Windows Explorer cannot unpack some files generated with libdeflate (reward offered) HOT 9
- Tests fail on Windows in Release mode with VS platform toolset v141 HOT 1
- Please also generate a static source code archive for release HOT 2
- Libdeflate compressed stream in kunzip fail to decompress HOT 12
- Slow level 0 "compression" HOT 2
- I added stream & multi-thread support for libdeflate HOT 16
- ESA matchfinder HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libdeflate.