donmccurdy/zstddec-wasm

ZSTD (Zstandard) decoder for Web and Node.js, using WebAssembly.

zstddec

ZSTD (Zstandard) decoder for Web and Node.js, using WebAssembly.

Installation

npm install --save zstddec

API

import { ZSTDDecoder } from 'zstddec';

const decoder = new ZSTDDecoder();

await decoder.init();

const decompressedArray = decoder.decode( compressedArray, uncompressedSize );

Limitations: The decoder may fail with the error wasm function signature contains illegal type when the uncompressedSize is not known in advance and given to the decode() method. This is presumably a bug in the WASM bindings, which I am not yet sure how to fix.

Contributing

To build the project locally, run:

npm install
npm run dist

To test changes:

npm test

Building from source

Compiled from https://github.com/facebook/zstd/tree/dev/contrib/single_file_libs, with the following steps:

./combine.sh -r ../../lib -o zstddeclib.c zstddeclib-in.c
emcc zstddeclib.c -Oz -s EXPORTED_FUNCTIONS="['_ZSTD_decompress', '_ZSTD_findDecompressedSize', '_ZSTD_isError', '_malloc', '_free']" -s ALLOW_MEMORY_GROWTH=1 -s MALLOC=emmalloc -o zstddec.wasm
base64 zstddec.wasm > zstddec.txt

The base64 string written to zstddec.txt is embedded as the wasm variable at the bottom of the source file. The rest of the file is written by hand, in order to avoid an additional JS wrapper generated by Emscripten.

License

JavaScript wrapper is provided under the MIT License, and the WASM ZSTD decoder is provided by Facebook under the BSD 3-Clause License.

zstddec-wasm's Issues

Remove dist/ folder

Including the dist/ folder in source control is a regular source of confusion for new contributors, because the files are modified when building or testing changes, but shouldn't be included in PRs. These files cannot be reviewed easily, and are meant to be rebuilt by an author when publishing a new release. To avoid this confusion, the dist/ folder should be added to .gitignore and explicitly added to the published files in package.json. Because it's published to NPM, it will still be available from CDNs like Unpkg.

Compile Issues / Guide

Hello, i wanted to reproduce compilation and ran into 2 issues.

First of all zstd dev branch seems to no longer work with this code, i don't know exactly why.
I'm just using zstd tag 1.5.0. (tags 1.4.8/1.4.9 work as well)
1.5.1 and later gives the following error when loading the js:

Uncaught (in promise) TypeError: WebAssembly.instantiate(): Import #1 module="wasi_snapshot_preview1" error: module is not an object or function

This isn't important as the decompression code shouldn't need the most recent version.

If you get Uncaught RuntimeError: memory access out of bounds, then try compiling without -s MALLOC=emmalloc.
The filesize goes a bit up but i have no clue how to fix this.
This is necessary for me both with emcc 3.1.15 and emcc 1.40.1 (tried that one due to it being from around the time this github project was started).

Used the emscriptend docker image to avoid installing that stuff:
https://hub.docker.com/r/emscripten/emsdk

In newer emcc versions --no-entry needs to be added to the emcc options, but that's pretty straight forward.

This should at least give you a working start point for compiling this.

command line used:

wget https://github.com/facebook/zstd/releases/download/v1.5.0/zstd-1.5.0.tar.gz
tar xf zstd-1.5.0.tar.gz
cd zstd-1.5.0/build/single_file_libs
./combine.sh -r ../../lib -o zstddeclib.c zstddeclib-in.c
docker pull emscripten/emsdk:3.1.15
docker run --rm -v $(pwd):/src -u $(id -u):$(id -g) emscripten/emsdk:3.1.15 emcc zstddeclib.c --no-entry -O3 -s EXPORTED_FUNCTIONS="['_ZSTD_decompress', '_ZSTD_findDecompressedSize', '_ZSTD_isError', '_malloc', '_free']" -s ALLOW_MEMORY_GROWTH=1 -o zstddec.wasm

Depending on your application, consider testing -O3 vs -Oz (O3 optimizes for speed, Oz optimizes for binary size).
Now for the 4016 KB json, compressed to 712KB, the decompression time on my 11th gen Intel laptop processor in Google Chrome under Linux was 20ms vs 28ms decompression time with wasm binary size 48KB vs 35KB.
This is both very quick, so it will probably not matter to most people, but someone might find it useful.

wasm function signature contains illegal type when the uncompressedSize is not known in advance

I haven't tried your library yet, and maybe you already tried this, but I wanted to bring this comment facebook/zstd#606 (comment) by a zstd developer to your attention because it looks like a solution to the problem you mention in the readme. 🙂

you can use ZSTD_getFrameContentSize() to find the uncompressed size of the data, and then dynamically allocate a correctly sized buffer yourself.

Publish only ES modules, remove CJS target

Moving forward, this will be an ESM package.