mattdesl/mp4-wasm-encoder

mp4-wasm-encoder

💡 Update - Project Moved

See the mp4-h264 module which wraps most of this demo up as a ES module.

What follows below is early R&D / proof-of-concept.

---

Fast client-side MP4 encoding demo based on a fork of Trevor Sundberg's npm library. Currently only works in Chrome due to OffscreenCanvas.

[ Demo Link ]

• Creates a H264-encoded MP4 video in the browser
• Can be used for long videos (thousands of frames)
• The WASM+JS dependency is ~200 KB before gzip (much smaller than ffmpeg.wasm)
• Uses WASM SIMD if enabled (Chrome only; first enable #enable-webassembly-simd in about:flags)
• Uses OffscreenCanvas to speed up rendering in a web worker (Chrome only)

A 5 second 1920x1080 60FPS MP4 takes about 7 seconds to encode with Chrome and SIMD enabled (may be faster/slower depending on your hardware).

How is this fast? (for a browser)

This is mostly based on Trevor Sundberg's work with h264-mp4-encoder (thanks!). Here, I've mostly just been exploring how to improve performance:

• Use workers and OffscreenCanvas for rendering
  • Save frames with transferToImageBitmap() then into a RGB buffer with WebGL
  • Convert RGB to YUV in the worker and then post that to the main encoder thread
• Uses a true WASM file, and WASM SIMD where available
• Sets Emscripten memory directly to avoid passing any array buffers to C/C++
• A few additional tweaks and new flags added to my C/C++ fork
• I'm also using an entirely different library to build the MP4 container – shrinking the final WASM+JS size from ~700KB to ~200KB. Compare to the pure-JS version of h264-mp4-encoder which is ~1.7MB.

How it could be faster?

It's still pretty slow compared to native, some ways it could be faster/cleaner:

• Use a second worker (thread) to handle encoding, this might not speed things up much but at least will take a load off main UI thread
• Ensure that WASM version of minih264 library is indeed taking advantage of SIMD (lots of NEON code that doesn't compile there)
• Open to other ideas! Please create an issue if you think you see any ways to make it faster.

How can it work on FireFox, mobile phones, etc?

The h264-mp4-encoder already works on most browsers, this demo is just to see how more advanced browser features could make it faster: OffscreenCanvas, dynamic imports in a web worker, WASM, and SIMD.

Just Give me the WASM!

In the h264 folder is a drop-in WASM+JS files for SIMD and non-SIMD supported environments. These must be imported as an ES module, see ./encoder/main.js, but otherwise you can get it running without any dependencies or build tools.

Feel free to use the WASM for your own purposes, though this repo won't be maintained like a typical library, so you might rather use the original non-WASM version at h264-mp4-encoder library which will probably receive more frequent updates.

In future, I plan to wrap some of this R&D work in a more 'official' MP4+GIF encoder library that will receive regular updates.

License

MIT, see LICENSE.md for details.

mp4-wasm-encoder's Issues

Compiling code with NEON intrinsics to Wasm SIMD

Ensure that WASM version of minih264 library is indeed taking advantage of SIMD (lots of NEON code that doesn't compile there)

Really cool work :)

I'm wondering if you tried https://emscripten.org/docs/porting/simd.html#compiling-simd-code-targeting-arm-neon-instruction-set. @seanptmaher did some work in SIMDe + Emscripten (cc @tlively), which transparently supports a subset of ARM NEON intrinsics. This might get you over some of the compilation errors, and could help with performance too.

Also, I work on WebAssembly SIMD in V8, if you have questions or bug reports, please file an issue on me ([email protected]) at https://bugs.chromium.org/p/v8/issues/entry. Thanks!

Use WebCodecs for a faster encoding!

The README says to create an issue for suggestions on how to make it faster :)

https://web.dev/webcodecs/ offers a way to directly use the hardware encoders for the various codecs that Chrome supports. It's in origin trial right now, so feedback is appreciated!

https://github.com/WICG/web-codecs
https://bugs.chromium.org/p/chromium/issues/entry?components=Blink%3eMedia%3eWebCodecs