zaplib/zaplib

⚡ Zaplib is an open-source library for speeding up web applications using Rust and WebAssembly.

⚡ Zaplib

Zaplib is an open-source library for speeding up web applications using Rust and WebAssembly. It lets you write high-performance code in Rust, alongside your existing JavaScript code, using simple APIs.

To get started, view the docs.

License

Zaplib is distributed under the terms of both the MIT license and the Apache License (version 2.0).

See LICENSE-APACHE and LICENSE-MIT for details. Third party license notices are available in LICENSES-THIRD-PARTY.

zaplib's Issues

Docs look bad on mobile

Something about the styling for the open source core team section makes my face stick out on the right, breaking the responsive design.

Enforce JS buffer restrictions

On the JS side of the JS-Rust bridge, we have some restrictions on what you can do with the buffers that you get, but we don't enforce these restrictions, which can lead to tricky bugs (race conditions; corrupted memory; etc).

We should add a mode to enforce these restrictions (enabled by default in development, disabled by default in release builds).

These are the restrictions:

1. Read-only buffers. We can periodically checksum these.
2. Unusable buffers. These are buffers that have been passed to the Rust side (and therefore ownership is relinquished), or buffers that point to WebAssembly memory after the Rust side has panic'ed (in which case the memory might be corrupted). In both cases we can overwrite the buffer with dummy data, and also periodically checksum it, to make sure it's not used in any meaningful way.

Deploy process for bleeding edge users

Since we have users who make PRs to this repo and then want to use the changes in their own repo, we should have a better deployment story. Ideally we want to be able for users to point to a specific Zaplib git commit and use that (both on the Javascript and on the Rust side).

We also want a good story for people who care more about stable releases, but we can worry about that later (maybe make a ticket for that once we've solved the bleeding edge use case).

Run CEF test_suite in CI on MacOS

We don't use CEF a lot but we should at least make sure the test suite doesn't break!

Move server from Python 2 to `cargo zaplib`

Python 2 is old and unsupported; we've had multiple people complain about this.

callRustInSameThreadSync on main thread

This is is critical for incremental development. The problem is that we can't use Atomics.wait on the main thread, which means that we can't really do multithreading, unless we implement Atomics.wait using busy-waiting in the Rust standard library.

A quicker fix would be to add a mode for running Zaplib single-threaded on the main library, but then we'd prevent people from seamlessly moving into multithreading later. But we can probably build this much more quickly, so it's worth exploring.

Better error messages for unsupported Zaplib stdlib / JS bridge calls

Often the errors are just "Not yet implemented", or even less helpful than that. We should go through all the unsupported behavior (e.g. by going through the compatibility tables in docs) and update the error messages to at least link to a Github Issues ticket + encouragement to post about it in Slack (so we know which issues to prioritize).

Add "contributing" guide

If people want to contribute we should make that as easy as possible. Some things to think about (though we can split them out into separate tickets where necessary):

• How to engage with the community (Slack, how to ask questions, how to submit bug reports, etc)
• Be welcoming
• Community guidelines / code of conduct
• Templates for issues / PRs
• Running local version of cargo zaplib
• Information about our CI / test suite
  • Formatting / linting / testing / etc
• Other local tools

Make createReadOnlyBuffer and createMutableBuffer synchronous on the main thread

Since we now have a Wasm instance running on the main thread!

• Also, when we do this we can make the JS bridge tutorials use callRustSync, simplifying them a bit.

Allow registering callJs functions in a WebWorker

Not super important right now. At some point this is probably useful.

Fix test_suite in all browsers

• Firefox: seems to work locally, but in Browserstack the arc deallocation call times out (garbage collector might not run?). Maybe we can force GC through https://stackoverflow.com/a/41587598 and about:memory? For other browsers this might be useful: https://stackoverflow.com/a/46288544
• Safari (desktop+ios): actual bug where we're trying to send the wasm module+memory from the main worker (which is not supported in Safari for some reason).
• Samsung Galaxy crashes on the examples after a while; should probably restart the browser.

When fixed, enable the commented out tests for these in Browserstack.

Storybook support

Lots of people use Storybook for their development, and even for screenshot tests. Zaplib should be able to run in that context.

• A challenge here is getting it to work with COOP/COEP headers.
• We should document how to set this up.
• Prevent regressions in CI.

Better website deploy process

Right now you have to run the examples locally which is not great.

• We need to send COOP/COEP headers, which Github pages doesn't appear to support.
  • We could use a service worker to do this, though it requires a reload (but might be okay).
  • Or we can move to Netlify or something.
• We should make sure our examples work well on mobile too (since people might casually come across them on their phone/tablet).
• Better process for deploying docs (e.g. Github Actions or something else)

"Set up in your repo" guide

We already talk a bit about how to set up local tooling with our examples, in basic_tooling.html. But not yet how to integrate into an existing repo.

With things like

• How to set up Rust
  • For development
  • In CI
  • rust-toolchain.toml
  • rustfmt.toml
  • workspace.code-workspace recommendations
• How to set up JS
  • With Webpack
  • With Jest/NodeJS (see also #18)
  • With Storybook (see also #19)
• Versioning
  • What is our versioning scheme?
  • How to use specific git commits in the zaplib repo (see also #26)
    • Rust: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories
    • JS
• "Hello worlds" examples
  • From Rust at startup
  • Through a JS function call to Rust
  • By rendering

Needing a dummy instance variable in shaders

If you don't need multiple instances of a geometry, you currently need a dummy instance variable. That's kind of confusing. It's not super common though, so for now we can maybe deal with it with just documentation?

Alternatively we can address this more properly in the framework itself, but it's a fairly deep change unfortunately.

Add some more Slack channels

Suggestion of @ChristianBlank, which I like a lot:

"1. We created an introduction channel, where everyone should introduce themselves. The idea goes that they introduce themself, what they are working on, and to leave a sentence what other people should connect over with. We use Dots that has a GreetBot integration to send users a direct message and asks to introduce themselves: https://www.producthunt.com/posts/dots-7
https://zaplib.slack.com/apps/A2LJCT3GR-greetbot?tab=more_info

2. We also have a channel for projects and showcases where people can post what they are working on, post their progress what they are working on, and can get inspirations and feedbacks on things they are building.

The main reason for dedicated channels is that it is grouped for new users to join and scroll through.

As an extra (and that might first need a critical mass) you can also create a hiring channel for people that are looking for RUST contractors or full time employees, especially when you grow and there are engineers that are more specialized in utilizing your library. We are planning to do that for electrical engineers once we are out of private beta."

Run zaplib/examples/ in CI

With at the minimum running the test_* examples in Wasm in Chrome, but ideally in more browsers and native platforms.

We'll have to add actual assertions to tests, and/or use screenshot tests (especially for the other examples / tutorials).

• Screenshots of static examples
• Pause animations for other examples
• Add assertions to multithread example
• Make test_shader_2d_primitives work in wasm context (not just CEF)
• Better hook into when rendering is done so we don't have to wait 2 seconds
• Remove randomness from example_charts

Docs: Zaplib vs Wasm-Pack

https://github.com/rustwasm/wasm-pack

Docs versioning

It can be quite annoying if you're on an older library version but the online docs are on the newest version. It would be good to make it easy to view the docs associated with the current library version.

Some ideas:

• Host old versions of the docs online, by git hash. We could even print a link to these docs in JS/Rust on startup (in development mode).
• Add a command to cargo zaplib to locally serve the docs for the version / git hash that the current repo depends on (can grab this either from Cargo.toml or package.json?)

Expands on #26.

Support WebWorkers in CEF JS runtime

Not super important right now, but at some point we should make the CEF JS runtime on par.

Throw an error when calling `on_call_rust` without being in a JS context (Wasm/CEF)

This is otherwise very confusing, since you end up with an app that just hangs. Per https://github.com/Zaplib/zaplib/pull/41/files#r806011983

Jest support

Lots of people use Jest for their unit tests; Zaplib should be able to run in that context.

• We should document this.
• Prevent regressions in CI.

Make some Cx APIs available in threads

• Timers; cx.start_timer
• Profiling; cx.profile_start (we already have UniversalInstant; maybe do more with that?)

Custom Rust stdlib

• Ideally we build a custom standard library, so that more libraries work out of the box with Zaplib.
  • Merge Universal* APIs into that new stdlib.
• Also, we can implement busy waiting in place of the atomics wait instructions, when on the main thread (since blocking is not allowed there).
  • This is also required to make a bunch of APIs work on the main thread; e.g. #72.
• We might want a custom allocator: #103

Lots of smaller docs improvements ("catch all" docs ticket)

Let's put a grab bag of small improvements here as we come across them (but bigger than "lets just fix this right away as we encounter them). Can split into separate tickets if some of these turn out too big.

• Split Rendering 3D Meshes into multiple pages + make clear that this is a full "incrementally port JS to Rust" example
• Add clearer information on what the stability of things is, and what we're focusing on (e.g. version numbers / labels).
• General copyediting (make sure spelling and grammar is correct).
• More information about our history / vision / etc.
• More information about who this is for (both right now and at different stages in the future).
• Info about headers for getting SharedArrayBuffer to work.

Fix OSX memory leak

It leaks massive amounts of memory on draw().

JS<=>Rust bridge tests in CI

We currently don't run these tests. Ideally we run them in a few different browsers, but even just a single browser would be a good start.

Docs: Zaplib vs wasm-bindgen

Update dependencies / dependabot / etc

Let's make it so our dependencies don't get out of date by setting up some infrastructure for this.

Add links to Rust / wasm documentation

We're targeting Typescript developers, but we don't actually make any recommendations for how to learn about Rust and WebAssembly more broadly! A reading list and some guidance on when the best time is to read things would be helpful.

Suggested by @dget; ty!!

Support full Zaplib stdlib in JS (user) workers

• Spawning threads (universal_thread)
• Reading local files (UniversalFile)
• Posting signals (Cx::post_signal)

Add an ability to re-initialize Zaplib

If Rust panics, we currently enter an unrecoverable state across the whole library. We should add the ability to reload without forcing a page refresh.

One alternative way to solve this would be to move to a class/instance approach, where we don't try to recover, but a user can just create a new zap instance for future work.

Regularly bust the Docker cache

In the future we might want to bust the cache periodically, e.g. using an ARG CACHE_BUST in the Dockerfile-ci, combined with --build-arg CACHE_BUST=$(date +%Y-%m-%d). If we do that we should couple it with a nightly build, so we don't get one-off super long builds in our regular PRs. And by itself this would just test compilation; we might also want to automatically update the docker commit hash in our ci.yaml.

Remove NodeJS+Yarn as a dependency for running tutorials

• Tag auto-published npm packages on the main branch with latest
• Update html files to use unpkg for zaplib@latest by default, except when adding a ?local query param (for local development / CI)
• Add section in "Contributing" to explain the NodeJS+Yarn dependency and the ?local query param.

Examples should run on all platforms

Windows is broken; Linux hasn't been tested. (We can use https://lightsail.aws.amazon.com/ls/webapp/home/instances for both)

Prevent regression using CI.

Writing local files

E.g. dragged in files.

Quickly drawing 2d lines

Is surprisingly not very easy! We might want to abstract some logic from our charting code. Some things to keep in mind:

• Clipping should ideally be done on the GPU (just like our axis-aligned boxes).
• Drawing an axis-aligned box for a line that is 45 degrees oriented can be pretty slow since there are a lot of unused pixels.
• Geometry is faster than instancing but it probably doesn't matter too much for most use cases.

Include unminified files in package + document source maps in Webpack

Including full JS source / sourcemaps will make local development much easier when running into something confusing in the framework.

Expose low-level (WebGL-like) rendering API

Our rendering API is pretty high level. That makes it harder to experiment with different approaches, or to build stuff specific to particular users.

By exposing a WebGL-like API, people can opt out of our more ergonomic APIs when necessary. And we can experiment with different approaches to our APIs in "user space".

This would likely also mean that we should split out the current 2d rendering API a bit. The Makepad folks already did that (https://github.com/makepad/makepad/tree/98d716cd0fa1df1af74fa40bc4d5bda0070965f1/platform/src/draw_2d) so we can do something similar.

Multithreading issues

Spawning threads all the time is a bit of an antipattern (you should use a thread pool) so I don't think this should be part of https://github.com/Zaplib/zaplib/milestone/1, but there are some issues with threading that should be addressed at some point:

• test_multithread deadlocks in Chrome after a while (not sure about other browsers).
• In Safari 15.2 test_multithread doesn't work well at all.

There are some deeper issues that might require language changes, and are less urgent, but I'll mention them here anyway. We might want to split this out to a different ticket at some point.

• Threads leak memory since we never deallocate the TLS/shadow stack (see also this issue).

Run "Other tests" in test_suite in CI

These require a page reload since they deliberately crash the wasm instance.

Convenient `cargo zaplib` command for rebuilding Rust + rebuilding JS + server

Currently you need to open 3 different tabs for all these things. Might be nice to unify.

That said, when using Zaplib as a library, you probably already have your own server, so you just need the cargo watch -- zaplib build --workspace command. Might still be nice to add a convenience command for just that though!

Add flow typing to the library

Add accurate types for Flow JS based projects.

After doing an initial spike, I struggled to get this working by simply including *.js.flow files in our distributed package. Additionally, I found our project incompatible with flowgen for automatic generation.

The best bet would be to generate a module declaration somewhere else and publish it as its own package or add it to https://github.com/flow-typed/flow-typed.

I'll add a partial example of how our library is currently working around this.

Docs Improvements

Allow single-threaded usage

Currently Zaplib needs to load a worker for the main Rust thread. Often it's convenient to just throw a little WebAssembly in where you need it, e.g. synchronously on the main thread (#51) or standalone in a WebWorker. That is a bit hard right now, which makes it harder to start using Zaplib for small things.

Some benefits of allowing single-threaded usage:

• Easier to get started
• Better browser/runtime support (older browsers, nodejs, etc)
• Faster loading
• No need for COEP/COOP headers (since we don't need SharedArrayBuffer)
• Fewer unstable compiler features

Should probably be done after #51. We can start by omitting rendering support in this mode, and adding that back later.

"edit this on github" link on all docs pages

Thanks @sw-yx for the suggestion!

Better representation of typed arrays in the browser console

@disambiguator "Would be cool if we could clean up our monkey patching such that normal TypedArrays don't get represented as ZapTypedArrays, not sure how though. i.e."

Maybe this still works? https://stackoverflow.com/a/38652208

Or maybe we can create it not as an ES6 class but in the old prototype style so we can give it a custom name?

@disambiguator: "I dont want to try right this moment but also curious what would happen if we did.."

diff --git a/zaplib/web/zap_buffer.ts b/zaplib/web/zap_buffer.ts
index 9f5ce3f..b0a54b0 100644
--- a/zaplib/web/zap_buffer.ts
+++ b/zaplib/web/zap_buffer.ts
@@ -81,7 +81,7 @@ function zapBufferExtends(cls: any) {
         super(...args);
         this.__zaplibBuffer = buffer;
       } else {
-        super(...args);
+        cls(...args);
       }
     }

Using Zaplib without a canvas

Currently we always create a canvas when using Zaplib. That's a bit annoying if you just want to do some computation. We can probably make the canvas part of defaultStyles, and if that is set to false allow the user to specify it (or omit it).

Better way of sending JS objects to Rust, and back

Right now we JSON-serialize on both sides. It would be nice if we can add some basic functions around this mechanism. Some considerations:

• In JS the convention is camelCase, in Rust it is snake_case. We should probably avoid magically converting though, since that makes it hard to grep for variable names.
• It would be nice if the interface can be enforced through typescript type generation (or Rust struct generation).
• Maybe this is a separate ticket, but it would be nice if we don't have to copy stuff back and forth, and can keep referring to such data structures through e.g. proxy objects.

Better Rust error handling

• Panics should raise an error to user code so it can deal with it
• Console shouldn't get spammed with errors
• Panics should be catchable to user code from handle and draw calls.