hirofa/quickjs_es_runtime

this is a wrapper library for the javascript runtime quickjs written in rust which works with typescript, modules, promises, async, await and much more

quickjs_runtime

quickjs_runtime is a library for quickly getting started with embedding a javascript engine in your rust project.

as of 2024 this lib no longer relies on libquickjs-sys but on our own hirofa-quickjs-sys adding flexibility in used quickjs version

quickjs_runtime runs all javascript action in a single thread using an EventLoop. This means you can call javascript safely from several threads by adding tasks to the EventLoop.

quickjs or quickjs-ng

quickjs_runtime supports both the original quickjs and the quickjs-ng project.

You can try out quickjs-ng by adding the dep to quickjs_runtime like this (use at your own risk as I have not extensively tested it yet):

quickjs_runtime = {git="https://github.com/HiRoFa/quickjs_es_runtime", features=["console", "setimmediate", "setinterval", "settimeout", "typescript", "quickjs-ng"], default-features=false}

Usage and Features

An example on how to embed a script engine in rust using this lib can be found here: github.com/andrieshiemstra/ScriptExtensionLayerExample. It was published in TWIR as a walkthrough.

quickjs_runtime focuses on making quickjs easy to use and does not add any additional features, that's where these projects come in:

• A more feature-rich (e.g. fetch api support, http based module loader and much more) runtime: GreenCopperRuntime.
• The commandline client: GreenCopperCmd.

Please see the DOCS for all inner workings

This lib serves two main goals:

1. Provide simple utils for working with quickjs (these are located in the quickjs_utils mod)

• The QuickJsRuntime struct, this is to be used from a single thread
• E.g. objects::set_property(), functions::invoke_func()
• Wrap JSValue to provide reference counting (+1 on init, -1 on drop) (QuickJsValueAdapter)
• Pass a module loader

2. Wrap quickjs for use as a ready to go JavaScript Runtime

• Start at the QuickjsRuntimeFacade, it provides an EventQueue which has a thread_local QuickJsRuntimeAdapter
• All values are copied or abstracted in a JsValueFacades
• So no need to worry about Garbage collection
• evaluate script and invoke functions while waiting for results blocking or with async/await
• Get Promise result blocking or with async/await

What works?

Script and Modules

• Typescript (via SWC)
• console (.log/info/debug/trace/error) (docs)
• Eval script (docs)
• Create promises in JavaScript which execute async
• Eval modules (docs)
• Load modules (dynamic and static) (docs)
• fetch api (impl in GreenCopperRuntime)
• setImmediate
• setTimeout/Interval (and clear)
• script preprocessing (impls for ifdef/macro's/typescript can be found in GreenCopperRuntime)

Rust-Script interoperability

• Return Promises from rust functions and resolve them from rust (docs)
• Add functions from rust (docs)
• Invoke JS functions from rust (docs)
• Pass primitives, objects and arrays from and to rust (docs)
• Create Classes from rust (docs)
• async/await support on eval/call_function/promise resolution (docs)
• import native Modules (e.g. dynamic loading of rust functions or Proxy classes) (docs)

Goals

Embedding a script engine in a rust project seems a very tedious job which involves learning a lot about the inner workings of that engine.

The main goal of this project is to make that job easy!

The manner in which this is achieved is primarily focused on abstracting the workings of the engine from the implementor, therefore some functionality may not be the fastest way of getting things done.

So a second goal is to make implementing a fast and efficient integration doable for the uninitiated, the most common tasks you do with the engine should be doable with the utils in this package and working examples should be provided in the test modules.

The reason I chose QuickJS as the engine is that I've been dealing with less modern engines in my java projects and not being able to use the latest and greatest ECMA-script features becomes quite disappointing at times.

The fun stuff about QuickJS:

• small footprint
• fast compilation / startup
• great JS compatibility

examples

Cargo.toml

[dependencies]
quickjs_runtime = "0.13.1"

Here are some quickstarts:

• start by reading the DOCS
• eval a script

The quickjs Api utils:

• quickjs_utils

quickjs_es_runtime's Issues

EventLoop revamp

use https://docs.rs/futures/0.3.13/futures/executor/struct.LocalPool.html

use an extra scheduling thread for setTimeout/setInterval

ESRuntime refactor

• get rid of inner, use the Arc around ESRuntime
• move all publics to a Trait which we can reuse in ES_Runtime

Leaked promise inside a dropped context causes panic!

Hi there!,
Please check below full code of main.rs with inline comments to reproduce the issue.

In summary, when a js code, executed using add_rt_task_to_event_loop to run within a context, leaks a promise. quickjs_runtime panics when the leaked promise resolves as the context was already dropped.

main.rs

use futures::executor::block_on;
use hirofa_utils::js_utils::{JsError, Script};
use log::LevelFilter;
use quickjs_runtime::esruntime::EsRuntime;
use quickjs_runtime::esruntimebuilder::EsRuntimeBuilder;
use quickjs_runtime::esvalue::EsValueFacade;
use std::sync::Arc;
use std::thread;
use std::time::Duration;

async fn test(rt: Arc<EsRuntime>) -> anyhow::Result<(), JsError> {
    // create context
    rt.create_context("abc")?;
    let result = rt
        .add_rt_task_to_event_loop(move |q_js_rt| {
            // use the above created context to run the eval
            let q_ctx = q_js_rt.get_context("abc");
            let res = q_ctx.eval(Script::new(
                "basics.es",
                r#"
            // the below promise is leaked
            new Promise((resolve) => {
                setTimeout(() => {
                    console.log("leaked promise resolved...");
                    resolve("done done");
                }, 1000);
            });
            // the below promise is returned
            new Promise((resolve) => {
                setTimeout(() => {
                    console.log("promise resolved...");
                    resolve("done");
                }, 200); // increase this time to 2s to see the leaked promise resolved first and hence doesn't crash
            });
        "#,
            ));

            match res {
                Ok(res) => EsValueFacade::from_jsval(q_ctx, &res),
                Err(e) => {
                    log::error!("{}", e);
                    Err(e)
                }
            }
        })
        .await;

    match result {
        Ok(res) => {
            // resolve the returned promise
            let fut = res.get_promise_result();
            let val = fut.await;
            log::info!("{:?}", val);
        }
        Err(e) => {
            log::error!("{}", e);
        }
    };

    // drop the context as the js execution is complete
    rt.drop_context("abc");
    log::info!("context dropped...");

    // wait or do something else...
    // at this point... the leaked promise get's resolved and panics with 'no such context'

    thread::sleep(Duration::from_secs(5)); // if this line is commented then it doesn't panic as the thread get's dropped here

    Ok(())
}

fn main() {
    simple_logging::log_to_stderr(LevelFilter::Info);
    let rt = EsRuntimeBuilder::new().build();
    match block_on(test(rt)) {
        Ok(_) => {}
        Err(e) => {
            log::error!("{}", e);
        }
    }
    thread::sleep(Duration::from_secs(5));
}

Error

thread '<unnamed>' panicked at 'no such context', /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/quickjsruntime.rs:308:31
stack backtrace:
   0: rust_begin_unwind
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
   1: core::panicking::panic_fmt
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:92:14
   2: core::option::expect_failed
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/option.rs:1243:5
   3: core::option::Option<T>::expect
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/option.rs:351:21
   4: quickjs_runtime::quickjsruntime::QuickJsRuntime::get_context
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/quickjsruntime.rs:308:9
   5: quickjs_runtime::features::set_timeout::set_timeout::{{closure}}::{{closure}}::{{closure}}
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/features/set_timeout.rs:90:33
   6: quickjs_runtime::quickjsruntime::QuickJsRuntime::do_with::{{closure}}
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/quickjsruntime.rs:424:13
   7: std::thread::local::LocalKey<T>::try_with
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/thread/local.rs:400:16
   8: std::thread::local::LocalKey<T>::with
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/thread/local.rs:376:9
   9: quickjs_runtime::quickjsruntime::QuickJsRuntime::do_with
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/quickjsruntime.rs:422:9
  10: quickjs_runtime::features::set_timeout::set_timeout::{{closure}}::{{closure}}
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/features/set_timeout.rs:87:17
  11: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ops/function.rs:227:5
  12: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/alloc/src/boxed.rs:1575:9
  13: hirofa_utils::eventloop::EventLoop::run_timeouts_and_intervals
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.2.1/src/eventloop.rs:134:13
  14: hirofa_utils::eventloop::EventLoop::new::{{closure}}::{{closure}}
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.2.1/src/eventloop.rs:97:37
  15: std::thread::local::LocalKey<T>::try_with
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/thread/local.rs:400:16
  16: std::thread::local::LocalKey<T>::with
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/thread/local.rs:376:9
  17: hirofa_utils::eventloop::EventLoop::new::{{closure}}
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.2.1/src/eventloop.rs:71:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread '<unnamed>' panicked at 'cannot access a Thread Local Storage value during or after destruction: AccessError', /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/thread/local.rs:376:26
stack backtrace:
   0: rust_begin_unwind
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
   1: core::panicking::panic_fmt
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:92:14
   2: core::result::unwrap_failed
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/result.rs:1355:5
   3: core::result::Result<T,E>::expect
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/result.rs:997:23
   4: std::thread::local::LocalKey<T>::with
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/thread/local.rs:376:9
   5: <quickjs_runtime::quickjscontext::QuickJsContext as core::ops::drop::Drop>::drop
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/quickjscontext.rs:298:13
   6: core::ptr::drop_in_place<quickjs_runtime::quickjscontext::QuickJsContext>
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mod.rs:192:1
   7: core::ptr::drop_in_place<(alloc::string::String,quickjs_runtime::quickjscontext::QuickJsContext)>
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mod.rs:192:1
   8: core::ptr::mut_ptr::<impl *mut T>::drop_in_place
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mut_ptr.rs:995:18
   9: hashbrown::raw::Bucket<T>::drop
             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/hashbrown-0.11.0/src/raw/mod.rs:344:9
  10: hashbrown::raw::RawTable<T,A>::drop_elements
             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/hashbrown-0.11.0/src/raw/mod.rs:598:17
  11: hashbrown::raw::RawTable<T,A>::clear
             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/hashbrown-0.11.0/src/raw/mod.rs:591:13
  12: hashbrown::map::HashMap<K,V,S,A>::clear
             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/hashbrown-0.11.0/src/map.rs:774:9
  13: std::collections::hash::map::HashMap<K,V,S>::clear
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/collections/hash/map.rs:566:9
  14: <quickjs_runtime::quickjsruntime::QuickJsRuntime as core::ops::drop::Drop>::drop
             at /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.5.1/src/quickjsruntime.rs:507:9
  15: core::ptr::drop_in_place<quickjs_runtime::quickjsruntime::QuickJsRuntime>
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mod.rs:192:1
  16: core::ptr::drop_in_place<core::option::Option<quickjs_runtime::quickjsruntime::QuickJsRuntime>>
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mod.rs:192:1
  17: core::ptr::drop_in_place<core::cell::UnsafeCell<core::option::Option<quickjs_runtime::quickjsruntime::QuickJsRuntime>>>
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mod.rs:192:1
  18: core::ptr::drop_in_place<core::cell::RefCell<core::option::Option<quickjs_runtime::quickjsruntime::QuickJsRuntime>>>
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mod.rs:192:1
  19: core::ptr::drop_in_place<core::option::Option<core::cell::RefCell<core::option::Option<quickjs_runtime::quickjsruntime::QuickJsRuntime>>>>
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/ptr/mod.rs:192:1
  20: core::mem::drop
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/mem/mod.rs:889:24
  21: std::thread::local::fast::destroy_value
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/thread/local.rs:656:13
  22: std::sys::unix::thread_local_dtor::register_dtor::run_dtors
             at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys/unix/thread_local_dtor.rs:89:17
  23: <unknown>
  24: __pthread_tsd_cleanup
  25: __pthread_exit
  26: __pthread_start

Proxy+ProxyInstance .toString

EsValueFacade.stringify

todo:

• all types

console

full support as in es_runtime

move ScriptPreProcessor and EsScript to utils

JSValueRef revamp

revert back to the consume_value method for calling quickjs methods which take ownership insterad of just doing *borrow_value()

revamp FetchResponseProvider to be a trait

do we can built struct impls which can be configured to filter allowed http locations and such just like the HttpModuleLoader in GreenCopperRuntime

error reporting if module load fails

refactor obj handling in EsValueFacade

make it a 'live' link just like Function and Promise, and only serialize to Map if get_object is called, then add methods like get_property / has_property etc

remove all .ok().expect()

and replace with error handling / logging

module loading gc workings unclear

currently in modules.rs we have LOADED_MODULE_REGISTRY in which we check to see if a module is loaded..

can this be replaced by using q::JS_DetectModule ?

when are modules garbage collected?

e.g. does this even work?

eval: {import("test.mes").then((module) => {
    console.log(module.foo);
})}
gc();
eval: {import("test.mes").then((module) => {
    console.log(module.foo);
})}

revamp all utils to use *mut JSContext instead of q_js_rt

async/await support

EsValueFacade.promise_future()
EsValueFacade.invoke_function_future()

start from EsRuntime.add_task_future()

these are all added as extra func so current fire_and_forget and /sync version may still exist

todo:

• EventQueue.asyncTask
  • EventQueue.asyncTask -> impl with waker()
• EsRuntime methods
• EsValueFacade.invoke_function
  • EsValueFacadeFuture -> impl with waker()
• EsValueFacade.get_promise_result() (also rename get promise_result_blocking to get promise_result_sync

rename EsRuntime(Builder) to QuickjsRuntimeFacade/Builder

and QuickjsContext/Runtime to *Adapter

setImmediate

script preprocessor support

For

• Macro's
• Ifdef
• Ts
• Etc

fetch api

cargo audit action

https://github.com/RustSec/cargo-audit
maybe also https://crates.io/crates/cargo-crev

Compile utils

compile to bytecode and run bytecode

use JS_UpdateStackTop

after init of the runtime use this set the correct stack top

and maybe set the max size as a large number by default

runtimebuilder.gcInterval

run gc every x period

add InteruptHandler to EsRuntimeBuilder

uses JS_SetInteruptHandler

compile test on 32 bits

if thats really thre problem, running on raspberry pi gives segfaults

bigint and date support

mostryly to get rid of the copypasted cfg stuff

revamp EsValueFacade

based on utils::JsValueFacade just like StarlightRuntime

32bit tests in github actions

if thats really the problem, (running on raspberry pi gives segfaults)

rebuild EsValueFacade to enum of different types

Instead of is_str type functions

console, only constrcut line if loglevel is high enough

in console.rs we constrcut a string from "log something here %s", paramA

that should only happen if we're actually going to log that

Ability to load pre-compiled modules

Hi there!
I'm able to pre-compile a JS/TS function and save as bytecode and later execute using run_compiled_function.

I'm looking for a similar option for pre-compiling some modules (e.g. lodash, date-fns etc) to avoid loading and compiling on every invocation.

May be ByteCodeModuleLoaderAdapter that will be almost similar to ScriptModuleLoaderAdapter except for the below line.

quickjs_es_runtime/src/quickjsruntimeadapter.rs

Line 83 in 0225d27

let compiled_module = unsafe { compile_module(q_ctx.context, script)? };

Any thoughts?

EsValueFacade::invoke_function_batch(_sync)(Vec<Vec<EsValueFacade>>)

performance thing when using stuff like mysql feature in esseslib-q

How about event loop?

replace helper_thread_pool in EsRuntime with tokio

so we can do .await in promise resolvers and such

timeouts on invocations like call_function and eval

for theduke/quickjs-rs#4

i think i want to impl this a bit broader (basicly in all public methods of JSRealmAdapter) so you can allways recover from endless loops and such...

Also i want to reimpl timeouts for get_promise_result (which was removed because it was difficult to impl with Futures)

native module utils

be able to add native modules on demand

e.g.

import("hirofa.http").then((http) => {
    // http is a native module here
    return http.get("www.google.com");
}).then((result) => {
    console.log("got %1", result.textContent);
});

fix tests (free runtime issue)

Worker support

quickjs has os.Worker but stuff like module loading in quickjs_runtime will probably be borked because of the extra thread...

test cases, not just in doc

a lot of methods are now just tested in docs and not in a mod tests, in order to test them all wait valgrind we need them also in a test mod

remove droppable value

setTimeout

In addtoqueue run addpjobs task in closure so we can use add-jobfromworker

And save a trip by the mutex

make full featured tokio optional

Update README.md

Replace quick_es_runtime into quickjs_es_runtime

primitives::from_i32, from_bool and from_f64 should use new exported functions

q::JS_NewBool, q::JS_NewInt32 and q::JS_NewFloat64

Proxy EventTarget todo's

• addEventListener from rust
• removeEventListener from rust
• dispatchEvent from rust

test todos

• removeEventListener
• dispatch from javascript
• addListener from rust

Panic when using nested setTimeout and when calling clearTimeout()

Getting the below error when using nested setTimeout

panicked at 'already borrowed: BorrowMutError'

Code to reproduce the issue

[dependencies]
quickjs_runtime = "0.4.2"
futures = "0.3"
log = "0.4.11"
simple-logging = "2.0.2"

use futures::executor::block_on;
use log::LevelFilter;
use quickjs_runtime::esruntime::EsRuntime;
use quickjs_runtime::esruntimebuilder::EsRuntimeBuilder;
use quickjs_runtime::esscript::EsScript;
use std::sync::Arc;

async fn test(rt: Arc<EsRuntime>) {
    let res = rt
        .eval(EsScript::new(
            "basics.es",
            r#"
            new Promise((resolve) => {
                setTimeout(() => {
                    console.log('in timeout 1');
                    setTimeout(() => {  // panic
                        console.log('in timeout 2');
                        resolve(true);
                    }, 1000)
                }, 1000);
            });
        "#,
        ))
        .await
        .ok()
        .unwrap();

    let fut = res.get_promise_result();
    let val = fut.await;
    println!("{:?}", val);
}

fn main() {
    simple_logging::log_to_stderr(LevelFilter::Info);
    let rt = EsRuntimeBuilder::new().build();

    block_on(test(rt));
}

Other example js that causes panic

new Promise((resolve) => {
    const timeout = setTimeout(() => {
        console.log('in timeout 1');
        clearTimeout(timeout);  // panic
        resolve(true);
    }, 1000);
});

new Promise((resolve) => {
    const timeout = setTimeout(() => {
        console.log('in timeout 1');
    }, 10000);
    setTimeout(() => {
        console.log('in timeout 2');
        clearTimeout(timeout); // panic
        resolve(true);
    }, 1000);
});

unit tests for every method in utils

EsFunction struct for creating functions as EsValueFacade

use code from EsRuntime::set_function

and replace that with EsFunction::new()

also needed for simple proxy creation for fs module

32bit support / windows