denoland/deno_core

The core engine at the heart of Deno

Deno Core

The core engine at the heart of Deno.

Contributing

We appreciate your help!

To contribute, please read our contributing instructions.

deno_core's Issues

WeakRef, FinalizationRegistry, and SharedArrayBuffer are not available during snapshotting

cc @bnoordhuis @piscisaureus

Use const `OpDecl` arrays in extensions

With the work by @littledivy and others, we finally have a fully-const OpDecl that is Copy. This should allow us to generate const arrays of ops in extensions, and reduce the number of Vec allocations at startup.

Replace `op_arraybuffer_was_detached` and `op_transfer_arraybuffer` with built-in JS APIs

The ops op_arraybuffer_was_detached (in core/ops_builtin.rs) and op_transfer_arraybuffer (in ext/web) were defined to do things that couldn't be done at that time from JS. But V8 11.4 ships with the ArrayBuffer.prototype.transfer proposal (which also includes the ArrayBuffer.prototype.detached boolean property), which provides that from JS. We should switch any usages of those ops in Deno to the built-in APIs.

Port extension_with_esm to deno_core

We're missing some test coverage for snapshots and source-inclusion.

Write a test case for #112

#112 needs a regression test. One was provided in denoland/deno#19903 but it needs to be simplified for deno_core only.

`ExtensionFileSourceCode` variant for computed strings

As part of the approach I'm favouring for denoland/deno#19399, we need another variant for computed strings. Since it would need to be cloned on .load() it should be stored as Arc<str>.

pub enum ExtensionFileSourceCode {
  IncludedInBinary(&'static str),
  LoadedFromFsDuringSnapshot(&'static str), // <- Path
  Computed(Arc<str>),
}

And then I think we can remove snapshot_module_load_cb in favour of just computing the TS transpilations, which we would still only do during snapshot creation (or a no-snapshot mode for dev in the future).

---

Using a module_load_cb (expanding that feature to work outside of snapshot creation) has the advantage of holding one less copy of modules throughout the process runtime including the one allocated by V8. But it also means it needs to be recomputed for other realms, so that allocation is meaningful. For CLI we don't care very much since again it's only done on snapshot creation.

Another solution is ExtensionFileSourceCode::IncludedInBinary(computed_string.leak()) which will be stable later this month. That can take advantage of static ascii opts. But leaking may not always be an option for embedders, there's no harm in having this option.

Assertion error with Jemalloc: 'assertion failed: layout.size() != 0'

Hello,

I'm encountering an assertion error when using the deno_core crate in conjunction with jemalloc. I've detailed the setup and error reproduction steps below.

Cargo.toml:

[package]
name = "deno_core-test"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "1.0", features = ["full"] }
deno_core = "0.204.0"

[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator = "0.5"

main.rs:

#[cfg(not(target_env = "msvc"))]
use tikv_jemallocator::Jemalloc;

#[cfg(not(target_env = "msvc"))]
#[global_allocator]
static GLOBAL: Jemalloc = Jemalloc;

#[tokio::main]
async fn main() {
    let mut js_runtime = deno_core::JsRuntime::new(deno_core::RuntimeOptions::default());
    if let Err(e) = js_runtime.execute_script_static("", "var a = {test:1;") {
        eprintln!("{}", e);
    }
}

Error with RUST_BACKTRACE=1:

thread 'main' panicked at 'assertion failed: layout.size() != 0', /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tikv-jemallocator-0.5.4/src/lib.rs:102:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
   2: core::panicking::panic
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:117:5
   3: <tikv_jemallocator::Jemalloc as core::alloc::global::GlobalAlloc>::alloc
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tikv-jemallocator-0.5.4/src/lib.rs:102:9
   4: __rust_alloc
             at ./src/main.rs:6:16
   5: alloc::alloc::alloc
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/alloc.rs:102:9
   6: v8::string::<impl v8::data::String>::to_rust_string_lossy
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-0.74.3/src/string.rs:565:20
   7: deno_core::error::JsStackFrame::from_v8_message
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.204.0/error.rs:210:13
   8: deno_core::error::JsError::inner_from_v8_exception
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.204.0/error.rs:414:36
   9: deno_core::error::JsError::from_v8_exception
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.204.0/error.rs:265:5
  10: deno_core::error::exception_to_err_result
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.204.0/error.rs:705:22
  11: deno_core::runtime::jsrealm::JsRealm::execute_script
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.204.0/runtime/jsrealm.rs:390:16
  12: deno_core::runtime::jsruntime::JsRuntime::execute_script_static
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.204.0/runtime/jsruntime.rs:1196:5
  13: deno_core_test::main::{{closure}}
             at ./src/main.rs:11:5
  14: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/park.rs:282:63
  15: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:107:5
  16: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:73:5
  17: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/park.rs:282:31
  18: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/blocking.rs:66:9
  19: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  20: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/runtime.rs:65:16
  21: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  22: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/runtime.rs:349:45
  23: deno_core_test::main
             at ./src/main.rs:11:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ops/function.rs:250:5

I've deliberately introduced a syntax error in the sample JavaScript code. It seems that when such an error is present, an attempt is made to allocate memory of size 0 in to_rust_string_lossy.

I'm unsure if this is a compatibility issue between deno_core and jemalloc, or if I've missed something in the configuration. Any assistance on this would be greatly appreciated. Thank you.

Changing a module with remapped specifier doesn't cause a snapshot rebuild

Not sure if that's a problem in deno or deno_core, but when I was changing ext/node/polyfills/async_hooks.ts which maps to node:async_hooks the snapshot was not getting rebuilt. I have a hunch that we might be outputting a wrong value to cargo:rerun-if-changed directive.

@nayeemrmn would you be able to look into this?

op2: Investigate `#[string]` performance for LARGE_1000000

op2 strings are faster in every case other than LARGE_1000000 (1,000,000 ASCII characters). We need to investigate why.

test baseline                               ... bench:         878 ns/iter (+/- 96)
test bench_op_option_u32                    ... bench:      49,690 ns/iter (+/- 22,588)
test bench_op_string                        ... bench:      18,925 ns/iter (+/- 2,256)
test bench_op_string_large_1000             ... bench:     297,396 ns/iter (+/- 41,330)
test bench_op_string_large_1000000          ... bench:   2,622,869 ns/iter (+/- 298,615)
test bench_op_string_large_utf8_1000        ... bench:   3,946,605 ns/iter (+/- 403,230)
test bench_op_string_large_utf8_1000000     ... bench:  38,985,146 ns/iter (+/- 2,266,213)
test bench_op_string_old                    ... bench:      19,870 ns/iter (+/- 2,354)
test bench_op_string_old_large_1000         ... bench:     246,036 ns/iter (+/- 40,192)
test bench_op_string_old_large_1000000      ... bench:   1,082,275 ns/iter (+/- 104,487)
test bench_op_string_old_large_utf8_1000    ... bench:   5,485,882 ns/iter (+/- 489,366)
test bench_op_string_old_large_utf8_1000000 ... bench:  51,652,968 ns/iter (+/- 3,158,678)
test bench_op_string_option_u32             ... bench:      82,449 ns/iter (+/- 10,669)
test bench_op_u32                           ... bench:       4,508 ns/iter (+/- 575)
test bench_op_void                          ... bench:       5,054 ns/iter (+/- 419)

Remove "event_loop_middleware"

Current we have two "event_loop_middleware" - one for deno_ffi extension and one for deno_napi extension. These middlewares are called on each tick of the event loop and they incur some clones and polling receivers. This is totally unnecessary if neither of the extensions is used (which is the situation in HTTP server benches).

Ideally we would remove them altogether and rearchitecture how these extensions work, but we can settle on the middleground for now if we make these middlewares zero-cost if user doesn't use FFI or NAPI APIs.

CC @littledivy @aapoalas

Restore type check test

denoland/deno#19658 (comment)

`serde_v8::Value` does not implement Hash and Eq, resulting in being unable to serialize into a HashMap

Doing something like

let object: HashMap<serde_v8::Value, serde_v8::Value> = serde_v8::from_v8(value).unwrap();

gives two compile time errors.

Add `fast` non-serde `ByteString` parameters

We can make ByteStrings without using serde. While the slow path will require a scope, the fast path can make use of the *const u8 string that gets passed to the method.

At the same time, we should support #[string(onebyte)] &[u8] which could be a zero-copy version of ByteString that would be extremely useful for HTTP headers and values.

It would also make sense to add Cow and Vec parameters.

The following parameter types should be supported:

• #[string(onebyte)] Cow<[u8]>
• #[string(onebyte)] Vec<u8>
• #[string(onebyte)] ByteString
• #[string(onebyte)] &[u8]

Use of #[serde] ByteString should result in an error as users should migrate to one of the above forms.

Use serde_v8 for `serialize_for_snapshotting` and `update_with_snapshotted_data`

The ModuleMap code is manually serializing data to and from the snapshot which is a fairly large chunk of code. While serde_v8 may add a little bit of overhead, the code savings will be pretty immense and it should be easier to reason about.

`op2` should allow `where` clause with `'static`

op2 supports where T: Trait , but it does not support where T: 'static. This should be considered valid as there may be cases where the op requires a 'static trait but the trait does not need to be 'static.

Move benchmarks for various ops from Deno repo

Benchmarks like https://github.com/denoland/deno/blob/main/cli/bench/async_ops_deferred.js should be moved into this repo.

#[serde] return type coerces ResourceId to a string

#[op2(core)]
#[serde]
pub fn op_resources(state: &mut OpState) -> Vec<(ResourceId, String)> {
  state
    .resource_table
    .names()
    .map(|(rid, name)| (rid, name.to_string()))
    .collect()
}

returns this in JavaScript

{ "1": "stdout", "2": "stderr" }

The keys are turned into strings, even though they are defined as numbers.

Realms should be constructable from a `v8::HandleScope`

In order to make the ShadowRealm constructor work, it must be possible to create a new realm in the same way as JsRuntime::create_realm, but with a &mut v8::HandleScope rather than a &mut JsRuntime. This implies:

• Making JsRuntime::create_realm and the methods it depends on static methods taking a scope or an isolate reference. In the case of create_realm, since it's public, it can stay with the current API, but the implementation would move to JsRuntime::create_realm_from_scope.
• The extensions, global_template_middlewares, global_object_middlewares, preserve_snapshotted_modules and init_mode fields of JsRuntime must be available from the JsRuntimeState. @mmastrac suggested making a RuntimeExtensionInfo struct with those fields, which would also make things easier for #52.
• Don't run the entire isolate event loop in JsRuntime::init_extension_js. #99 takes care of this.
• Currently JsRealm::execute_script and module loading and evaluation use the scope returned by JsRealm::handle_scope. However, rusty_v8 needs each scope to take in a reference to the previous scope in order to keep track of the lifetimes, and JsRealm::handle_scope creates a new root scope. If this scope is created when the stack is not empty, as would happen when constructing a new realm from inside a JS execution context, handle_scope will panic. We would need to somehow make these JsRealm methods accept a parent scope, and use v8::ContextScope to switch into the realm.

Add support for forced-included or forced-loaded scripts to `extension!`

There are two major remaining uses of the extension customizer/extension builder in deno, and these are to specify scripts that should be included in the binary, or loaded from the FS during snapshot. We should replace these with an extension! DSL to specify them and remove the customizers.

When we do this, we should also be able to remove the remaining non-extension! implementations in the deno_core tests.

op2 u64/usize should have a `fast` forced-number mode

Many use cases can fit within the range of https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER -- we should have an integer annotation for these.

`ZeroCopyBuf` and `MagicBuffer` should not be `Send` or `Sync`

ZeroCopyBuf wraps a reference to a backing store that is potentially shared with an ArrayBuffer object, and thus is only safely accessible from one thread. Therefore, ZeroCopyBuf, and MagicBuffer which wraps it, must not be Send or Sync – yet they are both. In fact, ZeroCopyBuf unsafely implements Send, even though that is not needed – which seems to be a holdover from previous iterations of this API.

Once denoland/deno#12678 is solved, it might be possible to allow a way safely send ZeroCopyBuf values, as long as their backing store was created for a SharedArrayBuffer – but implementing Send directly on ZeroCopyBuf would still not be sound. We can tackle this when the time comes.

See also denoland/rusty_v8#798.

This was discovered when investigating denoland/deno#12341.

Expose `v8_fn_ptr` from an op

Some usages in deno depend on remapping OpDecl to change some details, currently following error is returned:

error[E0624]: associated function `v8_fn_ptr` is private
   --> runtime/ops/os/mod.rs:58:43
    |
58  |       v8_fn_ptr: deno_core::op_void_sync::v8_fn_ptr as _,
    |                                           ^^^^^^^^^ private associated function
    |
   ::: /Users/ib/dev/deno_core/core/ops_builtin.rs:107:1
    |
107 | #[op2(core, fast)]
    | ------------------ private associated function defined here

`op2` doesn't support `Result<*mut c_void>`

Multiplex multiple isolate/contexts over a single inspector instance

I'll update the description later

Related denoland/deno#13152
Related denoland/deno#13572

Deeper integration with Chrome DevTools

Deno already has great support for inspection of APIs via DevTools: we support the panels for CPU profiling, memory analysis, console, and source browsing. Chrome DevTools has a lot more to offer though: the network panel, the applications panel for browsing through local storage, or the performance panel.

It would be awesome to integrate support for these. For example it should be possible to view outbound fetch and WebSocket connections in the Network panel. Our local storage APIs could be hooked up to the application panel, and the user timing measurements from the performance global could be wired into the Performance tab.

An initial prototype for network panel integration is in progress in denoland/deno#11660. It is already working quite well:

deno_chrome_devtools_network_panel.mov

Next to the implementation of the "backend" APIs, we should investigate a custom build of DevTools in chrome://inspect (or at https://devtools.deno.land) like Node has. It would show these extra panels that Node does not support, without showing things like the Elements panel.

Debugger/Chrome DevTools umbrella issue

• denoland/deno#13152
• denoland/deno#13572
• denoland/deno#13664
• denoland/deno#13908
• denoland/deno#16662
• denoland/deno#16725
• denoland/deno#16757

segmentation fault when running multiple runtimes

I was trying to run multiple instances of deno_core::JsRuntime within one tokio task, but got:

    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/test1`
[Ok(()), Ok(())]
[Ok(()), Ok(())]
[Ok(()), Ok(())]
Segmentation fault (core dumped)

# Cargo.toml
[package]
name = "test1"
version = "0.1.0"
edition = "2021"

[dependencies]
deno_core = "0.202.0"
futures = "0.3.28"
tokio = "1.31.0"

// main.rs
use deno_core::{error::AnyError, op, Extension, ModuleCode, ModuleSpecifier, Op};

#[op]
async fn op_sleep_ms() {
    // not await-ing here works fine
    tokio::time::sleep(std::time::Duration::from_millis(100)).await;
}

pub async fn run(name: String, module: ModuleCode) -> Result<(), AnyError> {
    let extensions = Extension {
        ops: std::borrow::Cow::Borrowed(&[op_sleep_ms::DECL]),
        ..Default::default()
    };
    let mut js_runtime = deno_core::JsRuntime::new(deno_core::RuntimeOptions {
        extensions: vec![extensions],
        ..Default::default()
    });

    let module_path = ModuleSpecifier::from_file_path(name).unwrap();
    let mod_id = js_runtime
        .load_main_module(&module_path, Some(module))
        .await?;
    let result = js_runtime.mod_evaluate(mod_id);

    js_runtime.run_event_loop(false).await?;
    result.await?
}

async fn real_main() -> Result<(), AnyError> {
    loop {
        // 0..1 works fine
        let tasks = (0..2)
            .map(|i| {
                run(
                    format!("/just_sleep_{i}.mjs"),
                    ModuleCode::Static("Deno.core.ops.op_sleep_ms().await"),
                )
            })
            .collect::<Vec<_>>();
        println!("{:?}", futures::future::join_all(tasks).await);
    }
}

fn main() {
    tokio::runtime::Builder::new_current_thread()
        .enable_io()
        .enable_time()
        .build()
        .unwrap()
        .block_on(real_main())
        .unwrap();
}

tested on: rustc 1.71.1 (eb26296b5 2023-08-03) and rustc 1.73.0-nightly (1b198b3a1 2023-08-13)

the fault occurs in v8__Isolate__New()

Is this supposed to work? Is there a good other way to run multiple completely isolated scripts in parallel?
(I'd prefer to avoid creating new thread per runtime, however running multiple scripts/modules within the same runtime would be fine, as long as each has a separate Global object)

Per the spec, ES modules should always be decoded lossily as UTF-8

In the spec, the "fetch a single module script" algorithm handles fetching and parsing a specific ES module, and in step 13 it uses the encoding spec's "UTF-8 decode" to decode the response body into a string, which it then passes to the relevant parser. This algorithm decodes UTF-8 lossily (i.e. substituting invalid byte sequences with replacement characters), and if the body starts with a UTF-8 BOM, it strips it. It doesn't switch the encoding based on the MIME type, or on the presence of a UTF-16 BOM.

Deno's behavior depends on whether the module specifier has a file:// schema. For non-file:// schemas, the MIME type's charset parameter is used to pick an encoding if present. If not, UTF-8 is used. If the charset parameter doesn't correspond to a valid encoding, it throws. The BOM is ignored when choosing the encoding. For local files, the BOM is used, and if there isn't one, UTF-8 is used. In either case, if the decoding finds an invalid byte sequence in the chosen encoding at any point, the module load throws. And a BOM is not stripped when decoding, regardless of whether it was used to choose the encoding.

Note that Deno's behavior of choosing the MIME charset parameter over the BOM goes not only against the browser's handling of modules, but also against the way browsers handle classic scripts and page loads, which always prioritizes the BOM.

This was not noticed before now because most of the WPT tests that test for module loading are HTML-only, which our test runner doesn't support, since they typically use <script type="module">. The only JS tests for module loading deal with import assertions and JSON modules, and so this was only noticed when implementing them in denoland/deno#12866.

Since the behavior with local and remote files is so different, and this doesn't seem to have been noticed until now, it doesn't seem like switching to UTF-8 only would be very breaking in the Deno ecosystem. The one case where this might be breaking is with UTF-16 files with a BOM, since servers would be likely to detect and serve them as the correct encoding – but I doubt this is something to lose sleep over, since UTF-16 is very rarely used. Another potential concern might be CDN's that transpile Node.js code, since the served code might not have been run locally at any point, but the most used ones seem to always serve either with charset=utf-8 or without the charset parameter.

Build a const struct from the `extension!` macro

We should have enough infrastructure in place that the extension! macro can generate a const struct that can be fully computed at compile-time. If not, let's make the rest of this struct const-compatible and then deprecate and remove the ExtensionBuilder.

Deno.core.close(null) coerces to null to 0

  assertThrows(() => {
    Deno.close((null as unknown) as number);
  }, TypeError);

fails because instead of throwing TypeError the rid inside op_close is actually 0.

op2 Roadmap

• Fastcall support for v8::* (this does work!) (#25)
• In/out buffers
  • &[u8] (#33)
  • JsBuffer et al (for async) (#60)
• OpState support (#28)
  • #[state] arguments that automatically read OpState (Rc, &, &mut) (#29)
• async ops (#40) (#54)
  • HandleScope support in async ops (possible - needs design)
  • op tracker
  • Ensure Rc<RefCell<OpState>> is only available for async ops (will not do this)
• external/cvoid mapping (#146)
• Check error handling and error types for advanced argument types
• #[global] v8::Global support (#58)
• fast async ops #147

Nice to haves:

• #[resource] arguments that automatically read the resource table (Rc, &, &mut)
• Fastcall detection argument (#[is_fast] bool?)
• #[op(turbo)] for ops that don't touch any of the slow paths (ie: no state, no context, no scope)
• Port #[op(wasm)] from op1

Doesn't build on docs.rs anymore

https://docs.rs/crate/deno_core/0.197.0/builds/870081

`serde_v8::Value` in enum causes infinite recursion

Hi,

I'm trying to use serde_v8::Value to pass through a v8::Local<v8::Value> in a deserialized enum field like this:

pub enum SomeEnum<'s> {
  A {
    id: serde_v8::Value<'s>,
  },
}

The value being deserialized to the id field is a JS object with circular references inside, and serde_v8 gets stuck in an infinite recursion during deserialization.

Stack trace near the entry to from_v8:

#72021 0x0000560ec8ffd0ee in <serde::__private::de::content::ContentVisitor as serde::de::Visitor>::visit_map (self=..., visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:488                                          
#72022 0x0000560ec90adc1b in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_map (self=<optimized out>, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:315                                       
#72023 0x0000560ec90abb11 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_any (self=0x7ffec347aac0, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:134      
#72024 0x0000560ec8ffccca in <serde::__private::de::content::Content as serde::de::Deserialize>::deserialize (deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:297
#72025 0x0000560ec90ab50a in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785              
#72026 0x0000560ec90a6c33 in <serde_v8::de::MapAccess as serde::de::MapAccess>::next_value_seed (self=<optimized out>, seed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:486                                                     
#72027 0x0000560ec8ff11ca in serde::de::MapAccess::next_value (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1846                                                          
#72028 0x0000560ec8fff926 in <serde::__private::de::content::TaggedContentVisitor<T> as serde::de::Visitor>::visit_map (self=..., map=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:881   
#72029 0x0000560ec90ad411 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_map (self=<optimized out>, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:315                                       
#72030 0x0000560ec90abde4 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_any (self=0x7ffec347ae48, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:134                                        
#72031 0x0000560ec90d680c in rwv2::api::graphics::layout::_::<impl serde::de::Deserialize for rwv2::api::graphics::layout::Prop>::deserialize (__deserializer=0x0) at src/api/graphics/layout/mod.rs:91                                       
#72032 0x0000560ec90ab2da in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785                                                
#72033 0x0000560ec90aa031 in <serde_v8::de::ObjectAccess as serde::de::MapAccess>::next_value_seed (self=<optimized out>, seed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:548                
#72034 0x0000560ec90a5bba in serde::de::MapAccess::next_value (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1846                                                                        
#72035 0x0000560ec90b818b in <rwv2::api::graphics::layout::_::<impl serde::de::Deserialize for rwv2::api::graphics::layout::LayoutConstraint>::deserialize::__Visitor as serde::de::Visitor>::visit_map (self=..., __map=...) at src/api/graphics/layout/mod.rs:49
#72036 0x0000560ec90b30b7 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_struct (self=0x7ffec347b0c8, name=..., fields=..., visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:383
#72037 0x0000560ec90d55b3 in rwv2::api::graphics::layout::_::<impl serde::de::Deserialize for rwv2::api::graphics::layout::LayoutConstraint>::deserialize (__deserializer=0x0) at src/api/graphics/layout/mod.rs:49                                                             
#72038 0x0000560ec90ab3aa in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785              
#72039 0x0000560ec90a7376 in <serde_v8::de::SeqAccess as serde::de::SeqAccess>::next_element_seed (self=<optimized out>, seed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:593                                                   
#72040 0x0000560ec9058d4a in serde::de::SeqAccess::next_element (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1707                                                                      
#72041 0x0000560ec90558d5 in <serde::de::impls::<impl serde::de::Deserialize for alloc::vec::Vec<T>>::deserialize::VecVisitor<T> as serde::de::Visitor>::visit_seq (self=..., seq=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/impls.r
s:928                                                                                                                                                                                                                                         
#72042 0x0000560ec90ae810 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_seq (self=0x7ffec347b2c8, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:257                                        
#72043 0x0000560ec8fd3b1a in serde::de::impls::<impl serde::de::Deserialize for alloc::vec::Vec<T>>::deserialize (deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/impls.rs:939                                              
#72044 0x0000560ec90ab4ba in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785                                                
#72045 0x0000560ec90a9e71 in <serde_v8::de::ObjectAccess as serde::de::MapAccess>::next_value_seed (self=<optimized out>, seed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:548  
#72046 0x0000560ec90a5baa in serde::de::MapAccess::next_value (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1846                                                          
#72047 0x0000560ec90b7d2e in <rwv2::api::graphics::layout::_::<impl serde::de::Deserialize for rwv2::api::graphics::layout::LayoutRequest>::deserialize::__Visitor as serde::de::Visitor>::visit_map (self=..., __map=...) at src/api/graphics/layout/mod.rs:16
#72048 0x0000560ec90b4937 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_struct (self=0x7ffec347b5d8, name=..., fields=..., visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:383
#72049 0x0000560ec90d4d13 in rwv2::api::graphics::layout::_::<impl serde::de::Deserialize for rwv2::api::graphics::layout::LayoutRequest>::deserialize (__deserializer=0x0) at src/api/graphics/layout/mod.rs:16                
#72050 0x0000560ec90b555e in serde_v8::de::from_v8 (scope=<optimized out>, input=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:43                                                 
#72051 0x0000560ec90cfc3b in rwv2::api::util::v8_deserialize (scope=0x0, value=...) at src/api/util.rs:18   

And the repeating pattern is:

#71769 0x0000560ec8ffd673 in <serde::__private::de::content::ContentVisitor as serde::de::Visitor>::visit_seq (self=..., visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:477                                          
#71770 0x0000560ec90ae560 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_seq (self=0x7ffec3473878, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:257                                        
#71771 0x0000560ec90abac3 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_any (self=0x7ffec3473878, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:133                                        
#71772 0x0000560ec8ffccca in <serde::__private::de::content::Content as serde::de::Deserialize>::deserialize (deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:297                                                
#71773 0x0000560ec90ab50a in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785                                                
#71774 0x0000560ec90a62d2 in <serde_v8::de::MapAccess as serde::de::MapAccess>::next_entry_seed (self=<optimized out>, kseed=..., vseed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:506                                         
#71775 0x0000560ec8ff10da in serde::de::MapAccess::next_entry (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1860                                                                                                          
#71776 0x0000560ec8ffd046 in <serde::__private::de::content::ContentVisitor as serde::de::Visitor>::visit_map (self=..., visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:488                                          
#71777 0x0000560ec90adc1b in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_map (self=<optimized out>, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:315                                       
#71778 0x0000560ec90abb11 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_any (self=0x7ffec3473be8, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:134                                        
#71779 0x0000560ec8ffccca in <serde::__private::de::content::Content as serde::de::Deserialize>::deserialize (deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:297                                                
#71780 0x0000560ec90ab50a in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785                                                
#71781 0x0000560ec90a62d2 in <serde_v8::de::MapAccess as serde::de::MapAccess>::next_entry_seed (self=<optimized out>, kseed=..., vseed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:506                                         
#71782 0x0000560ec8ff10da in serde::de::MapAccess::next_entry (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1860                                                                                                          
#71783 0x0000560ec8ffd046 in <serde::__private::de::content::ContentVisitor as serde::de::Visitor>::visit_map (self=..., visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:488                                          
#71784 0x0000560ec90adc1b in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_map (self=<optimized out>, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:315                                       
#71785 0x0000560ec90abb11 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_any (self=0x7ffec3473f58, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:134                                        
#71786 0x0000560ec8ffccca in <serde::__private::de::content::Content as serde::de::Deserialize>::deserialize (deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:297                                                
#71787 0x0000560ec90ab50a in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785                                                
#71788 0x0000560ec90a62d2 in <serde_v8::de::MapAccess as serde::de::MapAccess>::next_entry_seed (self=<optimized out>, kseed=..., vseed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:506                                         
#71789 0x0000560ec8ff10da in serde::de::MapAccess::next_entry (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1860                                                                                                          
#71790 0x0000560ec8ffd046 in <serde::__private::de::content::ContentVisitor as serde::de::Visitor>::visit_map (self=..., visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:488                                          
#71791 0x0000560ec90adc1b in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_map (self=<optimized out>, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:315                                       
#71792 0x0000560ec90abb11 in <&mut serde_v8::de::Deserializer as serde::de::Deserializer>::deserialize_any (self=0x7ffec3474228, visitor=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:134                                        
#71793 0x0000560ec8ffccca in <serde::__private::de::content::Content as serde::de::Deserialize>::deserialize (deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/private/de.rs:297                                                
#71794 0x0000560ec90ab50a in <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize (self=..., deserializer=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:785                                                
#71795 0x0000560ec90a7756 in <serde_v8::de::SeqAccess as serde::de::SeqAccess>::next_element_seed (self=<optimized out>, seed=...) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_v8-0.11.0/src/de.rs:593                                                   
#71796 0x0000560ec8ff1b1a in serde::de::SeqAccess::next_element (self=0x0) at /home/zhy/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.130/src/de/mod.rs:1707

Migrate extension middleware functions in extensions to unboxed `fn()`

We have three middleware functions that do not carry state, and can be unboxed and converted to fn()-style functions rather than Boxed closures. This should save a number of allocations on startup.

deno_core/core/runtime/jsruntime.rs

Line 219 in fe93b1d

event_loop_middlewares: Vec<Box<EventLoopMiddlewareFn>>,

  event_loop_middlewares: Vec<Box<EventLoopMiddlewareFn>>,
  global_template_middlewares: Vec<Box<GlobalTemplateMiddlewareFn>>,
  global_object_middlewares: Vec<Box<GlobalObjectMiddlewareFn>>,

`op2` async numeric return values

The async return values for op2 are sparse. We need to ensure that all numeric return values are supported.

As part of this work, valid_retvals.md needs to be updated with a new column for async support.

Example op that we cannot port:

#[op2(async)]
#[bigint]
pub async fn op_cache_storage_open<CA>(
  state: Rc<RefCell<OpState>>,
  #[string] cache_name: String,
) -> Result<i64, AnyError>

Fetching from Private NPM Repo Doesn't Work/Misleading User Feedback

When I try to pull down a repo from my org's private NPM repository with the following command:

NPM_CONFIG_REGISTRY="https://ghe.myorg.com/_registry/npm/@myorg/package" DENO_AUTH_TOKENS="[email protected]/_registry/npm/@myorg/package"  deno cache --node-modules-dir cache_test

I get this output

Download https://ghe.myorg.com/_registry/npm/@myorg/package/vary (59/116)

I have tried a few different permutations of the domain for DENO_AUTH_TOKENS. I can verify that using my token in a header with cURL does in fact return npm package info.

At worst private package imports are broken. At best this user feedback via the CLI isn't helpful - Download is green, there's no troubleshooting information, and it looks as if the process is just hung.

Repro with this command:
NPM_CONFIG_REGISTRY="https://google.com" deno cache --node-modules-dir googs

Migrate `get_error_class_fn` to `OpCtx`

The code paths that retrieve get_error_class_fn are currently borrowing the OpState from a RefCell first. Let's move these to OpCtx directly (one copy of this function pointer per OpCtx so that we can simply read it as we need it.

This may impact both op and op2 code. If major changes are required to op code, let's leave the old copy of get_error_class_fn on OpState and get rid of it when we fully deprecate op1

Support customizing `import.meta` in `deno_core`.

When embedding deno_core it would be useful to be able to add custom properties to import.meta. This is not possible at the moment because the set_host_initialize_import_meta_object_callback and ModuleMap are private.

`op2` doesn't support `&mut [u8]` or `&mut [u32]`

Required for FFI and op_fs_stat_sync among others.

I'll try to use the fast path instead of `SafeIterator` from primordials

I'm reasonably certain it was intentional, and I believe engines are capable of optimizing it when Array.prototype[Symbol.iterator] is unmodified.

tc39/ecma262#827 (comment)

Deno panics with 'already mutably borrowed' when `WebAssembly.instantiateStreaming` is not awaited

Reproduction:

Download and save @connorskees/grass NPM package's data somewhere (use NPM or Deno, doesn't matter). Probably any other WebAssembly file will also work.

Then setup the following script:

const promise = await fetch(
    new URL(
        "file:///path/to/@connorskees/grass/grass_bg.wasm"
    )
);
const wsPromise = WebAssembly.instantiateStreaming(
    promise
);

Expected result

The process finishes without errors.

Actual result

A panic occurs:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: linux x86_64
Version: 1.32.3
Args: ["deno", "run", "-A", "test.ts"]

thread 'main' panicked at 'already mutably borrowed: BorrowError', core/ops_builtin_v8.rs:823:1
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
   2: core::result::unwrap_failed
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/result.rs:1790:5
   3: deno_core::ops_builtin_v8::op_store_pending_promise_rejection::v8_fn_ptr
   4: _ZN2v88internal12_GLOBAL__N_119HandleApiCallHelperILb0EEENS0_11MaybeHandleINS0_6ObjectEEEPNS0_7IsolateENS0_6HandleINS0_10HeapObjectEEENS8_INS0_20FunctionTemplateInfoEEENS8_IS4_EEPmi
             at ./../../../../v8/src/builtins/builtins-api.cc:113:36
   5: _ZN2v88internal21Builtin_HandleApiCallEiPmPNS0_7IsolateE
             at ./../../../../v8/src/builtins/builtins-api.cc:135:1
   6: Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit

Workaround

If the wsPromise is awaited, then the panic is avoided.

op2 doesn't handle result with optional serde

This doesn't compile:

#[op2(async)]
#[serde]
async fn op_worker_recv_message(
  state: Rc<RefCell<OpState>>,
) -> Result<Option<JsMessageData>, AnyError>

JsMessageData has Serialize derive, I would expect it to just work, but it refuses to compile :(

`extension!` `esm` setting should support defining a specifier

Currently esm property in extension! macro support syntax like:

  esm = [
    dir "polyfills",
    "00_globals.js",
    "02_init.js",
    "_brotli.js",
  ]

This creates a specifier like ext:<extension_name>/<dir>/<module_name>. In some cases we want to be able to explicitly set the extension for a module, @mmastrac suggests syntax like:

  esm = [
    dir "polyfills",
    "00_globals.js",
    "02_init.js",
    "_brotli.js",
    "foo.js" as "node:foo" 
  ]

If ordering of op init between snapshot and runtime differ, a very undescript error occurs

What happens is that ops are linked together in the wrong way, and then the wrong underlying op gets called for each JS side fn.

For example a sync op will just start calling the backing for a different async op, resulting in fun errors like TypeError: invalid promise id: expected type 'v8::data::Integer', got 'v8::data::Value'.

It'd be great to add a debug assertion to core to ensure that ops are loaded in the same order between snapshot and runtime.

Make `#[ops(stable)]` opt-in instead of opt-out

We should use #[ops(stable)] to say an op is stable so that it's an explicit opt-in. This would reduce the chances of someone creating an unstable op as stable.

Edit: Oh, it seems we need #[ops(unstable)] for this to actually work.

Add documentation on `extension!` on how to use the resulting struct

In particular, when to use init_js_only vs init_ops_and_esm vs init_ops.

Create an `ExtensionSet` to handle logic around multiple extensions

There is a fair amount of logic in JsRuntime that deals with collating the data from multiple Extensions. We should move all of this logic to an ExtensionSet that can be initialized from a slice of extensions.

In addition, the ExtensionSet should be extendable to add more extensions to it at a later time. This will allow us to build a base ExtensionSet for the snapshot, then extend it when loading further extensions (for example, for the worker and CLI).

Finally, investigate if we can or should persist the ExtensionSet as part of the snapshot for the purpose of verifying that the correct extensions were passed in as part of snapshot init, or potentially to avoid having to re-create the set of Extensions for the base snapshot.

Port `op2` tests to v8 internals (ie: `%OptimizeFunctionOnNextCall`)

run_test2 currently forces fastcalls by running the test 10,000 times. We can instead use the v8 internals to force a fastcall:

%PrepareFunctionForOptimization({op});
{call};
%OptimizeFunctionOnNextCall({op});
{call};

See benches/ops/sync.rs for an example.

Run runtime async?

I'm currently making my own little js-environment based on deno-core. My goal is to let the user modify the UI from the JS. When I use the code from the examples, the runtime is executed blocking. Ho can I change that? Could you add an example?

`op2` fast async ops

Fast async ops scheduled the op future in a fast call. This is used a lot in ext/websocket when we don't want to spend time polling the future in a slow call. eg:

https://github.com/denoland/deno/blob/main/ext/websocket/lib.rs#L576-L580

#[op(fast)]
pub async fn op_ws_next_event(
  state: Rc<RefCell<OpState>>,
  rid: ResourceId,
) -> u16 {

I couldn't find a way to do this with #[op2]. We should add support for it