golemcloud/wasm-rpc

wasm-rpc's Issues

Re-Initialising workspace with a new target for the same source overwrites the cargo compose commands

Issue criticality:
For me, it is Low to Medium (not as concerning/critical as cyclic dependency issue).
Also, I know what to do in my code if at all this issue persists - therefore low for me.

wasm-rpc commit: 33c4d1b

My golem-cli points to the above wasm-rpc revision. And I initialise the workspace with caller as caller and targets as callee

./golem/target/debug/golem-cli stubgen initialize-workspace --targets callee --callers caller --wasm-rpc-path-override /Users/afsalthaj/github/wasm-rpc/wasm-rpc

At this stage everything works, and I can see the Makefile.toml has compose commands (for dev and release versions) correctly.

Now, if I initialise the workspace again (previously we didn't support this, but now we do) with target as caller2, the new stub generation (for caller2) as well new add-stub-dependency (caller to have dependency on caller2 as well) is there, but the compose commands gets rewritten from callee to caller2. I think this is unintended behaviour.

Either we update generate, add-stub-dependency and compose from callee to callee2. But currently, generate and add-stub-dependency get appended and compose commands gets overwritten.

Also, in a large setup, when something fails, tracking down this can be troublesome for users, even if we document this behaviour and ask users to add these commands.

For a better understanding see the git diff screenshot

The Makefile generated is in the comment section.

Use List of targets for the caller - as an alternative approach.

So what I did after bumping into this issue is, I re-set everything with a clean workspace of 3 modules.https://github.com/afsalthaj/testwasmrpc/tree/sample_workspace

A cargo component build here works

    Finished dev [unoptimized + debuginfo] target(s) in 3.82s
    Creating component /Users/afsalthaj/github/testwasmrpc/target/wasm32-wasi/debug/callee2.wasm
    Creating component /Users/afsalthaj/github/testwasmrpc/target/wasm32-wasi/debug/callee.wasm
    Creating component /Users/afsalthaj/github/testwasmrpc/target/wasm32-wasi/debug/caller.wasm

And running the following, resulted in the following failure.

 afsalthaj@users-iMac  ~/github/testwasmrpc   master  ../golem/target/debug/golem-cli stubgen initialize-workspace --targets callee,callee2 --callers caller --wasm-rpc-path-override /Users/afsalthaj/github/wasm-rpc/wasm-rpc

Writing cargo-make Makefile to "/Users/afsalthaj/github/testwasmrpc/Makefile.toml"
Generating initial stub for callee,callee2
Error: Failed to gather information for the stub generator

stubgen build does not result in a component

stubgen add-stub-dependies should not fail if there is no [package.metadata.component.target.dependencies] key

Generated code with `golem-cli stubgen` gives compilation error

I thought of creating this issue here from a user perspective.
The stubgen created the code that resulted in the following error

note: method defined here
   --> /Users/afsalthaj/.cargo/registry/src/index.crates.io-6f17d22bba15001f/golem-wasm-rpc-0.0.20/src/builder.rs:43:8
    |
43  |     fn f64(self, value: f64) -> Self::Result;
    |        ^^^
help: consider dereferencing the borrow
    |
319 |                                                             ) => case_builder.f64(*inner),
    |                                                                                   +

error[E0308]: mismatched types
   --> core-stub/src/lib.rs:322:84
    |
322 | ...                   ) => case_builder.bool(inner),
    |                                         ---- ^^^^^ expected `bool`, found `&bool`
    |                                         |
    |                                         arguments to this method are incorrect
    |
note: method defined here
   --> /Users/afsalthaj/.cargo/registry/src/index.crates.io-6f17d22bba15001f/golem-wasm-rpc-0.0.20/src/builder.rs:46:8
    |
46  |     fn bool(self, value: bool) -> Self::Result;
    |        ^^^^
help: consider dereferencing the borrow
    |
322 |                                                             ) => case_builder.bool(*inner),

The way to reproduce

git clone https://github.com/afsalthaj/golem-timeline.git
cd golem-timeline
cargo make build-flow

The make files has stubgen commands in it using the latest golem-cli 0.0.91.

If we are exposing this feature of updating cargo make file automatically with the composition commands, then we need to make sure it is 100% validated code because otherwise, the subsequent steps of composition becomes manual.

Problem when package name contains dashes

Generating Cargo.toml to /Users/vigoo/projects/demo/shopping-cart-rpc/purchase-history-stub/Cargo.toml
thread 'main' panicked at /Users/vigoo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/golem-wasm-rpc-stubgen-0.0.11/src/rust.rs:29:21:
"purchase-history_stub" is not a valid Ident```

Dash vs underscore problem in generated makefile

Example:

Creating component /Users/vigoo/projects/demo/shopping-cart-rpc/target/wasm32-wasi/debug/purchase_history.wasm
Creating component /Users/vigoo/projects/demo/shopping-cart-rpc/target/wasm32-wasi/debug/shopping_cart.wasm
Creating component /Users/vigoo/projects/demo/shopping-cart-rpc/target/wasm32-wasi/debug/purchase_history_stub.wasm
[cargo-make] INFO - Execute Command: "golem-cli" "stubgen" "compose" "--source-wasm" "target/wasm32-wasi/debug/shopping-cart.wasm" "--stub-wasm" "target/wasm32-wasi/debug/purchase-history_stub.wasm" "--dest-wasm" "target/wasm32-wasi/debug/shopping-cart-composed.wasm"

Ability to manipulate the generated makefile

The initialize-workspace command sets up a cargo-make makefile with an initial set of callers and targets, but we need more commands to add or remove more connections between the components, so people can continue using this generated workspace during development.

Migrate from wasm-tools compose to wac

Based on bytecodealliance/wasm-tools#1555

stubgen initialize-workspace do not add member if it already in the list

Error in `golem-wasm-rpc` when trying to install `golem-cli`

I tried to install golem-cli

cargo install golem-cli

Get the following error:

error: could not compile `golem-wasm-rpc` (lib) due to 8 previous errors
warning: build failed, waiting for other jobs to finish...
error: failed to compile `golem-cli v0.0.107`, intermediate artifacts can be found at `/var/folders/fs/mjcxy3_s7ls84f603q6gl9fw0000gn/T/cargo-install0o9Iga`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
ERROR Cargo errored! ExitStatus(unix_wait_status(25856))
ERROR Fatal error:
  × For crate golem-cli: subprocess /opt/homebrew/Cellar/rust/1.79.0/bin/cargo
  │ install golem-cli --version 0.0.107 errored with exit status: 101
  ╰─▶ subprocess /opt/homebrew/Cellar/rust/1.79.0/bin/cargo install golem-cli
      --version 0.0.107 errored with exit status: 101

Here is the error from rust compiler:

https://gist.github.com/tonidy/e291ce9a83a0fe1854a0828b6c7c6623

Decode Error in wasm-rpc when variant type has no payload

Bumped into the following error when the variant type in WIT was just a unit. This is the logs from worker-executor

 Remote internal error: Runtime error: Value mismatch: could not get type information for case leaf
golem-worker-executor_1               | thread '<unnamed>' panicked at core-stub/src/lib.rs:463:14:
golem-worker-executor_1               | Failed to invoke remote timeline:core/api/initialize-timeline: RpcError::RemoteInternalError("Runtime error: Value mismatch: could not get type inform

This is when I used stub to call remote worker

The variant type used was as simple as

  variant timeline-node {
    leaf,
  }

Now, I think it has to do with unit-type, and I got it going by passing a dummy s32

  variant timeline-node {
    leaf(s32),
  }

And then it worked correctly. Some logs from the remote worker after the change

  | 2024-04-15T01:31:59.603734Z  INFO golem_worker_executor_base::worker: Worker e7a50810-9cab-4543-9f1d-dc269f4987b2/raw-events/0 activated
golem-worker-executor_1               | [core/src/lib.rs:24:9] timeline = TimelineOp {
golem-worker-executor_1               |     nodes: [
golem-worker-executor_1               |         TimelineNode::Leaf(
golem-worker-executor_1               |             1,
golem-worker-executor_1               |         ),
golem-worker-executor_1               |     ],

Support Circular dependency case by inlining the definitions in the stub generator

If workers are spawn from the same component, and these workers need to communicate with each other,
it ends up in a circular dependency case where in each generated stub which depends on the original wit (from which the stub was generated), becomes a dependency to the original wit.

For example, the original wit here is called timeline-processor.wit and the generated stub will be _stub.wit

# This is the timeline-processor.wit

package timeline:timeline-processor;


interface api {
  record foo {
    value: string
  }

  initialize: func(start: foo) -> string;
}

world timeline-processor {
  export api;
}

The corresponding generated stub wit will be (obviously)

# this is the generated stub wit which comes under the generated stub module timeline-processor-stub
package timeline:timeline-processor-stub;

interface stub-timeline-processor {
  use golem:rpc/[email protected].{uri};
  use timeline:timeline-processor/api.{foo};

  resource api {
    constructor(location: uri);
    initialize: func(start: foo) -> string;
  }

}

world wasm-rpc-stub-timeline-processor {
  export stub-timeline-processor;
}

Now we add this stub as dependency to timeline-processor itself (because we need the communication between workers that were spawned from timeline-processor template)

However whe using the current golem-cli stubgen add-stub-dependency (0.62 version), we end up having timeline_processor.wit along with _stub.wit into the wit dependencies of timeline-processor itself, making it a redundant circular dependency.

The subsequent cargo component build will result in the following error

thread 'main' panicked at /Users/afsalthaj/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wit-parser-0.13.1/src/resolve.rs:1034:9:
assertion failed: prev.is_none()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Solution

The solution is to basically inline the definitions in the original wit (timeline_processor.wit) into the _stub.wit that is under wit/deps in timeline-processor.

i.e,

# stub.wit

package timeline:timeline-processor-stub;

interface stub-timeline-processor {
  use golem:rpc/[email protected].{uri};
  
  record foo {
    value: string
  }

  resource api {
    constructor(location: uri);
    initialize: func(start: foo) -> string;
  }

}

world wasm-rpc-stub-timeline-processor {
  export stub-timeline-processor;
}

And thereby, not ending up having to paste timeline-processor.wit itself in wit/deps, and things will compile

cargo make build flow shows unnecessary git diffs (minor issue)

When running cargo make build with Golem's stubgen and compositions, minor formatting changes occur in different lines, causing unnecessary differences in git diff. Though minor, this can lead to users having to make sure or eyeball the changes every time.

Example:

Stub generator does not work if the caller and target components have the same WIT package name

This should be checked and failed with a meaningful error.

Reproducer:

mkdir rpc
cd rpc
golem-cli new --example rust-actor-minimal --component-name component1
golem-cli new --example rust-actor-minimal --component-name component2

echo '[workspace]' > Cargo.toml
echo 'resolver = "2"' >> Cargo.toml
echo 'members = ["component1", "component2"]' >> Cargo.toml

golem-cli stubgen initialize-workspace --targets component1 --callers component2

cargo make build-flow

stubgen initialize-workspace is not adding the stubs to the workspace members list

Cyclic dependency from x->y->x returning is_none error

As a followup to the comments on #40, I created a repro for the x->y->x dependency case using testwasmrpc and I am seeing the is_none error crop up again:

cargo make build-flow
[cargo-make] INFO - cargo make 0.36.3
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: build-flow
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Execute Command: "cargo-component" "clean"
  Generating bindings for callee (/Users/sameer/workspace/testwasmrpc/callee/src/bindings.rs)
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /Users/sameer/workspace/testwasmrpc/callee-stub/Cargo.toml
workspace: /Users/sameer/workspace/testwasmrpc/Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /Users/sameer/workspace/testwasmrpc/caller-stub/Cargo.toml
workspace: /Users/sameer/workspace/testwasmrpc/Cargo.toml
     Removed 840 files, 213.9MiB total
[cargo-make] INFO - Execute Command: "/Users/sameer/workspace/golem/target/debug/golem-cli" "stubgen" "generate" "-s" "callee/wit" "-d" "callee-stub" "--wasm-rpc-path-override" "/Users/sameer/workspace/wasm-rpc/wasm-rpc"
[/Users/sameer/workspace/wasm-rpc/wasm-rpc-stubgen/src/wit.rs:172:5] "The exported world is the selected world for the stub {}." = "The exported world is the selected world for the stub {}."
[/Users/sameer/workspace/wasm-rpc/wasm-rpc-stubgen/src/wit.rs:172:5] world.name.clone() = "callee"
Generating stub WIT to callee-stub/wit/_stub.wit
Copying root package test:callee
  .. callee/wit/callee.wit to callee-stub/wit/deps/test_callee/callee.wit
Writing wasm-rpc.wit to callee-stub/wit/deps/wasm-rpc
Generating Cargo.toml to callee-stub/Cargo.toml
Generating stub source to callee-stub/src/lib.rs
Done
[cargo-make] INFO - Execute Command: "/Users/sameer/workspace/golem/target/debug/golem-cli" "stubgen" "add-stub-dependency" "--stub-wit-root" "callee-stub/wit" "--dest-wit-root" "caller/wit" "--overwrite" "--update-cargo-toml"
Overwriting caller/wit/deps/test_callee
Copying "callee-stub/wit/deps/test_callee" to "caller/wit/deps/test_callee"
Copying "callee-stub/wit/deps/wasm-rpc" to "caller/wit/deps/wasm-rpc"
Writing updated Cargo.toml to "caller/Cargo.toml"
Done
[cargo-make] INFO - Execute Command: "/Users/sameer/workspace/golem/target/debug/golem-cli" "stubgen" "generate" "-s" "caller/wit" "-d" "caller-stub" "--wasm-rpc-path-override" "/Users/sameer/workspace/wasm-rpc/wasm-rpc"
[/Users/sameer/workspace/wasm-rpc/wasm-rpc-stubgen/src/wit.rs:172:5] "The exported world is the selected world for the stub {}." = "The exported world is the selected world for the stub {}."
[/Users/sameer/workspace/wasm-rpc/wasm-rpc-stubgen/src/wit.rs:172:5] world.name.clone() = "caller"
Generating stub WIT to caller-stub/wit/_stub.wit
Copying package golem:[email protected]
  .. caller/wit/deps/wasm-rpc/wasm-rpc.wit to caller-stub/wit/deps/wasm-rpc/wasm-rpc.wit
Copying package test:callee
  .. caller/wit/deps/test_callee/callee.wit to caller-stub/wit/deps/test_callee/callee.wit
Copying package test:callee-stub
  .. caller/wit/deps/test_callee-stub/_stub.wit to caller-stub/wit/deps/test_callee-stub/_stub.wit
Copying package test:caller-stub
Skipping copying stub package test:caller-stub
Copying root package test:caller
  .. caller/wit/caller.wit to caller-stub/wit/deps/test_caller/caller.wit
Writing wasm-rpc.wit to caller-stub/wit/deps/wasm-rpc
Skipping updating WIT dependency for test:caller-stub
Generating Cargo.toml to caller-stub/Cargo.toml
Generating stub source to caller-stub/src/lib.rs
Done
[cargo-make] INFO - Execute Command: "/Users/sameer/workspace/golem/target/debug/golem-cli" "stubgen" "add-stub-dependency" "--stub-wit-root" "caller-stub/wit" "--dest-wit-root" "callee/wit" "--overwrite" "--update-cargo-toml"
Copying "caller-stub/wit/deps/test_callee" to "callee/wit/deps/test_callee"
Copying "caller-stub/wit/deps/wasm-rpc" to "callee/wit/deps/wasm-rpc"
Copying "caller-stub/wit/deps/test_caller" to "callee/wit/deps/test_caller"
Copying "caller-stub/wit/deps/test_callee-stub" to "callee/wit/deps/test_callee-stub"
Writing updated Cargo.toml to "callee/Cargo.toml"
Done
[cargo-make] INFO - Execute Command: "cargo-component" "build"
  Generating bindings for caller (/Users/sameer/workspace/testwasmrpc/caller/src/bindings.rs)
thread 'main' panicked at /Users/sameer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wit-parser-0.13.1/src/resolve.rs:1034:9:
assertion failed: prev.is_none()
stack backtrace:
   0:        0x102bb4ee4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb478ebbfb46e27ce
   1:        0x102bd6100 - core::fmt::write::he4d5fa2daff1f531
   2:        0x102bb1410 - std::io::Write::write_fmt::hc5a47a68eba63d9f
   3:        0x102bb4d18 - std::sys_common::backtrace::print::h79bd952cc5812e7a
   4:        0x102bb64c0 - std::panicking::default_hook::{{closure}}::h82301f6222887737
   5:        0x102bb6208 - std::panicking::default_hook::h1e49abbb3f1d7dbf
   6:        0x102bb6908 - std::panicking::rust_panic_with_hook::h1e70c5d905e30e9d
   7:        0x102bb67d0 - std::panicking::begin_panic_handler::{{closure}}::h399e32952efd26a4
   8:        0x102bb5368 - std::sys_common::backtrace::__rust_end_short_backtrace::h2ab87f841a2323e7
   9:        0x102bb6584 - _rust_begin_unwind
  10:        0x102c3fd2c - core::panicking::panic_fmt::h33e40d2a93cab78f
  11:        0x102c3fdb4 - core::panicking::panic::h57fd475c037a9df3
  12:        0x10292b3bc - wit_parser::resolve::Remap::append::hf0a4902c2e5efeb3
  13:        0x10293d6fc - wit_parser::ast::SourceMap::rewrite_error::h411a009861a15487
  14:        0x10291f85c - wit_parser::resolve::Resolve::push::h1a01cae65197fed0
  15:        0x10269aa04 - cargo_component::bindings::BindingsGenerator::create_target_world::h64849129f3cc8b9b
  16:        0x102696a54 - cargo_component::bindings::BindingsGenerator::new::h6efaa1a2ada9b9d3
  17:        0x1024b7cc4 - cargo_component::run_cargo_command::{{closure}}::h9a627e9877ab8633
  18:        0x1024ccdac - cargo_component::main::{{closure}}::hb955c16fd1ace6a5
  19:        0x1024c74e0 - tokio::runtime::park::CachedParkThread::block_on::ha1b8d054c8f2a772
  20:        0x1025b4ffc - tokio::runtime::context::runtime::enter_runtime::hea8bb7e9fd218a77
  21:        0x102586df0 - tokio::runtime::runtime::Runtime::block_on::h5494f2a473f673da
  22:        0x10255fdcc - cargo_component::main::h4b2437b0e35f78b7
  23:        0x102574844 - std::sys_common::backtrace::__rust_begin_short_backtrace::h322ab5ba4fd90cb7
  24:        0x10258b544 - std::rt::lang_start::{{closure}}::h99915cb0bfc2b2bb
  25:        0x102ba958c - std::rt::lang_start_internal::hf4f3eb1e51305b96
  26:        0x10255fed4 - _main
[cargo-make] ERROR - Error while executing command, exit code: 101
[cargo-make] WARN - Build Failed.

I made a PR into the testwasmrpc repo with my repro: afsalthaj/testwasmrpc#1

Thanks

The add-stub-dependency command should automatically add (if not yet added) an import to the caller's worlds

Have integration tests for the stub generator