second-state/wasm-joey

Serverless Wasm - A lightweight Node.js application for deploying and executing WebAssembly(Wasm) binary-code via HTTP

START UPDATE - August 2020

Easy instructions

For a simple and easy to follow demonstration of how to create, deploy and execute WebAssembly please read this Medium article called Programming the Web with WebAssembly

Feedback is very welcome

We have an area devoted to feedback. Please feel free to raise any questions, issues and/or feature request via the link in this paragraph.

END UPDATE

Wasm-joey

A Joey is a baby Kangaroo. Wasm-joey is a portable, flexible, lightweight application for deploying and executing WebAssembly(Wasm) binary-code via HTTP requests and responses.

Wasm-joey is written in Node.js. It is easy to set up and deploy. All configuration is stored in the application's base directory. It can be deployed on any machine that can run Node. It's continual operation is performed via HTTP requests and responses. This means that developers, end users and even computer-code itself can easily manage and execute WebAssembly code in a number of ways (including but not limited to) Javascript (XMLHttpRequest), jQuery(ajax), Curl (command line/shell) or any HTTP REST client such as Postman.

Documentation

For more information, please see the documentation section.

Simplicity

"Any program is only as good as it is useful" - Linus Torvalds. This application is being built to be as useful as possible. Part of achieving this involves putting in extra effort and design to ensure that it is as simple as possible, for you to use. Please create an issue if you have any ideas on how this application can be improved.

wasm-joey's Issues

Add an example wasm executable in binary-code format to the database as part of installation

This is so that the request/response can be tested immediately

Move the wasm_description to the header

This ensures that the body is only wasm file content

Error executing this function, please check function name, input parameters, return parameter for correctness

I run server and upload this hello wasm example then i run on postman below code

curl --location --request POST 'https://127.0.0.1:8081/api/run/1/say' \
--header 'Content-Type: text/plain' \
--data-raw 'Tim'

Response error;

{"return_value":"Error executing this function, please check function name, input parameters, return parameter for
correctness"}

Console error;

1,b212ab14-f2e4-4104-b7a3-b9ee75312b37.so
Loading 1's AOT file into cache (b212ab14-f2e4-4104-b7a3-b9ee75312b37.so)
/api/run/:wasm_id/:function_name ...
Creating log object
No callback found in the header keys
No fetchable information found in the header keys
SSVM Options: {"EnableAOT":true,"EnableMeasurement":true,"args":["{}"],"env":{"wasm_id":"1","storage_key":"6008e2a80d77b1d22eb5752357dac64e"},"preopens":{"/":"/tmp"}}
Instantiating SSVM with AOT filename of: aot/b212ab14-f2e4-4104-b7a3-b9ee75312b37.so which has a typeof: string
Instantiation success!
It is resolved that the JSON object is not empty, returning to function execution ...
Parameters len == 1
Executing function WITHOUT a callback...
[2021-10-19 12:24:00.055] [error] instantiation failed: unknown import, Code: 0x62
[2021-10-19 12:24:00.055] [error]     When linking module: "__wbindgen_placeholder__" , function name: "__wbindgen_describe"
[2021-10-19 12:24:00.055] [error]     At AST node: import description
[2021-10-19 12:24:00.055] [error]     At AST node: import section
[2021-10-19 12:24:00.055] [error]     At AST node: module

Implement header prefix of SSVM

Add the SSVM- prefix to headers so that they do not conflict with existing header keys.

Create an installation script

It would be really great to have a bash script that executed all of the installation commands (which are detailed in the installation documentation).

The script would be suitable to Ubuntu server and would:

• update the system
• install MySQL, Node.js etc.
• Write the configuration files using SED etc.
• Configure any security and provide user prompts to complete this i.e. secure MySQL and set password etc.

Write tests for multipart using every combination of GET, POST and Callback

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A test to show how multipart can fetch both GET requests
A test to show how multipart can fetch POST requests
A test to show how multipart can fetch both GET and POST requests
A test to show how multipart can fetch both GET and POST requests and also perform a callback at the end
Here is an example of the requests that require testing
GET

curl --location --request POST 'https://dev.rpc.ssvm.secondstate.io:8081/api/multipart/run/109/process_three_inputs' \
--header 'Content-Type: multipart/form-data' \
--form 'first_input_example_1="one"' \
--form 'second_input_example_2="two"' \
--form 'fetch_third_input_example_3=https://raw.githubusercontent.com/tpmccallum/test_endpoint2/master/tim.txt'

POST

curl --location --request POST 'https://dev.rpc.ssvm.secondstate.io:8081/api/multipart/run/109/process_three_inputs' \
--header 'Content-Type: multipart/form-data' \
--form 'first_input_example_1="one"' \
--form 'second_input_example_2="two"' \
--form 'fetch_asdf_3={"body": "asdf", "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}'

GET and POST

curl --location --request POST 'https://dev.rpc.ssvm.secondstate.io:8081/api/multipart/run/109/process_three_inputs' \
--header 'Content-Type: multipart/form-data' \
--form 'first_input_example_1="one"' \
--form 'second_input_example_2="two"' \
--form 'fetch_third_input_example_3=https://raw.githubusercontent.com/tpmccallum/test_endpoint2/master/tim.txt' \
--form 'fetch_again_4={"body": "asdf", "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}'

GET and Callback

curl --location --request POST 'https://dev.rpc.ssvm.secondstate.io:8081/api/multipart/run/109/process_three_inputs' \
--header 'Content-Type: multipart/form-data' \
--form 'first_input_example_1="one"' \
--form 'second_input_example_2="two"' \
--form 'fetch_third_input_example_3=https://raw.githubusercontent.com/tpmccallum/test_endpoint2/master/tim.txt' \
--form 'SSVM_Callback={"body": "asdf", "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}'

POST and Callback

curl --location --request POST 'https://dev.rpc.ssvm.secondstate.io:8081/api/multipart/run/109/process_three_inputs' \
--header 'Content-Type: multipart/form-data' \
--form 'first_input_example_1="one"' \
--form 'second_input_example_2="two"' \
--form 'SSVM_Callback={"body": "asdf", "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}' \
--form 'fetch_asdf_3={"body": "asdf", "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}'

GET and POST and Callback

curl --location --request POST 'https://dev.rpc.ssvm.secondstate.io:8081/api/multipart/run/109/process_three_inputs' \
--header 'Content-Type: multipart/form-data' \
--form 'first_input_example_1="one"' \
--form 'second_input_example_2="two"' \
--form 'fetch_third_input_example_3=https://raw.githubusercontent.com/tpmccallum/test_endpoint2/master/tim.txt' \
--form 'fetch_again_4={"body": "asdf", "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}' \
--form 'SSVM_Callback={"body": "asdf", "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}'

Rust Wasm
The following type of simple Rust will suffice as a test function (take many parameters and pass back a single join)

use wasm_bindgen::prelude::*;
  
#[wasm_bindgen]
pub fn process_three_inputs(input_a: String, input_b: String, input_c: String) -> String{
    let res = input_a + &input_b + &input_c;
    res
}

Implement multipart/form-data to support html file upload

This will be useful for calling functions of wasm executable. Details below.

Enquire about return_value of RunUint8Array

When the vm.RunUint8Array runs in conjunction with a callback [1], we need to decide how the return_value (which is given directly from SSVM execution) is packaged up in readiness for the callback.

Reason being, the callback object is always a JSON string. Therefore the return_value needs to be converted from it's native byte array to Array / String in the JSON and then potentially back again depending on the incoming Content-Type of the callback.

[1] https://github.com/second-state/wasm-joey/blob/master/src/server.js#L407

Specify an "argument_url" in the header

I think the function caller should be able to give an argument_url in the function exec request. Joey will fetch the URL content and pass it into the function as call argument (as a string or byte array depending on the argument_url content type).

This way, we can make it easier to chain Joey functions. For example, we could create an image / video processing function that simply takes a URL and returns the results back into a remote hosting service.

This issue is related to #12

⚠️ Prevent SQL injection

Is your feature request related to a problem? Please describe.
I think current SQL code didn't escape or prevent SQL injection? e.g.

var sqlSelect = "SELECT wasm_binary from wasm_executables WHERE wasm_id = '" + _wasm_id + "';";

Describe the solution you'd like
We should prevent SQL injection somehow.

• placeholders
• named placeholders
• node-mysql module

as explain here https://blog.sqreen.com/preventing-sql-injection-in-node-js-and-other-vulnerabilities/

Provide logging via `logging_callback_url`

Create a mechanism to write details of activities in a way that it is queryable

If a caller sends a request and does not want the response in a body, but rather provides a callback_url to direct the response to another service/task

In these cases the caller is not waiting on a promise etc. they just assume that the rest of the RESTful requests were performed and worked as expected

We could provide a logging callback_url which would essentially spawn off a request which will write to database and log action i.e. who was just called? what were the parameters?

This way if a part of the system is failing the logs will show where the callback_url chain halted i.e. what the last successful thing to execute was.

This can all be done in parallel with the actual functional work that is being performed via callback_urls

Add unique parameter i.e. `joey_remote_url` or `joey_arg_url` for remote m2m data exchange

If a caller is using the multipart endpoint to run a function, Joey should allow the caller to specify a joey_remote_data_url as one of the parameters i.e. if the caller is executing a function that edits a large video/image they should not be required to have that image on their local device (in order to add the data as a function argument/parameter).

This way callers can be light clients like mobile phones, yet still initiate high performance/compute intensive/data intensive processes from a small remote device.

How would this work?
If call to /api/multipart/run contains a string called joey_remote_data_url then Joey need to go and fetch the raw data at that URL's location and pass it into the ssvm function call (all server side)

Ensure consistent language

After the restful_api branch is merged, ensure that the language for set, get, execute, update, delete is consistent. This is an end-user focussed application, whereby the end-user does not really need to understand what happens beyond the HTTP POST call. There may be places in the existing documentation and code that employ words like load which the end-users might find confusing.

Limit the number of callbacks to rpc.ssvm.secondstate.io

Being able to callback to another wasm executable is brilliant. However, this could be used to tie up the system in a DOS way.
Add a feature to limit the number of callbacks allowed.

Test new issue template

Describe the bug

I was trying to ...

However the following occurred ...

Endpoint
Please provide the full endpoint in question i.e.

https://rpc.ssvm.secondstate.io:8081/api/ephemeral_storage/d3f6e52d-ca88-46ef-ad95-f84725276fd5

Type of request
Please let us know what verb you were using i.e. POST, GET, DELETE etc.

Type of data being sent in the request
Please provide an example of the data you were sending in as part of the original request.

Content-Type
Please provide the content type that you specified as part of the original request's headers.
i.e. application/json or text/plain etc.

Features like SSVM_Fetch
Please let us know what, if any, extra features you were trying to use i.e. remote data fetching

curl --location --request POST 'https://rpc.ssvm.secondstate.io:8081/api/run/1/say' \
--header 'Content-Type: text/plain' \
--header 'SSVM_Fetch: https://raw.githubusercontent.com/tpmccallum/test_endpoint2/master/tim.txt' \

Callback
Please provide details about how you used a callback (in the header, in the body etc.). This info must also include an example of the data and the content type that the call back was processing i.e.

curl --location --request POST 'https://rpc.ssvm.secondstate.io:8081/api/run/1/say' \
--header 'Content-Type: text/plain' \
--header 'SSVM_Callback: { "hostname":"rpc.ssvm.secondstate.io","path": "/api/run/1/say", "method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}' \
--data-raw 'world!'

Multipart
If you were using Multipart/form-data please provide an example of the keys and values that you were using i.e.

curl --location --request POST 'https://dev.rpc.ssvm.secondstate.io:8081/api/multipart/run/49/say' \
--header 'Content-Type: multipart/form-data' \
--form 'first_input_example_1="one"' \
--form 'second_input_example_2="two"' \
--form 'fetch_third_input_example_3={"body": "body","hostname": "rpc.ssvm.secondstate.io","path": "/api/run/1/say","method": "POST","port": 8081,"headers": {"Content-Type": "text/plain"}}'

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

RPC endpoint for associating a function call's return value with a message queue

The RPC caller provides Azure credentials in the HTTP request.

This operation will redirect the function's return value to a message queue. We should support

• Azure Queue Storage
• Azure Events Hub
• AWS SNS
• AWS SQS
• Twilio

Support text/plain in /api/state

The context state string could be a simple text/plain string. Joey will just save the string and pass it to the function at /api/run.

RPC endpoint to upload a WASM file and return an UUID

This is an HTTP POST endpoint. The wasm file content is in the POST body

We can use the file system to store the file for now .wasm

Specify a “callback url” in the header

When the function returns, it will post the result to the callback url.

Levels of logging (verbose vs minimal)

Investigate standards around remote procedure and REST in terms of logging.
Perhaps add a parameter that a caller can set i.e. increase/decrease the amount of logging.

Create new format for executing Wasm function

The function call endpoint should be as follows
/api/run/:wasmId/:functionName
This allows the request body to only contain the parameters for the function

Update the endpoint for Wasm function execution so that it can accept JSON body as bytes and return JSON as bytes

The bytes section represents a binary http body
/api/run/:wasmId/:functionName/bytes
A call to this endpoint would return a binary response

RPC endpoint for function calls in byte array

1 UUID and function name are in URL path:
/bytes//function_name

2 Input parameter is a byte array in the HTTP POST body

3 Return value is a byte array in the response body

RPC endpoint for function calls in JSON

1 UUID and function name are in URL path:
/json//function_name
This allows the function to be webhooks for services like Azure Events, AWS SQS / SNS, github and twilio

2 Input parameters are encoded in a single JSON string in the HTTP POST body or in a GET parameter called "params"

3 Return value is encoded in a single JSON string in the response body

Add logic for cases where wasm_id does not exist

Check logic and test current implementation of PUT (updates) and DELETE endpoints.

Also add this to the GET where a single wasm_id is passed in as req.param.wasm_id, then check logic and test this also.

Allow caller to submit request and then terminate request/response process

In most cases, if a caller is actioning a callback url they will want to receive the response from the callback (as a returned promise etc.)

In some cases, a caller might action a callback that they know will take a very long time i.e. render a video and save to remote storage etc. Rather than have the caller's request/response session time out/hang or fail, Joey must allow the caller to specify a wait true/false parameter when making a request to Joey.

Add data validation on incoming JSON data in the body

We will need to add data validation to incoming body data for each of the POST endpoints
i.e. will need to validate body like this to ensure database does not end up with bad data

{"wasm_description": "Put here by the API", "wasm_hex": "0x1234567890"}

RPC endpoint for associating a function call's return value with a webhook

The RPC caller provides a webhook URL in the HTTP request.

This operation will redirect the function's return value to a HTTP POST to the webhook URL

Provide easy access to store small amounts of data frequently

Provide the easiest and quickest form of ephemeral data storage.
This data storage is technically just held in memory and therefore is very fast.
This data storage does not require any Rust / Wasm whatsoever.
This feature allows calling code to store small amounts of data frequently i.e. a sensor that measures the temperature every second.
This feature provides a data storage key which lasts for up to 1 hour.
If the data is not updated after one hour then the key and data are deleted.
If the data is updated before the Time To Live TTL value of 1 hour expires, then a new TTL of 1 hour is initiated.

This could remain part of a free tier offering and could be subject to rate limiting i.e. size of request body is less than 1000kb and frequency is up to one request per second only etc.

POST

POST new data

/api/ephemeral_storage

Get

GET existing data at key

/api/ephemeral_storage/:key

Update

PUT updated data at key

/api/ephemeral_storage/:key

Delete

DELETE data at key

/api/ephemeral_storage/:key

Support stateful context

There is a context for each wasm file. The context is a string and can be set with a separate API call. Every wasm function call’s first argument is always this context string — even if it is empty. The function developer must take this into account when he processes the input arguments.

Documentation

A list of documentation that is required

• callback url (how to create a callback request) done
• how to write a Rust function that adds a callback url
• how to set a wasm executable
• how to update a wasm executable
• how to execute a function of a wasm executable
• how to read details about a wasm executable i.e. filterBy["wasm_as_buffer"] etc.
• how to execute a function of a wasm executable (by passing in a byte array i.e. image data)
• how to execute a function of a wasm executable (by passing in multipart form data)
• how to use the joey_remote_data_urlparameter in a multipart form data execution in order to load data remotely (and have that executed on the server)
• how Rust Wasm and the caller have to agree on function parameter data

Please keep adding to the above. Once complete, please link to the documentation via the above list's text as shown in the first item

We now have a short video for each of the RESTful actions.
We also have the postman documentation for all of the RESTful actions