deislabs/spiderlightning

A set of WIT definitions and associated implementations to enable app developers to work at a faster pace and require less knowledge of the environment in which they are executing.

SpiderLightning

A set of WIT interfaces that abstract distributed application capabilities and a runtime CLI for running Wasm applications that use these capabilities.

Warning: Alpha quality software, do not use in production.

---

🚧 Important Development Update 🚧

As of 01/01/2024, the development of the Slight CLI is on hold. SpiderLightning, as a set of WIT interfaces, is now standardizing as a WASI World under the WASI Subgroup named wasi-cloud-core. Various open-source projects are working to implement host APIs that are part of wasi-cloud-core, and we encourage you to check them out:

• Spin
• WasmCloud
• Wasmtime, which now ships with a wasi-http world.

We appreciate your continued support.

---

About

SpiderLightning defines a set of *.wit files that abstract distributed application capabilities, such as key-value, messaging, http-server/client and more.

Slight CLI is a runtime for running Wasm applications that compile to WASI and use SpiderLightning capabilities.

This repo also contains libraries that implement of SpiderLightning capabilities to common cloud services including AWS, Azure and more.

Installation

UNIX

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/deislabs/spiderlightning/main/install.sh)"

Windows

iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/deislabs/spiderlightning/main/install.ps1'))

Getting Started

slight relies on wit-bindgen v0.2.0 and currently only supports their language offerings for guest applications (e.g., Rust, C, etc.), plus JavaScript.

JS

slight new -n [email protected] js && cd spidey
# ^^^ starts a new js project under SpiderLightning's v0.5.1 spec

slight buildjs -e slightjs_engine.wasm -o main.wasm src/main.js
# ^^^ builds the js application

slight -c slightfile.toml run main.wasm -l
# At this point, you should see: "Hello, JS Wasm!"

Note: All SpiderLightning dependencies are being injected directly into JavaScript's context. This allows you to write SDK-less applications, but, at the same time, it's a can be a bit less clear what functionalities are available to you. For a comprehensive list of examples on how to use SpiderLightning's capabilities in JS, see here.

C

slight new -n [email protected] c && cd spidey
# ^^^ starts a new c project under SpiderLightning's v0.5.1 spec

# you might want to install wasi-sdk dependencies...
# on unix, run: 
# make install-deps
# on windows, run:
# make install-deps-win

# next, to build...
# on unix, run:
# make bindings && make build
# on windows, run:
# make bindings && make build-win

slight -c slightfile.toml run spidey.wasm
# At this point, you should see: "Hello, SpiderLightning!"

Rust

slight new -n [email protected] rust && cd spidey
# ^^^ starts a new rust project under SpiderLightning's v0.5.1 spec

cargo build --target wasm32-wasi

slight -c slightfile.toml run target/wasm32-wasi/debug/spidey.wasm
# At this point, you should see: "Hello, SpiderLightning!"

Building C Examples

git clone https://github.com/deislabs/spiderlightning.git && cd spiderlightning/ # clone our repo locally and go into it
make install-deps # installs the WASI-SDK
make build # builds SpiderLightning/Slight
make build-c # builds our c example
make run-c # runs our c example

Building Rust Examples

There are also Rust examples that can be built (build-rust) and ran (run-rust). However, we do not recommend running them because some of these examples have dependencies on environment variables or local programs (e.g., etcd), so it probably won't work right off the gate like the C one did.

Repository Structure

• /crates: runtime, core library and service implementations
• /docs: useful documentation to help you understand design decisions, and how to contribute
• /examples: Slight examples
• /proposals: design documents
• /src: the SpiderLightning cli (i.e., Slight)
• /templates: templates used by slight add to create a new Rust or C project
• /tests: integration tests
• /wit: the SpiderLightning specification written in *.wit format (see WIT)

Looking for Contributors

Do you want to contribute to SpiderLightning's growth?

Start with our CONTRIBUTING.md

Aside from the CONTRIBUTING.md doc, here are a few other good starting points to contributing:

• the service-implementation-101.md doc: a step-by-step guide to develop your first service implementor, and
• the SpiderLightning YouTube Playlist: a collection of informative and tutorial videos on SpiderLightning/slight.

FAQ

What problems does SpiderLightning address?

SpiderLightning is a collection of common application interfaces exposed through WebAssembly intended to make cloud-native application development simpler. These interfaces are available to WebAssembly applications through the slight Command Line Interface (CLI) and host runtime that implements these interfaces using backing implementations of your choosing. For example, for the "message queue" interface, SpiderLightning may provide backends implemented by Azure Service Bus, Apache Kafka, and more.

SpiderLightning's interfaces offer developers a set of provider-agnostic APIs which enable developers to write portable applications without having to take direct dependencies on vendor specific APIs and SDKs.

Applications targeting SpiderLightning can leverage these interfaces to reduce the amount of code written to achieve tasks such as persisting key/values, participating in pub/sub, handling messages from a message queue, and much more. By reducing the code footprint, SpiderLightning also enables application binaries to be much smaller, often an order of magnitude smaller than similar container-based applications. This feature further increases the portability of applications to target constrained runtime environments like edge devices.

Check out this talk from the Cloud Native Rejekts conference for more about SpiderLightning's goals and design.

What is the difference between SpiderLightning and WebAssembly System Interface (WASI)?

WASI is a set of standardized APIs for Wasm. Its first preview version is a set of POSIX-like APIs to enable Wasm applications to run on a variety of operating systems. WASI Preveiw 2 is much more modular, adds the Wasm Component Model type system and introduces the concept of "worlds" with the WIT IDL to WASI. SpiderLightning could be seen as a WASI World that provides state management, event-driven patterns, and distributed locking.

SpiderLightning has evolved to wasi-cloud-core and is now being standardized under the WASI Subgroup. See here

What is the difference between SpiderLightning and Dapr?

SpiderLightning and Dapr share the same goal of providing capabilities to distributed applications, but each project's approach to achieving this goal is very different. For example, while Dapr runs as either a sidecar container or one-per-node container and provides an HTTP/gRPC interface for applications, slight links applications directly to generated WASM bindings, which means all calls are executed in-process.

Why the name "SpiderLightning"?

Spider Lightning is the name of a phenomenon of "long, horizontally travelling flashes often seen on the underside of [..] clouds" (source: nssl), pictured on the right. From that and from the fact we are developing SpiderLightning/slight based off of WebAssembly's lightning-fast technologies, the name fit.

spiderlightning's Issues

set up containerd to run slight

[slight] impl add command to slight to download interfaces of a specific version

What's the user experience we want for developers making back-end services w/ Slight? That is, say that they have slight installed on their local machine or server... How do we want them do make use of our interfaces? Are we expecting them to just to go GitHub and copy-and-paste the .wit file?

I think it'd be interesting to, one day, give our slight CLI app the functionality of allowing the user to add capabilities to their project.

$ slight add [email protected]

After this, download the interface.

allow multiple secret-stores (i.e., per capability)

Describe the solution you'd like
Currently, we only allow the user to set one secret_store in the toml file. That means that, in a multi-capability example, all capabilities will need to search for their configs in the exact same way. We'd like to change this behaviour to allow capability specific secret configuration.

Additional context
n/a

avoid cloning `self` when pushing new resource to resource map

in theory, would it be possible to avoid cloning self here and elsewhere if Self implemented something like the ToBox trait?

Originally posted by @arschles in #46 (comment)

Inject credentials to gh action to allow agent to run on azure/aws/gcp

Improve our secrets/settings situation

Describe the solution you'd like
There are a couple of ways to tackle this ranging from simply documenting all the environment variables required to use a service, to something I mentioned in #71 like implementing a key-vault service and dynamically getting secrets/settings as per the user specification.

Additional context
None.

Feature: implement resource configuration

a Resource Configuration is defined to be a map of properties and values. It can be created in the Wasm app to configure the wasi-cloud host capabilities.

// capability configuration is just a map of properties 
capConfig := map[string]string{
 "type" : "AZURE_SERVICE_BUS",
 "name" : "insurance_policies",
 /* free standing key/value form eg:*/
  "connectionString" : "....", 
}
// TODO: Configuring logical queue
// Then it can be used to configure the host from the WASM app
// note cap configuration  created by WASM app is also a resource
capDescriptor = ConfigureCapability(capConfig)
// now WASM app can use it 
queueDescriptor := GetQueue("insurance_policies")

Discussion: wasi-cloud dynamic configuration

Intro

Currently the way wasi-cloud consumes the application configuration is not ideal. We require developer to bring two wasm modules, one for the application, and another one for the configuration. Here is an example

wasi-cloud -m update-blob.wasm -c blob-config.wasm

Having to write code purely for some key-value configuration does not represent a good developer experience. Developers want a simple manifest file, like in YAML or TOML format, to define the key-value configs, or the ability to simply define configs inside their application.

# wasi-cloud.toml
[configs.blob]
url = "azblob://my-container'"

or

fn main() -> Result<()> {
    let resource_descriptor = get_kv("azblob://my-container")?; // telling wasi-cloud I want a azure blob host capability
    ...
}

WASM Dependency

There are some challanges to implement the host if the configuration are defined in the wasm application. Currently, wasi-cloud will parse the path url and link to the specific host capability using dynamic dispatch. See here:

match parsed.scheme() {
        "azblob" => {
            KvAzureBlob::add_to_linker(linker)?;
            KvAzureBlob::build_state(parsed)
        },
        "file" => {
            KvFilesystem::add_to_linker(linker)?;
            KvFilesystem::build_state(parsed)
        },
        ...
    }

One challange is that, in order to get the config, we need to first run the wasm module, but running the module will give us error because the wasm module expects imports for the respective host functionalities, such as kv.get, set, etc, which hasn't been linked yet.

The host could examine the import functions of the wasm module, and determine which host capability needs to be linked first, before running the module. This, however, requires the host to link to a specific host capability, like Azure Blob Storage, for a capability interface. What if the config turns out to be that the user wants a filesystem host implementation?

Thus, we must know the configs before linking the modules.

Solution

A possible solution for this is that we build a dispatcher capability based on resource_descriptor. For example, a kv_dispatcher can consume a url which specifies which host provider the module wants to use, and based on the scheme of the url dispatching to specific host provider functionality.

impl kv::Kv for KvDispatcher {
type ResourceDescriptor = u64;
    fn get_kv(&mut self,path: &str,) -> Result<Self::ResourceDescriptor,Error> {
        let parsed = Url::parse(path).unwrap();
        match parsed.scheme() {
            "azblob" => {
                KvAzureBlob::get_kv(url)?
            },
            "file" => {
                KvFilesystem::get_kv(url)?
            },
            _ => bail!("invalid url: {}, currently wasi-cloud kv interface only supports 'file', 'azblob'", parsed),
        }
    }
}

A great thing about this implementation is that we only need to implement one binding for the generated trait from wit-bindgen and all host provider implementations can be implemented without taking a dependency on wit-bindgen.

The dispatcher will redirect the function to a specific host provider function based on the resource_descriptor.

Static Config

A static config, or manifest file, is a schematized yaml/toml formatted file to configure wasi-cloud host providers. Here is an example:

specversion = 1
name = 'my app'

[configs]

[configs.blob]
url = "azblob://my-container"

[configs.pubsub]
url = "gcppubsub://my-topic"

This manifest file could either be stored in the local filesystem packged up to an image together with the wasm modules, or can live in Kubernetes ConfigMap or Volumns.

Dyanmic Config

@devigned mentioned that we would like to have dynamic configuration. The config file can be changed during runtime, and the application running in wasi-cloud should not be forced to restart. A possible solution is to provide an interface to access external application configuration providers. Here are some common providers (from go-cloud:

1. Local file system: "file:///path/to/config.txt"
2. Blob: "azblob://myvar.txt"
3. AWS Parameterstore: "awsparamstore://myvar?region=us-west-1"
4. GCP Runtime Configurator: "gcpruntimeconfig://myvar"
5. Azure App Configuration: "azappconfig://myvar"
6. etcd: "etcd://myvar"

Application that wishes to adapt to the changing configuration from external provider can use the watch() function on the configuration and it returns a snapshot for obtaining a version of config values.

Secrets Store

Besides application configuration, developers may find an interface to access secrets store to be useful. For example, they can store database connection secrets to Azure Key Vault or AWS KMS.

1. Azure keyvault: "azkeyvault://mykey"

create a real-world example usage of slight

Describe the solution you'd like
Here are some possible examples:

• ping pong game,
• http server,
• web crawler

Additional context
n/a

Move `events_handler` out of `runtime/resource.rs`

Design `guest_resource` as a generic, dynamic type, as host resources are

Refactor: better error handling

As of now, if an error occurs in a service implementation, they are surfaced as:

• descriptor-error,
• io-error, or
• other-error.

This isn't descriptive enough, so we have to think how we want to improve upon it.

Feature: implement a C exmaple for the key-value store

We should have an exmaple written in Go-lang to demonstrate that wasi-cloud provides a consistent interface for application developers using different programming languages.

Note that wit-bindgen currently only support C and Rust guest programming languages. We will leverage on the cgo package to directly call C functions in Go and use TinyGo compiler to compile to wasi32-wasm target.

Add cache to gh action to reduce build time.

add integration tests for all capabilities in GH Actions

This will also involve adding repository secrets for all capabilities that require secrets accessible through envvars.

• #114
• #115
• #185

Design a way to distribute WIT files

From Ralph:

how should we share the .wit, as an ecosystem? The BCA is trying to use bindle as a default impl while they work over how a registry should be built, but we can easily compile and share the .wit modules in ACR. Doing so would mean that you work against versions but would have to pull them in.

I think this is tightly related to #35

Spec: design an interface for a simple message queue

The interface should have common message queue functionalities like receive() and send().

Propagate errors from guest to host

Consider the following guest program written in Rust

fn main() -> Result<()> {
    anyhow::bail!("just wrong")
}

When we run it with wasi-cloud, we get the following stdout:

(base) ➜  wasi-cloud git:(main) ✗ wasi-cloud -m tests/kv-test/target/wasm32-wasi/debug/kv-test.wasm -c tests/wc.toml
Error: just wrong
(base) ➜  wasi-cloud git:(main) ✗ echo $?
0

The status code 0 indicates that the program exits successfully, but clearly is not.

We want the host to propogate the error to the user.

If we change the line to panic, the host will know the wasm module panics and then surface the panic to the user:

fn main() -> Result<()> {
    panic!("just wrong")
}

thread 'main' panicked at 'just wrong', src/main.rs:9:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: wasm trap: wasm `unreachable` instruction executed
wasm backtrace:
    ...

How can we achieve the same thing with anyhow::Result<>? @danbugs

Feature: Implement Kafka host functionality for message queue

This depends on #7 spec.

refactor implementations of the same capability to be dispatched to the same resource (i.e., similar to the `runtime_configs` crate)

love this! Do you think we should refactor other capabilities to do similar dynamic dispatcher? We can combine both filekv and azblobkv together and etc.

Originally posted by @Mossaka in #85 (comment)

create Azure App Configuration implementation for config capability

Describe the solution you'd like
As soon as #85 gets merged, our config capability will be capable of dynamically dispatching to two implementations of the interface:

• usersecrets: quick and easy configs to set up development configs/secrets that encrypts values directly in your slightfile.toml, and
• envvars: just uses the host environment's env vars.

In addition to these, we'd like to be able to dynamically dispatch to an implementation powered by Azure App Configuration.

Additional context
n/a

impl watch for kv filesystem

Test: Add integration tests to the repo

So far, we have implemented a working example, which compiles to a wasm module and can run by wasi-cloud CLI.

What we need is automated integration tests for the host implementations.

impl watch for azure blob storage

Design: Blob storage interface

I had a discussion with @devigned and we both felt like Azure Blob Storage should implement a blob store interface, instead of key-value interface. As a replacement, we can implement Redis, CosmosDB or DynamoDB for the key-value interface. As for the blob storage implementation, we should have Azure Blob Storage, and AWS S3.

Equalize the semantic between azblobkv and fileskv regarding container creation

If the KV is using a filesystem, then a directory will be created under /tmp when get_kv() is being called. However, if the KV is using Azure Blob Storage, a container will not be created by the host. This is done by the user in Azure.

Do we want to maintain this level of consistency across different resources using the same capability? This seems to me depends on the underlying implementation. Filesystem and azure blob storage surely have very different access control model and different permissions.

Docs: Improve Documentation

• README,
• CONTRIBUTING, and
• code comments.

`resource_map` is locked for `exec()` to invoke resource functions

Should we allow event handler to listen to events?

If we allow event listening in event handler, this has fundamentally implications to our current design.

First, wasi-cloud uses a pair of two wasmtime::Store, Linker and Instance to run the main program and the handler function. If the function bounces back to main to call another handler function, I assume we should have more wasmtime::Instance instantiated in the host.

Second, this could only be a possibility if multi-handler #68 is implemented.

Feature: Implement a Azure Blob Storage binding for the kv interface

We should have a cloud storage to prove that our kv interface could work.

make a SDK for Slight

Today wasi-cloud host is a pure CLI implementation. Just like wasmtime has both embedded SDK and CLI implementations, wasi-cloud should provide language dependent SDKs to allow developers embed wasi-cloud as their wasm runtime. wasi-cloud could expose a list of add_to_linker functions that add each capability/resource APIs to wasmtime::Linker.

create aws/gcp implementations for kv and mq

Describe the solution you'd like

• mq aws implementation
• kv gcp implementation

Additional context
n/a

Spec: design an interface for pub/sub.

A related question: should we abstract away the difference between message queue and pub/sub?

I see the difference is:

1. messages in message queue will only be consumed once. If there are N consumers, each message is still consumed once. So it's N times faster
2. in contrast, subscribers to a topic will all get notified when a message is sent by a publisher to the topic.

Let me know your thoughts!
@devigned @bketelsen @danbugs

#18 subtask: design and implement multi-handler and handler vtable

CI build time significantly increased

the CI build time increased from ~2-3 mins to 10mins with the most recent merge to main.

https://github.com/deislabs/spiderlightning/runs/7453559521?check_suite_focus=true#step:7:9

I wonder if the refactoring on the slight negatively affects the cache in our CI build action. @danbugs

create a sql capability

Allow coordinated cancellation

See details in https://github.com/Mossaka/wasi-cloud/blob/main/docs/design-principles.md#coordinated-cancelation-support

Refactor: clean all the `unwrap()` statements

Spec: Design an lease interface

A distributed lock service is common in designing CQRS distributed systems. We would like wasi-cloud to offer a consistent interface for grabbing a lock.

The host implementation could be Etcd, Kubernetes API, Azure Storage Blob lease, etc..

Feature: Implement a Azure Service Bus host functionality for message queue

Once the spec #7 is designed, we should provide a crate that implements a host implementation of the message queue. The host could either be a Kafka queue or Azure Service Bus.

create RedisLock implementation for lockd

Describe the solution you'd like
see redlock (RedisLock) for more info: https://github.com/badboy/redlock-rs

Additional context
There are some other options mentioned here: https://github.com/deislabs/wasi-cloud/blob/main/docs/primer.md

Draft initial implementation for `events` and `event-handler`s in the host

Design "plugin model" for host services

Ideally, we should abstract the host away from capabilities. In other words, the host does not depend on specific resources. The wasi-cloud could have a profile that's similar to wasi, and this profile will define the APIs the host exposes to the client.

https://github.com/deislabs/wasi-cloud/blob/107fc2d4cac66ccd6ed85814234bbeebd71e23c4/src/bin/wasi-cloud.rs#L51

Make pubsub and lockd user interaction more generic

Feat: change name to SpiderLightning

We finally decided to officially rename wasi-cloud to SpiderLightning. Here is a list of items to track

• update repo name
• update everything wasi-cloud to spiderLightning and wasi-cloud-cli binary name to slight
• add new logo
• update documentation

Perform Soak Testing

• design user workloads
• run them on a cluster that runs wasi-cloud
• check memory allocation, database usage, potential performance degradation, TCP drop rates, response time deterioration over a long period of time.

separate pubsub to two resources

I am thinking about seperating pub/sub to two resources, sort of like

get-pub: function(name: string) -> ...
get-sub: function(name: string) -> ...

In the host, we verify that the send-message can only be called on pub descriptor and receive-message can only be called on sub descriptor. .

In this way, we could have

1. named resources
2. multi-pub and multi-sub. One pub can send messsages to multiple subs, and one sub can subscribe to multiple pubs.

Originally posted by @Mossaka in #71 (comment)

design event struct to follow CloudEvents spec

create a http server capability

#18 epic: allow user to "watch" for events/changes

@devigned and @khenidak discussed we want some interface for the event-sourcing, or change data capture( CDC) pattern.

events.
  Listen([kv.On("somefilter"), sql.On("some other filter")]).
  Exec(|event| {
    code...
  })

kv:= get_kv(some_name)
kv.On('someFilter', 'some description of events').Exec(kvInterface, k, v){
code...
}

There will be a series of PRs to tackle on this problem.

• #51
• #50
• #66
• #65
• #140
• #141
• #68
• #110
• #111
• #112
• #113
• #63
• #53