go-plugin is a Go (golang) plugin system over WebAssembly (abbreviated Wasm).
As a plugin is compiled to Wasm, it can be size-efficient, memory-safe, sandboxed and portable.
The plugin system auto-generates Go SDK for plugins from Protocol Buffers files.
While it is powered by Wasm, plugin authors/users don't have to be aware of the Wasm specification since the raw Wasm APIs are capsulated by the SDK.
It uses the same definition as gRPC, but go-plugin communicates with plugins in memory, not over RPC.
It is inspired by hashicorp/go-plugin.
The Go plugin system supports a number of features:
Auto-generated Go interfaces: The plugin system generates Go code for hosts and plugins from Protocol Buffers files like gRPC.
It is easy to learn how to use go-plugin for protobuf/gRPC users.
Plugins are Go interface implementations: Raw Wasm APIs are hidden so that user can write and consume plugins naturally. To a plugin author: you just implement an interface as if it were going to run in the same process. For a plugin user: you just use and call functions on an interface as if it were in the same process. This plugin system handles the communication in between.
Safe: Wasm describes a memory-safe, sandboxed execution environment. Plugins cannot access filesystem and network unless hosts allow those operations. Even 3rd-party plugins can be executed safely. Plugins can't crash the host process as it is sandboxed.
Portable: Wasm is designed as a portable compilation target for programming languages. Plugins compiled to Wasm can be used anywhere. A plugin author doesn't have to distribute multi-arch binaries.
Efficient: The Wasm stack machine is designed to be encoded in a size- and load-time-efficient binary format.
Bidirectional communication: Wasm allows embedding host functions. As Wasm restricts some capabilities such as network access for security, plugins can call host functions that explicitly embedded by a host to extend functionalities.
Stdout/Stderr Syncing: Plugins can use stdout/stderr as usual and the output will get mirrored back to the host process. The host process can control what io.Writer is attached to stdout/stderr of plugins.
Protocol Versioning: A very basic "protocol version" is supported that can be incremented to invalidate any previous plugins. This is useful when interface signatures are changing, protocol level changes are necessary, etc. When a protocol version is incompatible, a human friendly error message is shown to the end user.
go-plugin generates Go SDK for a host and TinyGo SDK for plugins.
As the Wasm support in Go is not mature, plugins need to be compiled to Wasm by TinyGo, which is an alternative compile for Go source code, at the moment.
The plugin system works by loading the Wasm file and communicating over exporting/exported methods.
This architecture has a number of benefits:
- Plugins can't crash your host process: a panic in a plugin is handled by the Wasm runtime and doesn't panic the plugin user.
- Plugins are very easy to write: just write a Go application and
tinygo build. - Plugins are very easy to distribute: just compile the TinyGo source code to the Wasm binary once and distribute it.
- Plugins are very easy to install: just put the Wasm binary in a location where the host will find it.
- Plugins can be secure: the plugin is executed in a sandbox and doesn't have access to the local filesystem and network by default.
Download a binary here and put it in $PATH.
To use the plugin system, you must take the following steps.
These are high-level steps that must be done.
Examples are available in the examples/ directory.
- Choose the interface(s) you want to expose for plugins.
- Generate SDK for a host and plugin by
go-plugin. - Implement the Go interface defined in the plugin SDK.
- Compile your plugin to Wasm.
- Load the plugin and call the defined methods.
The development flow is as below.
Let's create a hello-world plugin.
Install the following tools:
Create greeting.proto.
syntax = "proto3";
package greeting;
option go_package = "github.com/knqyf263/go-plugin/examples/helloworld/greeting";
// The greeting service definition.
// go:plugin type=plugin version=1
service Greeter {
// Sends a greeting
rpc SayHello(GreetRequest) returns (GreetReply) {}
}
// The request message containing the user's name.
message GreetRequest {
string name = 1;
}
// The reply message containing the greetings
message GreetReply {
string message = 1;
}Most of the definitions are simply as per the Protocol Buffers specification.
The only difference is the line starting with // go:plugin.
It defines parameters for go-plugin.
type=plugin means the service defines the plugin interface.
Run the following command.
$ protoc --go-plugin_out=. --go-plugin_opt=paths=source_relative greeting.protoThen, you will find 4 files generated in the same directory, greet.pb.go, greet_host.pb.go, greet_plugin.pb.go and greet_vtproto.pb.go.
The Greeter interface is generated as below in the previous step.
type Greeter interface {
SayHello(context.Context, GreetRequest) (GreetReply, error)
}A plugin author needs to implement Greeter and registers the struct via RegisterGreeter.
In this tutorial, we use plugin.go as a file name, but it doesn't matter.
//go:build tinygo.wasm
package main
import (
"context"
"github.com/path/to/your/greeting"
)
// main is required for TinyGo to compile to Wasm.
func main() {
greeting.RegisterGreeter(MyPlugin{})
}
type MyPlugin struct{}
func (m MyPlugin) SayHello(ctx context.Context, request greeting.GreetRequest) (greeting.GreetReply, error) {
return greeting.GreetReply{
Message: "Hello, " + request.GetName(),
}, nil
}Then, compile it to Wasm by TinyGo.
$ tinygo build -o plugin.wasm -scheduler=none -target=wasi --no-debug plugin.goLoad the plugin binary and call SayHello.
package main
import (
"context"
"fmt"
"log"
"github.com/path/to/your/greeting"
)
func main() {
ctx := context.Background()
// Initialize a plugin loader
p, err := greeting.NewGreeterPlugin(ctx)
if err != nil {...}
defer p.Close(ctx)
// Load a plugin
plugin, err := p.Load(ctx, "path/to/plugin.wasm")
if err != nil {...}
// Call SayHello
reply, err := plugin.SayHello(ctx, greeting.GreetRequest{Name: "go-plugin"})
if err != nil {...}
// Display the reply
fmt.Println(reply.GetMessage())
}$ go run main.go
Hello, go-pluginThat's it! It is easy and intuitive.
You can see the hello-world example here.
Wasm has limited capability as it is secure by design, but those can't be achieved with Wasm itself.
To expand the capability, many compilers implement system calls using WebAssembly System Interface (WASI).
But it is still draft (wasi_snapshot_preview1) and some functions are not implemented yet in wazero that go-plugin uses for Wasm runtime.
For example, sock_recv and sock_send are not supported for now.
It means plugins don't have network access.
Host functions can be used for this purpose. A host function is a function expressed outside WebAssembly but passed to a plugin as an import. You can define functions in your host and pass them to plugins so that plugins can call the functions. Even though Wasm itself doesn't have network access, you can embed such function to plugins.
You can define a service for host functions in a proto file.
Note that // go:plugin type=host is necessary so that go-plugin recognizes the service is for host functions.
The service name is HostFunctions in this example, but it doesn't matter.
// go:plugin type=host
service HostFunctions {
// Sends a HTTP GET request
rpc HttpGet(HttpGetRequest) returns (HttpGetResponse) {}
}NOTE: the service for host functions must be defined in the same file where other plugin services are defined.
Let's say Greeter is defined in the same file as HostFunctions.
Then, Load() will be able to take HostFunctions as an argument as mentioned later.
// go:plugin type=plugin version=1
service Greeter {
rpc SayHello(GreetRequest) returns (GreetReply) {}
}go-plugin generates the corresponding Go interface as below.
// go:plugin type=host
type HostFunctions interface {
HttpGet(context.Context, HttpGetRequest) (HttpGetResponse, error)
}Implement the interface.
// myHostFunctions implements HostFunctions
type myHostFunctions struct{}
// HttpGet is embedded into the plugin and can be called by the plugin.
func (myHostFunctions) HttpGet(ctx context.Context, request greeting.HttpGetRequest) (greeting.HttpGetResponse, error) {
...
}And pass it when loading a plugin.
As described above, Load() takes the HostFunctions interface.
greetingPlugin, err := p.Load(ctx, "plugin/plugin.wasm", myHostFunctions{})Now, plugins can call HttpGet().
You can see an example here.
You can define an interface version in the // go:plugin line.
// go:plugin type=plugin version=2
service Greeter {
// Sends a greeting
rpc Greet(GreetRequest) returns (GreetReply) {}
}This is useful when interface signatures are changing. When an interface version is incompatible, a human friendly error message is shown to the end user like the following.
API version mismatch, host: 2, plugin: 1Refer to this example.
TinyGo currently doesn't support encoding/json.
https://tinygo.org/docs/reference/lang-support/stdlib/
You have to use third-party JSON libraries such as gjson and easyjson.
Also, you can export a host function. The example is available here.
fmt.Printf can be used in plugins if you attach os.Stdout as below. See the example for more details.
mc := wazero.NewModuleConfig().
WithStdout(os.Stdout). // Attach stdout so that the plugin can write outputs to stdout
WithStderr(os.Stderr). // Attach stderr so that the plugin can write errors to stderr
WithFS(f) // Loaded plugins can access only files that the host allows.
p, err := cat.NewFileCatPlugin(ctx, cat.WazeroModuleConfig(mc))If you need structured and leveled logging, you can define host functions so that plugins can call those logging functions.
// The host functions embedded into the plugin
// go:plugin type=host
service LoggingFunctions {
// Debug log
rpc Debug(LogMessage) returns (google.protobuf.Empty) {}
// Info log
rpc Info(LogMessage) returns (google.protobuf.Empty) {}
// Warn log
rpc Info(LogMessage) returns (google.protobuf.Empty) {}
// Error log
rpc Error(LogMessage) returns (google.protobuf.Empty) {}
}A plugin author can use OCI registries such as GitHub Container registry (GHCR) to distribute plugins.
Push:
$ oras push ghcr.io/knqyf263/my-plugin:latest plugin.wasm:application/vnd.module.wasm.content.layer.v1+wasmPull:
$ oras pull ghcr.io/knqyf263/my-plugin:latestYou can refer to https://wazero.io/languages/tinygo/.
go-plugin uses wazero for Wasm runtime.
Also, it customizes protobuf-go and vtprotobuf for generating Go code from proto files.
Launching a plugin as a subprocess is not secure. In addition, plugin authors need to distribute multi-arch binaries.
Why not the official plugin package?
It is not schema-driven like Protocol Buffers and can easily break signature.
Why not using protobuf-go directly?
TinyGo doesn't support Protocol Buffers natively as of today.
go-plugin generates Go code differently from protobuf-go so that TinyGo can compile it.
You might be aware that your generated code imports github.com/knqyf263/go-plugin/types/known, not github.com/protocolbuffers/protobuf-go/types/known when you import types from google/protobuf/xxx.proto (a.k.a well-known types) in your proto file.
As described above, TinyGo cannot compile github.com/protocolbuffers/protobuf-go/types/known since those types use reflection.
go-plugin provides well-known types compatible with TinyGo and use them.
Why using // go:plugin for parameters rather than protobuf extensions?
An extension must be registered in Protobuf Global Extension Registry to issue a unique extension number.
Even after that, users needs to download a proto file for the extension.
It is inconvenient for users and the use case in go-plugin is simple enough, so I decided to use comments.
Go doesn't support WASI. You can see other reasons here. We might be able to add support for Go as an experimental feature.
go-plugin currently supports TinyGo plugins only, but technically, any language that can be compiled into Wasm can be supported.
Welcome your contribution :)
- Specification
- Packages
- Messages
- Nested Types
- Fields
- Singular Message Fields
- double
- float
- int32
- int64
- uint32
- uint64
- sint32
- sint64
- fixed32
- fixed64
- sfixed32
- sfixed64
- bool
- string
- bytes
- Repeated Fields
- Map Fields
- Oneof Fields (not planned)
- Singular Message Fields
- Enumerations
- Extensions (not planned)
- Services
- Well-known types
- Any (Some functions/methods are not yet implemented)
- Api
- BoolValue
- BytesValue
- DoubleValue
- Duration
- Empty
- Enum
- EnumValue
- Field
- Field_Cardinality
- Field_Kind
- FieldMask
- FloatValue
- Int32Value
- Int64Value
- ListValue
- Method
- Mixin
- NullValue
- Option
- SourceContext
- StringValue
- Struct
- Syntax
- Timestamp
- Type
- UInt32Value
- UInt64Value
- Value
- Generate codes
- Structs without reflection
- Marshaling/Unmarshaling
- Host code calling plugins
- Plugin code called by host
- Interface version
- Host functions
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent