hashicorp / go-plugin Goto Github PK
View Code? Open in Web Editor NEWGolang plugin system over RPC.
License: Mozilla Public License 2.0
Golang plugin system over RPC.
License: Mozilla Public License 2.0
I might be missing some detail, but I'm having a really difficult time following the code to try and understand the protocol used here. I'm not a Go-lang developer, and I'm trying to implement the protocol used in node for that very reason.
I've gotten as far as the initial handshake and then the code just loses all meaning to me. Too many levels of indirection to other libraries. It might have more meaning to someone more familiar with the Go stack.
Here's what I've been able to follow so far:
BaseProtocolVersion|AppPluginProtocolVersion|SocketProtocol|SocketAddress
After that I'm lost in multiplexers, redirected stdio streams, and go-specific network implementations. Once the plugin process starts listening on tcp/unix domain socket, I presume that the host application makes a connection to the socket. Who talks next, and what is the protocol?
Is there a flowchart or block diagram somewhere? Anything that describes the control flow for this protocol?
I don't expect a 100-page manual, but some next steps would be helpful. I'd be happy to produce some documentation for other people making this attempt if someone can help me get over this part of the learning curve.
This way i can lock Go app from changes at this module
Hey
I'm posting after trying for hours, the plugin structure is working fine, but setting up the bidirectional communication for my program is proving to be a real challenge.
I think I understand how this works, but probably I'm missing something important. Basically I've started the plugin, and extracted the broker to use it in the main code (host), it looks like this:
// Assign the implementation, calls to this are RPC now
test := raw.(sdk.Test_rpc_container)
// Bootstrap the connections to the module?
h := HeartbeatService{}
hbServiceServer := &sdk.GRPCHBServiceServer{Impl: &h}
var svr *grpc.Server
serverFunc := func(opts []grpc.ServerOption) *grpc.Server {
svr = grpc.NewServer(opts...)
proto.RegisterHeartbeatServer(svr, hbServiceServer)
return svr
}
go test.Broker.AcceptAndServe(1, serverFunc)
so the Broker should be listening to connections on channel 1. It's hardcoded, yes.
And in the plugin I have this:
func (s *FuncRPCServer) Call(ctx context.Context, req *proto.CallRequest) (*proto.CallResponse, error) {
conn, err := s.broker.Dial(1)
if err != nil {
return nil, errors.New("I HAZ FAIL")
}
This setup is very close to the bidirectional example, the RPC server creates the channel once the function Call gets called, yet it goes into timeout after 5 secs failing here.
So... this is the point where I ask for help, as the documentation doesn't seem to describe anything particular about this.
Please tell me if you need more details. Thanks for the useful library.
Thanks
Problem first happened with my teams project, but recreated the issue using the go-plugins example to be sure.
Steps:
Container built FROM golang:latest
run interactively
1 go get github.com/hashicorp/go-plugin
2 cd src/github.com/hashicorp/go-plugin/examples/grpc/
3 go build -o kv
4 go build -o kv-go-grpc ./plugin-go-grpc
5 export KV_PLUGIN="./kv-go-grpc"
6 ./kv put hello world
root@b93305f57ca2:/go/src/github.com/hashicorp/go-plugin/examples/grpc# ./kv put hello world
2017-10-02T22:51:35.213Z [DEBUG] plugin: starting plugin: path=/bin/sh args="[sh -c ./kv-go-grpc]"
2017-10-02T22:51:35.214Z [DEBUG] plugin: waiting for RPC address: path=/bin/sh
2017-10-02T22:51:35.220Z [DEBUG] plugin.sh: plugin address: timestamp=2017-10-02T22:51:35.220Z address=/tmp/plugin040334221 network=unix
*It just hangs here indefinitely. (Same behavior as in my current project).
If, I do the same for the basic example and it runs fine.
Thanks!
Hey there! When authoring a Vault plugin, if I vendor with any recent version of vault and with revision 1b0f542 or onward of this plugin, the plugin fails to properly communicate with vault.
You can test this by cloning https://github.com/hashicorp/vault-auth-plugin-example, removing the existing vendoring (vendor and Gopkg*) and starting from scratch with this Gopkg.toml
:
[[constraint]]
name = "github.com/hashicorp/vault"
version = "=1.0.1"
[[override]]
name = "github.com/hashicorp/go-plugin"
revision = "1b0f5429182b3157c6f22301a77fde89bab6dbd7"
Failing back to any previous revision and the plugin works as expected. It would appear that the Dial
function is failing in:
Line 376 in f444068
I'm starting here in the event it's not strictly related to vault. Thanks!
As reported in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897504
TestClient_syncStreams
fails as follows:
--- FAIL: TestClient_syncStreams (0.10s)
rpc_client_test.go:59: bad:
stdouttest^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
as the title, thank you
OS: linux
The plugins is abnormally closed, such as SIGKILL, the unix socket file will be left behind, and causing a leak in some case.
why not plugin support "Abstract sockets" to resolve this case.
http://man7.org/linux/man-pages/man7/unix.7.html
Abstract sockets
Socket permissions have no meaning for abstract sockets: the process
umask(2) has no effect when binding an abstract socket, and changing
the ownership and permissions of the object (via fchown(2) and
fchmod(2)) has no effect on the accessibility of the socket.
Abstract sockets automatically disappear when all open references to
the socket are closed.
The abstract socket namespace is a nonportable Linux extension.
WORKING #51
FAILED #52
[root@centos7 go-plugin]# git show --oneline -s
4b3b291 Add a Connection Broker for GRPC plugins (#52)
[root@centos7 go-plugin]# /home/terraform-provider-vcloud-director/go/src/test/test_catalog.sh
2018/01/21 04:50:30 [INFO][PLUGINLOG] __INIT__init
2018/01/21 04:50:30 [INFO][PLUGINLOG] _INIT_TestAccResourceCatalog
=== RUN TestAccResourceCatalogBasic
2018/01/21 04:50:30 [INFO][PLUGINLOG] INIT__LOGGING
2018/01/21 04:50:30 [INFO][PLUGINLOG] _INIT__testAccPreCheck
2018/01/21 04:50:30 [INFO][PLUGINLOG] _DONE__testAccPreCheck
2018-01-21T04:50:30.619-0500 [DEBUG] plugin: starting plugin: path=/usr/bin/sh args="[sh -c python3 /home/terraform-provider-vcloud-director/plugin-python/plugin.py]"
2018/01/21 04:50:30 [INFO][PLUGINLOG] __INIT__func providerConfigure
2018-01-21T04:50:30.620-0500 [DEBUG] plugin: waiting for RPC address: path=/usr/bin/sh
2018-01-21T04:50:31.013-0500 [DEBUG] plugin.sh: INFO:root:Already running / Not starting new server
Error: plugin "PY_PLUGIN" doesn't support gRPC
exit status 1
[root@centos7 go-plugin]#
===============================================================
FAILED #55
[root@centos7 go-plugin]# git show --oneline -s
485ef45 Provide a context for managing the lifecycle of gRPC plugins (#55)
[root@centos7 go-plugin]# /home/terraform-provider-vcloud-director/go/src/test/test_catalog.sh
2018/01/21 04:47:46 [INFO][PLUGINLOG] __INIT__init
2018/01/21 04:47:46 [INFO][PLUGINLOG] _INIT_TestAccResourceCatalog
=== RUN TestAccResourceCatalogBasic
2018/01/21 04:47:46 [INFO][PLUGINLOG] INIT__LOGGING
2018/01/21 04:47:46 [INFO][PLUGINLOG] _INIT__testAccPreCheck
2018/01/21 04:47:46 [INFO][PLUGINLOG] _DONE__testAccPreCheck
2018-01-21T04:47:46.084-0500 [DEBUG] plugin: starting plugin: path=/usr/bin/sh args="[sh -c python3 /home/terraform-provider-vcloud-director/plugin-python/plugin.py]"
2018/01/21 04:47:46 [INFO][PLUGINLOG] __INIT__func providerConfigure
2018-01-21T04:47:46.084-0500 [DEBUG] plugin: waiting for RPC address: path=/usr/bin/sh
2018-01-21T04:47:46.475-0500 [DEBUG] plugin.sh: INFO:root:Already running / Not starting new server
Error: plugin "PY_PLUGIN" doesn't support gRPC
exit status 1
[root@centos7 go-plugin]#
Hi.
Please create a release so changes from pull request #111 can be included into vault sdk
When a plugin function accepts an interface argument, supplied by the base process (as in the bidirectional example), it seems that the parameter is only usable until the plugin function returns and after that the grpc connection for that interface closes.
Is there a recommended way to allow that parameter be usable for as long as the plugin is running? So that I can for instance have an Init(service Service)
method of the plugin which can be used by the base application to supply an instance of the service
interface to the plugin, which can then be called from various go routines after the Init function has returned?
Hey all, I'm trying to make the system I'm building more stable, so I'm at the point where my app should know when a plugin crashes, in order to change a state variable.
Apparently there's no mechanism for getting the event (still it's logged), so I use at the moment the Ping()
function of the RPC client.
Is there a way to get the plugin lifecycle events to make things a bit more clean? A callback like OnModuleKill
would be perfect.
Thanks
Can't implement pluginmap as a map of grpcplugins
var PluginMap = map[string]plugin.GRPCPlugin
Have to use map[string]plugin.Plugin
and implement client and server funcs for both grpc flavor and rpc flavor.
Hi Team ,
is there a recommended way to hook the defer client.Kill() to a terraform providers clean up method .
I could use the [func providerConfigure(d *schema.ResourceData) (interface{}, error) ] to initialize the plugin , but will need to kill the client plugin after all the terraform operations are done
Thanks & Regards,
Sri
Following the README in github.com/hashicorp/go-plugin/examples/grpc
$ go version
go version go1.9.3 darwin/amd64
Executing the commands from the README.
# This builds the main CLI
$ go build -o kv
# This builds the plugin written in Go
$ go build -o kv-go-grpc ./plugin-go-grpc
# This tells the KV binary to use the "kv-go-grpc" binary
$ export KV_PLUGIN="./kv-go-grpc"
# Read and write
$ ./kv put hello world
$ ./kv get hello
world
I get the following output
$ ./kv get hello
2018-02-01T13:24:27.656Z [DEBUG] plugin: starting plugin: path=/bin/sh args="[sh -c ./kv-go-grpc]"
2018-02-01T13:24:27.658Z [DEBUG] plugin: waiting for RPC address: path=/bin/sh
2018-02-01T13:24:27.672Z [ERROR] plugin.sh: protocol init: timestamp=2018-02-01T13:24:27.672Z error=""kv" is not a GRPC-compatible plugin"
2018-02-01T13:24:27.674Z [DEBUG] plugin: plugin process exited: path=/bin/sh
Error: plugin exited before we could connect
Would you ever consider moving to an Apache license and/or dual licensing?
See #20466 in terraform.
I'm working on some changes to rpc_client.go and client.go to implement this.
2017/10/20 09:54:02 ok starting to client
2017-10-20T09:54:02.204-0400 [DEBUG] plugin: starting plugin: path=/usr/bin/sh args="[sh -c python3 /home/terraform-provider-vcloud-director/plugin-python/plugin.py]"
2017-10-20T09:54:02.204-0400 [DEBUG] plugin: waiting for RPC address: path=/usr/bin/sh
panic: runtime error: integer divide by zero
goroutine 14 [running]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc4200c4200, 0x0, 0xc42001e780)
/home/terraform-provider-vcloud-director/go/src/google.golang.org/grpc/clientconn.go:930 +0xf50
google.golang.org/grpc.(*addrConn).connect.func1(0xc4200c4200)
/home/terraform-provider-vcloud-director/go/src/google.golang.org/grpc/clientconn.go:736 +0x2f
created by google.golang.org/grpc.(*addrConn).connect
/home/terraform-provider-vcloud-director/go/src/google.golang.org/grpc/clientconn.go:735 +0x400
[root@worker3 test]#
I see the above error but no specific message why its happening
https://github.com/srinarayanant/terraform-provider-vcloud-director/blob/master/go/src/test/test.go
Above is the link to the source file , used as a test which throws this error.
This was working fine for a while till I had updated rpc interface , but now reverting the changes still the problem persists.
I'm checking the project and I don't find any license. Please, can you license it?
when I tried to build the example code with command "go build -o ./plugin/greeter ./plugin/greeter_impl.go", then I got these error info:
go: google.golang.org/[email protected]: unrecognized import path "google.golang.org/genproto" (https fetch: Get https://google.golang.org/genproto?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/sync" (https fetch: Get https://golang.org/x/sync?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/text" (https fetch: Get https://golang.org/x/text?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/sys" (https fetch: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/net" (https fetch: Get https://golang.org/x/net?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: google.golang.org/[email protected]: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: error loading module requirements
go-plugins can communicate over TLS, but there is no way to pass the TLS certificate to the plugin binary when it is spawned. Afaik the approach to do this currently is wrapping the certificate and key in Vault and passing the wrapped token through an environment variable, but that seems overkill to me. Would passing this over stdout when spawning the plugin be a good alternative? I'm happy to try upgrading the protocol.
I'm trying to run the system on Windows, and the way it uses to connect to plugins is TCP. I see that the functions refer to two environment variables, PLUGIN_MIN_PORT
and PLUGIN_MAX_PORT
. If they are undefined, a clueless error is returned.
I kind of understand why they have to be declared as environment variables, it makes good sense, though the error took me some time to find out what it actually was. Please report the absence of these variables more clearly. Or make it configurable in the code.
Thanks
Filing this here so we can think about it:
hashicorp/terraform#5043 (comment)
Unfortunately we don't have repro, but we do have one user reporting it.
I'm seeing intermittent errors being logged from go-plugin with a message of "reading plugin stderr", and an error of "read |0: file already closed". This seems to happen when plugins are terminated.
In looking at https://golang.org/src/os/exec/exec.go?s=17633:17682#L601, and client.go
, it seems like there may be a race condition where the reader side of the pipe is closed after the command exits, but the go-plugin client is still trying to read from it. The client's logStderr
is looking for an io.EOF
error, but in this case I think it's actually getting some variant of the os.ErrClosed
error. I'm not familiar enough with the code to understand exactly what's going on/what should be changed, though. Hope this can point someone in the right direction!
2017/02/13 22:38:36 [DEBUG] plugin: nomad: panic: close of closed channel
2017/02/13 22:38:36 [DEBUG] plugin: nomad:
2017/02/13 22:38:36 [DEBUG] plugin: nomad: goroutine 81 [running]:
2017/02/13 22:38:36 [DEBUG] plugin: nomad: panic(0x107ae00, 0xc4202b09d0)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: /home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/panic.go:500 +0x1a1
2017/02/13 22:38:36 [DEBUG] plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*controlServer).Quit(0xc42008e588, 0x1, 0x1a27ec8, 0x0, 0x0)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: /home/travis/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_server.go:118 +0x5a
2017/02/13 22:38:36 [DEBUG] plugin: nomad: reflect.Value.call(0xc420020960, 0xc42008e590, 0x13, 0x11f9fcb, 0x4, 0xc420466ef0, 0x3, 0x3, 0xc420260601, 0xc4202606a0, ...)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: /home/travis/.gimme/versions/go1.7.linux.amd64/src/reflect/value.go:434 +0x5c8
2017/02/13 22:38:36 [DEBUG] plugin: nomad: reflect.Value.Call(0xc420020960, 0xc42008e590, 0x13, 0xc420466ef0, 0x3, 0x3, 0x1, 0xc4202e6780, 0x20003)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: /home/travis/.gimme/versions/go1.7.linux.amd64/src/reflect/value.go:302 +0xa4
2017/02/13 22:38:36 [DEBUG] plugin: nomad: net/rpc.(*service).call(0xc420211740, 0xc420211700, 0xc42034aab0, 0xc420098800, 0xc420355760, 0xff8160, 0xc4202b0987, 0x181, 0xfe42e0, 0x1a27ec8, ...)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: /home/travis/.gimme/versions/go1.7.linux.amd64/src/net/rpc/server.go:383 +0x148
2017/02/13 22:38:36 [DEBUG] plugin: nomad: created by net/rpc.(*Server).ServeCodec
2017/02/13 22:38:36 [DEBUG] plugin: nomad: /home/travis/.gimme/versions/go1.7.linux.amd64/src/net/rpc/server.go:477 +0x421
2017/02/13 22:38:36 [DEBUG] plugin: /tmp/nomad-test.UjY/nomad: plugin process exited
Unfortunately do not have reproduction steps or full stack trace as I saw this in travis.
in packer we have a line that looks like
inPlugin := os.Getenv(plugin.MagicCookieKey) == plugin.MagicCookieValue
It would be great if this could be exposed through go-plugin so we can take different actions depending on if we're in a plugin or the main body
I the main program and here is what I'm getting:
bash-3.2$ ./basic
2018-07-05T00:32:05.780+0500 [DEBUG] plugin: starting plugin: path=./plugin/greeter args=[./plugin/greeter]
2018-07-05T00:32:05.784+0500 [DEBUG] plugin: waiting for RPC address: path=./plugin/greeter
2018-07-05T00:32:05.870+0500 [DEBUG] plugin.greeter: message from plugin: foo=bar timestamp=2018-07-05T00:32:05.868+0500
2018-07-05T00:32:05.871+0500 [DEBUG] plugin.greeter: plugin address: address=/var/folders/4v/3yrsz17s1m9bqlj35bgqvyvr0000gn/T/plugin847884065 network=unix timestamp=2018-07-05T00:32:05.871+0500
2018-07-05T00:32:05.877+0500 [DEBUG] plugin.greeter: message from GreeterHello.Greet: timestamp=2018-07-05T00:32:05.876+0500
Hello!
2018-07-05T00:32:05.877+0500 [DEBUG] plugin.greeter: 2018/07/05 00:32:05 [ERR] plugin: plugin server: accept unix /var/folders/4v/3yrsz17s1m9bqlj35bgqvyvr0000gn/T/plugin847884065: use of closed network connection
2018-07-05T00:32:05.878+0500 [DEBUG] plugin: plugin process exited: path=./plugin/greeter
Is "use of closed network connection" expected here? I tried running bidirectional example and there is no problem.
Given that this is MPL licensed library and go uses static compile, can I use this library in a GPLv3 licensed application written in go?
Is it possible to create a net/rpc only implementation (both host and plugin) without bloated by gRPC dependencies?
WIll the binaries always have all modules listed go.sum file?
The CleanupClients method is not defined on a type, and thus does not have direct access to a configured logger. It logs a single line using stdlib log
:
Line 235 in a5174f8
This is problematic because 1) it is not structured or formatted as we would expect it to be, and 2) it goes to stdout regardless of the actual logger configuration.
The package does manage a list of instantiated clients, each of which has a logger defined on its struct. I can think of two solutions:
Thoughts on the best option? Perhaps you can think of other options? Happy to send a patch assuming we are aligned
Hey, is there any sample code, how to handle io.Reader? Especially when they are a return value of a plugin function.
Hi,
Would it be possible to see more implementation examples? I'm wrapping my head around on how to create dynamically loaded plugins ;/
Hi!
We built a build system based on plugins and Im writing a facility which gives the user an ability to cancel a build pipeline. For this to happen I need to actually kill the pipeline which is the plugin binary itself.
I thought of creating an endpoint cancels the server, but is there a way from the client to send a kill signal? Or to look for the binary and kill it with os interrupt?
Hey ppl :-D
Thanks for great system!
Have a feature request, on line https://github.com/hashicorp/go-plugin/blob/master/client.go#L523, the host is passing the whole set of the environment variables to the plugins.
I'd like to have the ability to skip it (for security reasons).
Hello. I would like to know what is the recommended way of handling plugin crashes (i.e. the plugin process shutting down). I noticed go-plugin doesn't restart the process in this situation, so I was wondering if you are considering adding this feature, and if you are not then I would like to know the rationale behind that decision. Initially my idea was to verify the plugin status before any call, and start it if the process is down, but I wanted to check with you guys first in case I'm missing something. Thanks!
With AutoMTLS=true
, brokered connections dialed from the plugin back to the host fail with transport: authentication handshake failed: x509: certificate is valid for localhost, not unused
.
Additionally, brokering with AutoMTLS requires the use of broker.AcceptAndServe since the autogenerated TLS configuration is not exported and therefore cannot be used to set up a gRPC server to use with the listener returned from broker.Accept
.
During debugging of a Terraform run within a container, it's not super clear why a process exited. In the example we had, it was being OOM-killed by the kernel due to memory constraints:
Sep 26 12:01:19 ip-192-168-123-123 kernel: Memory cgroup out of memory: Kill process 2011 (terraform) score 257 or sacrifice child
@radeksimko found that this line seems to be relevant: https://github.com/hashicorp/go-plugin/blob/master/client.go#L571
c.logger.Debug("plugin process exited", "path", cmd.Path)
which meant the logs didn't show the exit process of why the process was killed:
2018-09-26T12:01:48.395Z [DEBUG] plugin: plugin process exited: path=/terraform/.terraform/plugins/linux_amd64/terraform-provider-aws_v1.37.0_x4
It would be beneficial to have an exit code here as well, as having 137
show would help diagnose a SIGTERM due to the OOM killer
I might be missing something completely, but it seems that the stdout/stderr sync feature is not implemented for the GRPC server path.
The GRPCServer.Stdout
and GRPCServer.Stderr
fields are not used anywhere. They are set up however in the same way as in RPCServer
in the common codepath in Serve()
it's just RPCServer has control over the channel, while in GRPC I guess it is not that obvious how the sync can be implemented. Is my understanding correct?
I think this asymmetry should be documented between netRPC and gRPC! Are there any more differences?
Logs in gRPC plugins can work using the log
, hclog
packages, they are pointing to the old stderr.
Workaround for getting the output of fmt.Println()
for example to stdErr
involves saving the old stdErr
in a var and using that (for some reason the same trick does not work with stdOut
):
var oldOut *os.File
var oldErr *os.File
// Here is a real implementation of KV that writes to a local file with
// the key name and the contents are the value of the key.
type KV struct{}
func (KV) Put(key string, value []byte) error {
fmt.Fprintf(oldOut,"Hello, I'm the plugin putter.1\n") # this does not work
fmt.Fprintf(oldErr,"Hello, I'm the plugin putter.2\n") # this works
fmt.Fprintf(os.Stderr,"Hello, I'm the plugin putter.3\n") # this does not work
fmt.Fprintf(os.Stdout,"Hello, I'm the plugin putter.4\n") # this does not work
log.New(os.Stderr, "hihii", 0).Printf("test") # this does not work
log.Printf("log hello:) - %v %v vs %v %v\n", os.Stdout, os.Stderr, oldOut, oldErr) # this works
value = []byte(fmt.Sprintf("%s\n\nWritten from plugin-go-grpc", string(value)))
return ioutil.WriteFile("kv_"+key, value, 0644)
}
func (KV) Get(key string) ([]byte, error) {
return ioutil.ReadFile("kv_" + key)
}
func main() {
oldOut = os.Stdout
oldErr = os.Stderr
plugin.Serve(&plugin.ServeConfig{
HandshakeConfig: shared.Handshake,
Plugins: map[string]plugin.Plugin{
"kv_grpc": &shared.KVGRPCPlugin{Impl: &KV{}},
},
// A non-nil value here enables gRPC serving for this plugin...
GRPCServer: plugin.DefaultGRPCServer,
})
}
In comparison the Sync feature does work well in the netRPC plugin when SyncStdout
and SyncStderr
are set on the plugin.ClientConfig
.
I was able to split the basic example into 2 binaries, I can get the response back to my host from the plugin, but I can't understand how the plugin can call into the host.
When closing the server with CTRL+c, a panic occurs:
panic: interface conversion: interface is float64, not string
goroutine 131 [running]:
panic(0xc524e0, 0xc4209df100)
/home/bro/programmer/go/go/src/runtime/panic.go:500 +0x1a1
github.com/hashicorp/go-plugin.parseJSON(0xc420445ef0, 0x84, 0x1, 0x1420360, 0xc4209fcbe0)
/home/bro/programmer/fabriscale/go/ws/src/github.com/hashicorp/go-plugin/log_entry.go:118 +0x4fc
github.com/hashicorp/go-plugin.(*Client).logStderr(0xc4205172d0, 0x1411be0, 0xc420036200)
/home/bro/programmer/fabriscale/go/ws/src/github.com/hashicorp/go-plugin/client.go:746 +0x277
created by github.com/hashicorp/go-plugin.(*Client).Start
/home/bro/programmer/fabriscale/go/ws/src/github.com/hashicorp/go-plugin/client.go:544 +0xb3c
I see it on the roadmap, but no other mention. Any chance for this to see daylight?
Could you please clarify for me if go-plugin
can be use in a way that the host application and the plugins are on different machines?
Since it uses gRPC
or net/rpc
, I would expect that it should be doable.
But the net/rpc
connection is forced to use unix
or 127.0.0.1
on Windows machine
Nomad uses go-plugin to spin up various plugins and auxiliary processes, and saw surprising (to us) behavior when host process dies in hashicorp/nomad#5598 .
Nomad uses go-plugin to spin up long-running plugins with lifecycle independent from host, to ease in-place upgrades and reconfiguration, and use the reattachment patterns ReattachConfig
supported by this.
However, we observe the following problems after host process is restarted:
The plugin gets a SIGPIPE signal upon the next log/Stdout/Stderr write operation. When the host (e.g. go-plugin client) process dies, Stdout/Stderr pipe closes and any write from plugin fails with io.ErrClosedPipe
error, and the plugin receive SIGPIPE, typically killing it.
SIGPIPE
, hclog may panics on log write failure in https://github.com/hashicorp/go-hclog/blob/6907afbebd2eef854f0be9194eb79b0ba75d7b29/intlogger.go#L370-L373On successful reattachment by a restarted host process, stdout/stderr syncing is lost, and any plugin log lines to Stdout/Stderr are lost.
Nomad works around this by having a dedicated log file for the plugin and not writing to the plugin Stderr in hashicorp/nomad#5598 .
Ideally, go-plugin
can makes handling host process restarting and re-attaching better. One possibility might be using fifo files such that plugin can always write to it with some buffer, but this may require clever use of non-blocking flags (to ensure plugin can proceed when fifo buffer is full).
I want a plugin interface to return a channel the host can listen on for asynchronous notifications (and vice versa). Is this supported?
I have a plugin that wants to use types defined in the host application. The plugin vendors the code from the host application. I'm getting an error at runtime:
reading body gob: name not registered for interface "github.com/me/plugin/vendor/github.com/me/hostapp/plugins.MyInterface"
I am calling gob.Register
for the types that implement the interface, but since I'm vendoring they get registered under a different type name and the gob decoding fails.
Are you aware of any projects that have used go-plugin
in this manner? I know I can either:
I just wanted to check to see if others have run into this came up with a different solution.
It seems to commit 0cf1b43, i.e. https://github.com/hashicorp/go-plugin/blob/master/examples/grpc/main.go#L67 causes a new sub/process for kv-go-grpc being created after each run, no matter the run is successful or not.
Also removing os.Exit(0) at L67 results in "[WARN] plugin: plugin failed to exit gracefully".
Quote:
Bidirectional communication. Because the plugin system supports complex arguments,
the host process can send it interface implementations and the plugin can call back
into the host process.
Any example of how can we approach this? Do we need separate (g)rpc connection for that? Is there any project on github that uses this kind of functionality? Also, is it somehow possible to use one plugin from another?
Hi @mitchellh ,
Actually I have doubts in rpc communication between main application and plugin , it'll be fine if you give some examples.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.