Unrecognized remote plugin message

[immutability-io]

Unrecognized remote plugin message

[0x234]

vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.2.0
Cluster Name    vault-cluster-41b32c41
Cluster ID      3a9a120f-544f-a6d9-8af3-93bf94eafbff
HA Enabled      true
HA Cluster      https://10.20.2.228:8201
HA Mode         active

[0x234]

./vault-ethereum
This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

[0x234]

This is the condition that raises this error: https://github.com/hashicorp/vault/blob/master/vendor/github.com/hashicorp/go-plugin/client.go#L684

[0x234]

Any thoughts @cypherhat?

[cypherhat]

So, this message:

2019-08-13T11:17:26.508-0400 [WARN]  received Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol. attempting as db plugin, attempting as auth/secret plugin

When you write a plugin to the catalog, the default behavior from vault is to treat the plugin as a "secret" type plugin. So if you:

vault write sys/plugins/catalog/eth-plugin ...

Vault will treat the plugin as a secret plugin. However, if you do that in vault 1.2.1, it chirps. so the fix is to:

vault write sys/plugins/catalog/secret/eth-plugin ...

[0x234]

Thanks @cypherhat. That still doesn't let me enable the plugin though:

/app # vault write sys/plugins/catalog/secret/ethereum-plugin \
>       sha_256=eb5f0057893e3444b9d2022563a2a241f96593e310de1890d3626b147ab4fe4e \
>       command="vault-ethereum --ca-cert=/app/root.crt --client-cert=/app/vault.crt --client-key=/app/vault.key"
Success! Data written to: sys/plugins/catalog/secret/ethereum-plugin
/app # vault secrets enable -path=ethereum-plugin -description="Immutability's Ethereum Wallet" -plugin-name=ethereum-plugin plugin
Error enabling: Error making API request.

URL: POST https://127.0.0.1:8200/v1/sys/mounts/ethereum-plugin
Code: 400. Errors:

* Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

[cypherhat]

Ok, let me test with Vault 1.2.1. I may have to rebuild.

[cypherhat]

My test was on MacOS Mojave 10.14.6. I downloaded this release.

My vault config for this quick test was:

$ cat ~/etc/vault.d/vault.hcl 
"default_lease_ttl" = "24h"
"disable_mlock" = "true"
"max_lease_ttl" = "24h"

"backend" "file" {
  "path" = "/Users/cypherhat/etc/vault.d/data"
}

"api_addr" = "https://localhost:8200"
"ui" = "true"
"listener" "tcp" {
  "address" = "localhost:8200"

  "tls_cert_file" = "/Users/cypherhat/etc/vault.d/vault.crt"
  "tls_client_ca_file" = "/Users/cypherhat/etc/vault.d/root.crt"
  "tls_key_file" = "/Users/cypherhat/etc/vault.d/vault.key"
}

"plugin_directory" = "/Users/cypherhat/etc/vault.d/vault_plugins"

I installed the plugin as follows:

$ vault secrets disable vault-ethereum
$ vault delete sys/plugins/catalog/secret/vault-ethereum
$ mv vault-ethereum $HOME/etc/vault.d/vault_plugins/vault-ethereum
$ export SHA256=$(shasum -a 256 "$HOME/etc/vault.d/vault_plugins/vault-ethereum" | cut -d' ' -f1)
$ vault write sys/plugins/catalog/secret/vault-ethereum \
      sha_256="${SHA256}" \
      command="vault-ethereum --ca-cert=$HOME/etc/vault.d/root.crt --client-cert=$HOME/etc/vault.d/vault.crt --client-key=$HOME/etc/vault.d/vault.key"
$ vault secrets enable -path=vault-ethereum -plugin-name=vault-ethereum plugin

Watching the Vault log messages... I saw a few warnings, but the plugin loaded.

2019-08-14T08:02:11.961-0400 [WARN]  secrets.vault-ethereum.vault-ethereum_cab3d845.vault-ethereum: error closing client during Kill: metadata=true err="rpc error: code = Canceled desc = grpc: the client connection is closing"
2019-08-14T08:02:11.961-0400 [WARN]  secrets.vault-ethereum.vault-ethereum_cab3d845.vault-ethereum: plugin failed to exit gracefully: metadata=true
2019-08-14T08:02:11.965-0400 [INFO]  core: successful mount: namespace= path=vault-ethereum/ type=vault-ethereum

Then I interacted with the plugin as follows:

$ vault write vault-ethereum/config api_key=$MY_COINMARKETCAP_API_KEY
Key                Value
---                -----
api_key            REDACTED
bound_cidr_list    <nil>
chain_id           4
rpc_url            https://rinkeby.infura.io

$ vault write vault-ethereum/convert unit_from=usd amount=1400 unit_to=eth
Key            Value
---            -----
amount_from    1400
amount_to      6.7750703015583604
unit_from      usd
unit_to        ether

[0x234]

Ah, the Dockerfile clones github.com/immutability-io/vault-ethereum which has 4 commits since that release. Perhaps one of those has introduced a regression. I'll try a build from that specific release.

[0x234]

Looking at the commits there doesn't seem to be anything that would stop this working. Hey @zambien - were you able to load & enable the plugin from that Dockerfile?

[0x234]

I'm unable to exec the binary built from goreleaser. Looks like it needs glibc:

/app # ldd vault-ethereum_SNAPSHOT-b8ba0d6
	/lib64/ld-linux-x86-64.so.2 (0x7f2f85305000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f2f85305000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f2f85305000)

/app # file vault-ethereum_SNAPSHOT-b8ba0d6
vault-ethereum_SNAPSHOT-b8ba0d6: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, Go BuildID=6p31EtFuG9KZrQ0QADHT/1O1GVQC3r6Y9Z3R3Zzfs/oIfBGOtGZSSGXsQdOgOf/GL_bbzooVwwzmhVkGYWe, BuildID[sha1]=aa0400b652073b52af8303c15fa6ccc488421f81, stripped

[0x234]

CGO_ENABLED=0 kills the build with:

   ⨯ release failed after 145.63s error=failed to build for linux_amd64: go build github.com/ethereum/go-ethereum/crypto/secp256k1: build constraints exclude all Go files in /go/pkg/mod/github.com/ethereum/[email protected]/crypto/secp256k1
# github.com/ethereum/go-ethereum/rpc
/go/pkg/mod/github.com/ethereum/[email protected]/rpc/endpoints.go:96:19: undefined: ipcListen
/go/pkg/mod/github.com/ethereum/[email protected]/rpc/ipc.go:50:10: undefined: newIPCConnection

[cypherhat]

Can't use CGO_ENABLED=0 as you stated due to the secp256k1 dependency... as you mentioned.

[0x234]

I have the plugin enabled and I observed the same behaviour as you @cypherhat with the RPC connection closing. However after three attempts:

Success! Enabled the ethereum-plugin secrets engine at: ethereum-plugin/

I built vault and vault-ethereum from source with libc instead of musl. The trade off is a massive build artefact and a much larger attack surface due to the bloat pulled in. If you'd like I can provide this Dockerfile? I suspect the best path here would be to get Alpine & musl builds operational though. Thanks for your help.

[cypherhat]

@JBunCE Please provide the Dockerfile. Thanks!