centrifuge/go-substrate-rpc-client

Substrate RPC client for go aka GSRPC

Go Substrate RPC Client (GSRPC)

Substrate RPC client in Go. It provides APIs and types around Polkadot and any Substrate-based chain RPC calls. This client is modeled after polkadot-js/api.

State

This package is feature complete, but it is relatively new and might still contain bugs. We advise to use it with caution in production. It comes without any warranties, please refer to LICENCE for details.

Documentation & Usage Examples

Please refer to https://godoc.org/github.com/centrifuge/go-substrate-rpc-client/v4

Usage test examples of Dynamic Parsing of events & extrinsics

Registry docs

Contributing

1. Install dependencies by running make
2. Build the project with go build
3. Lint make lint (you can use make lint-fix to automatically fix issues)
4. Run make run-substrate-docker to run the Substrate docker container

Testing

We run our tests against a Substrate Docker image. You can choose to run the tests within a tests-dedicated Docker container or without a container.

1. make test-dockerized Run tests within a docker container of its own against the Substrate docker container.

2. make test Run the tests locally against the Substrate docker container. Note that it expects the Substrate docker container to be up and running to execute the whole test suite properly.

Visit https://polkadot.js.org/apps for inspection

Note: To use a custom Substrate endpoint, first set the environment variable before running the tests: export RPC_URL="http://example.com:9934"

Adding support for new RPC methods

After adding support for new methods, update the RPC mocks.

1. Install mockery
2. Run go generate ./...

go-substrate-rpc-client's Issues

Metadata version 10 support

Seems like Substrate metadata versions are moving fast 🙃

Implement Event and EventRecord types

unsupported metadata version 9

unsupported metadata version 9

How can support metadata version 9?

Call contract through RPC

It is not possible to call a contract through the RPC client however it is supported in the toolbox.

It is part of the contracts endpoint.

storage AccountNonce not found within module System

On Example_makeASimpleTransfer and wherever is applicable,
this types.CreateStorageKey(meta, "System", "AccountNonce", signature.TestKeyringPairAlice.PublicKey, nil)
shoud become
types.CreateStorageKey(meta, "System", "Account", signature.TestKeyringPairAlice.PublicKey, nil)

Decode failed of GetStorage result of System Events

Following is my golang test case:

func TestGetStorageDecode(t *testing.T) {
	api, err := gsrpc.NewSubstrateAPI("wss://rpc.polkadot.io")
	if err != nil {
		panic(err)
	}

	meta, err := api.RPC.State.GetMetadataLatest()
	if err != nil {
		panic(err)
	}

	key, err := types.CreateStorageKey(meta, "System", "Events", nil, nil)
	if err != nil {
		panic(err)
	}

	blockHash, err := api.RPC.Chain.GetBlockHash(1311355)
	if err != nil {
		panic(err)
	}

	var data types.StorageDataRaw
	ok, err := api.RPC.State.GetStorage(key, &data, blockHash)
	if err != nil {
		panic(err)
	}

	if ok {
		events := types.EventRecords{}
		err = types.EventRecordsRaw(data).DecodeEventRecords(meta, &events)
		if err != nil {
			panic(err)
		}

		for _, e := range events.Balances_Transfer {
			fmt.Printf("Balances:Transfer:: (phase=%#v)\n", e.Phase)
			fmt.Printf("\t%v, %v, %v\n", e.From, e.To, e.Value)
		}
	}
}

But DecodeEventRecords returns error:

panic: unable to find field Scheduler_Scheduled for event #1 with EventID [1 0]

Then I use polkadot-js to implement the same purpose as following code:

/* eslint-disable header/header */
/* eslint-disable @typescript-eslint/require-await */
/* eslint-disable @typescript-eslint/unbound-method */

const { ApiPromise, WsProvider } = require('@polkadot/api');

async function main() {
  const wsProvider = new WsProvider('wss://rpc.polkadot.io');
  const api = await ApiPromise.create({ provider: wsProvider });

  const blockHash = await api.rpc.chain.getBlockHash(1311355)
  const events = await api.query.system.events.at(blockHash);
  events.forEach((record) => {
    const { event, phase } = record;
    const types = event.typeDef;

    if (event.section == 'balances' && event.method == 'Transfer') {
      console.log(`${event.section}:${event.method}:: (phase=${phase.toString()})`);
      console.log(`\t${event.meta.documentation.toString()}`);

      event.data.forEach((data, index) => {
        console.log(`\t\t${types[index].type}: ${data.toString()}`);
      });
    }
  });
}

main().catch((error) => {
  console.error(error);
  process.exit(-1);
});

It decodes correct:

balances:Transfer:: (phase={"ApplyExtrinsic":3})
	[ Transfer succeeded. [from, to, value]]
		AccountId: 15xiKzhdsFCxLwhgSxDrw2tYJZ7XiYGNkSdHP96bNmeW9jhs
		AccountId: 15kUt2i86LHRWCkE3D9Bg1HZAoc2smhn1fwPzDERTb1BXAkX
		Balance: 31669844000000

Golang version of DecodeEventRecords decodes event number right, but failed of second event decoding.

Calls and subscriptions should accept a context

Calls should accept a context so they can be cancelled and can receive a timeout.

Subscriptions need a bit more investigation – a context could be used to cancel or timeout the subscription, but the current subscription logic already exposes an Unsubscribe method that can be used for both purposes.

RUN Example_transactionWithEvents func ERROR

Run "Example_transactionWithEvents" func in main_test.go, here "meta, err := api.RPC.State.GetMetadataLatest()" ret error is "type *types.StorageMetadata does not support Decodeable interface and could not be decoded field by field, error: type *types.StorageFunctionMetadataV5 does not support Decodeable interface and could not be decoded field by field, error: type *types.DoubleMapTypeV5 does not support Decodeable interface and could not be decoded field by field, error: received unexpected storage hasher type 5 [recovered]
panic: type *types.StorageMetadata does not support Decodeable interface and could not be decoded field by field, error: type *types.StorageFunctionMetadataV5 does not support Decodeable interface and could not be decoded field by field, error: type *types.DoubleMapTypeV5 does not support Decodeable interface and could not be decoded field by field, error: received unexpected storage hasher type 5
"

Testing rpc for GSRPC

Metadata version 11 support

Hello @philipstanislaus any plans on this?
I can give it a try, not sure if I can have it resolved on reasonable time thoough.

Problem with metadatav11

When I updated to last commit to include metadatav11 I am getting this error:
*types.StorageMetadataV10 does not support Decodeable interface and could not be decoded field by field, received unexpected storage hasher type 6.

Public key (hex): 0x

go-substrate-rpc-client/signature/signature.go

Line 41 in 6f0547f

var rePubKey = regexp.MustCompile(`Public key \(hex\): 0x([a-f0-9]*)\n`)

after colon should be one or more space not exact one space

Implement transaction signing for extrinsic call using sr25519 signature scheme

Sr25519 is based on the same underlying Curve25519 as its EdDSA counterpart, Ed25519. However, it uses Schnorr signatures instead of the EdDSA scheme. This task is to implement signing in the rpc client with Sr25519.

Currently transaction signing is implemented by executing subkey command provided by substrate at https://github.com/centrifuge/go-substrate/blob/master/author.go#L157. The new implementation should go in https://github.com/centrifuge/go-substrate/tree/master/signature package and be called by author.go in for extrinsic calls.

Relevant links

• https://wiki.polkadot.network/en/latest/polkadot/learn/cryptography/
• https://polkadot.js.org/api/METHODS_RPC.html

Expose "scheme" to sign/verify signature code

As suggested here: #99 (comment)

We should expose the scheme option so all subkey cryptographic schemes are available: Ed25519, Sr25519, Ecdsa

How to parse extrinsics params of Call and associated with metadata

Chain.GetBlock not work in 6f0547f2818d5fbc659d4a9be363bb4e02ab223d , but 461cf42e77c6d3b65fefa801a1aeddf04f1174cf

Chain.GetBlock return error: type *types.ExtrinsicSignatureV4 does not support Decodeable interface and could not be decoded fiel
final error is : errors.New("Not supported: l>4 encountered when decoding a compact-encoded uint")

NewSubstrateAPI error

error message:
type *types.StorageMetadataV10 does not support Decodeable interface and could not be decoded field by field, error: type *types.StorageFunctionMetadataV10 does not support Decodeable interface and could not be decoded field by field, error: type *types.MapTypeV10 does not support Decodeable interface and could not be decoded field by field, error: received unexpected storage hasher type 6 [recovered]
panic: type *types.StorageMetadataV10 does not support Decodeable interface and could not be decoded field by field, error: type *types.StorageFunctionMetadataV10 does not support Decodeable interface and could not be decoded field by field, error: type *types.MapTypeV10 does not support Decodeable interface and could not be decoded field by field, error: received unexpected storage hasher type 6

url: ws://47.108.199.133:9944

Implement storage query RPCs

This is required for querying accounts state, balances, anchors etc.

Fails to decode empty values

Currently, scale will attempt to decode an empty value and fail with expected more bytes, but could not decode any more. This is related to the logic in #64.

enc := types.MustHexDecodeString("0x00")
var dec types.Call
err := types.DecodeFromBytes(enc, &dec)
if err != nil {
	panic(err)
}

Not 100% sure what a good way to resolve this is, happy to put in a fix if I can get some suggestions.

Invalid Transaction

hi, when run the makeASimpleTransfer, i got an error "makeASimpleTransfer", what's wrong with it.

Extrinsic with `utility.batch` is decoded incorrectly

This issue was identified while solving the issue at #118 .

Bug:

User is not able to correctly decode the utility.batch extrinsic.

Steps to Reproduce

For simplicity, I've created a Westend transaction @ https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fwestend-rpc.polkadot.io#/explorer/query/0xf423599fa7cabb680b19733b5ea0e9676ead737eb38c2bd5ceffcb50a0a950d4

This transaction has a utility.batch extrinsic. The batch contains:

1. a system.remark call.
2. a balance.transfer call

Correctly decoded using javascript

const wsProvider = new WsProvider('wss://westend-rpc.polkadot.io');
const api = await ApiPromise.create({ provider: wsProvider });

const block_hash = '0xf423599fa7cabb680b19733b5ea0e9676ead737eb38c2bd5ceffcb50a0a950d4';

const block = await api.rpc.chain.getBlock(block_hash);

block.block.extrinsics.forEach(ext => {
  console.log("Call Index: ", ext.callIndex[0], ' ', ext.callIndex[1])
});

Outputs:

Call Index: 2 0
Call Index: 16 0

Incorrectly decoded using Go

api, err := gsrpc.NewSubstrateAPI("wss://westend-rpc.polkadot.io")
if err != nil {
  panic(err)
}

var blockHashString = "0xf423599fa7cabb680b19733b5ea0e9676ead737eb38c2bd5ceffcb50a0a950d4"
blockHash, err := types.NewHashFromHexString(blockHashString)
if(err != nil) {
  panic(err)
}

// Get the block
block, err := api.RPC.Chain.GetBlock(blockHash)
if(err != nil){
  panic(err)
}

meta, err := api.RPC.State.GetMetadata(blockHash)
if err != nil {
  panic(err)
}

for i, ext := range block.Block.Extrinsics {
  fmt.Println("EXT's CALL INDEX ", ext.Method.CallIndex)
}

Outputs:

EXT's CALL INDEX {2 0}
EXT's CALL INDEX {61 229}  (Not printed note: this is where the problem is}

As you can see the second extrinsic's call index is {61 229} where it should be {16 0}.

Ask

Both call indexes should be the same. Can you tell us why is this the case?

Incompatibility with Other Chain Configs

Extrinsics are failing when submitted to the substrate-node-template chain.

Verification Error: Execution(ApiError("Could not convert parameter `tx` between node and runtime: No such variant in enum MultiSignature"))

GSRPC code can be found here: https://gist.github.com/ansermino/07df0dd8e35c522ec23c1d922860c786

This seems to have something to do with the type configuration for the chain (see https://polkadot.js.org/api/start/types.extend.html#impact-on-extrinsics)

Handle subtrate extrinsic events

Currently the client doesn't handle events returned from extrinsics.

Add missing event types to EventRecords

It seems that multiple event types are currently missing. When a block contains one of those missing event types, decoding it throws an error like this:

panic: unable to find field Elections_NewTerm for event #0 with EventID [8 0]

cannot find module providing package

build command-line-arguments: cannot load github.com/ethereum/go-ethereum/common: cannot find module providing package github.com/ethereum/go-ethereum/common

Replace subkey with go-subkey port

Have you considered moving to go-subkey port so that installing subkey is not needed? I see this port is made by @vedhavyas

I have replaced it on this fork and everything seems to be working as intended.

unable to find field Multisig_MultisigApproval for event #1 with EventID [31 1]

Hello
It error is falling when i decode event records on some blocks. Example, 4940400 block for kusama

hash, err := api.RPC.Chain.GetBlockHash(number)
if err != nil {
	return err
}

meta, err := api.RPC.State.GetMetadata(hash)
if err != nil {
	return err
}

key, err := types.CreateStorageKey(meta, "System", "Events", nil, nil)
if err != nil {
	return err
}

raw, err := api.RPC.State.GetStorageRaw(key, hash)
if err != nil {
	return err
}

events := types.EventRecords{}
err = types.EventRecordsRaw(*raw).DecodeEventRecords(meta, &events)
if err != nil {
	return err   ///<<- unable to find field Multisig_MultisigApproval for event #1 with EventID [31 1]
}

Upgrade GSRPC

To latest version

types.StorageMetadataV10 does not support Decodeable interface and could not be decoded field

when i code this:
api, err := gsrpc.NewSubstrateAPI(mynoderul)
if err != nil {
panic(err)
}
but occured this bug:
type *types.StorageMetadataV10 does not support Decodeable interface and could not be decoded field .
How to fix this error?

Issue when fetching AccountInfo

I used code from Example (TransactionWithEvents), but I have a problem with fetching AccountInfo. It fails with this error:

type *types.AccountInfo does not support Decodeable interface and could not be decoded field by field, error: type *struct { Free types.U128; Reserved types.U128; MiscFrozen types.U128; FreeFrozen types.U128 } does not support Decodeable interface and could not be decoded field by field, error: Cannot read the required number of bytes 16, only 13 available

I am running polkadot --dev where //Alice is trying to make transaction

Outdated examples at pkg.go.dev

I just tried the examples (simpleTransfer example in particular) at https://pkg.go.dev/github.com/centrifuge/go-substrate-rpc-client#pkg-examples and ran into a bunch of compatibility issue.

Eventually, I got it working using the main_test.go file and the answer at #99 .

Can someone update the docs on the pkg.go.dev site? I'm new to go and not sure how to do it. If you can point me in the right direction, I can update the examples myself.

Substrate 2.0.0

Hi guys,

The Substrate v2.0.0 has been released.
Would it be possible to update the go-substrate-rpc-client to work with this substrate release ?
I prepared a pull request for that purpose : #100

I hope it's ok for you :)
Thanks.

panic: unsupported metadata version 12

Steps to reproduce

1. import the package.
2. Connect to a network.

	api, err := gsrpc.NewSubstrateAPI("wss://kusama-rpc.polkadot.io/")
	if err != nil {
		panic(err)
	}

3. Check Metadata

  meta, err := api.RPC.State.GetMetadataLatest()
  if err != nil {
    panic(err)
  }

Output

panic: unsupported metadata version 12

Typo inside documentation

There is a small typo inside one of the examples (MakeASimpleTransfer) in documentation.

There is a check for error existence even though there is no reason for this check.

// Create the extrinsic
ext := types.NewExtrinsic(c)
if err != nil {
	panic(err)
}

Failing tests

I am not able to successfully run tests (I am running dockerized tests with make test-dockerized)

This is a relevant part of logs

gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client [build failed]
gsrpc-test_1  | ok      github.com/centrifuge/go-substrate-rpc-client/hash      0.030s  coverage: 60.0% of statements
gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client/rpc/author [build failed]
gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client/rpc/chain [build failed]
gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client/rpc/state [build failed]
gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client/rpc/system [build failed]
gsrpc-test_1  | ok      github.com/centrifuge/go-substrate-rpc-client/rpcmocksrv        0.030s  coverage: 90.9% of statements
gsrpc-test_1  | ok      github.com/centrifuge/go-substrate-rpc-client/scale     0.030s  coverage: 72.1% of statements
gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client/signature [build failed]
gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client/teste2e [build failed]
gsrpc-test_1  | FAIL    github.com/centrifuge/go-substrate-rpc-client/types [build failed]
gsrpc-test_1  | ok      github.com/centrifuge/go-substrate-rpc-client/xxhash    0.025s  coverage: 94.7% of statements

Problem is that go can't build this code, this is output from go build ./...

vendor/github.com/vedhavyas/go-subkey/sr25519/derive_junction.go:24:19: secret.DeriveKey undefined (type *schnorrkel.SecretKey has no field or method DeriveKey)
vendor/github.com/vedhavyas/go-subkey/sr25519/derive_junction.go:28:2: not enough arguments to return
vendor/github.com/vedhavyas/go-subkey/sr25519/key.go:132:18: ms.Encode undefined (type *schnorrkel.MiniSecretKey has no field or method Encode)
vendor/github.com/vedhavyas/go-subkey/sr25519/key.go:139:13: undefined: schnorrkel.NewSecretKey
vendor/github.com/vedhavyas/go-subkey/sr25519/key.go:142:14: undefined: schnorrkel.MiniSecretFromMnemonic

blake2b

this code :
func Test_SS58(t *testing.T) {
//seed := 0xdba9807c8606e46341dcac32dbf1c9ec25dae6a3bc386d83d8bde4d6ab72408e
publicKey := "aa29cb6398bc70bfedbd4f7af69dd4c24b2d0b07ebb962699954c0e06d99112e"
addr := "5FupQuyf8STcPCf3DfUPxhNWunxi8R4PiydFwUywdZiRE3cz"

pub, err := hex.DecodeString(publicKey)
if err != nil {
	panic(err)
}

checkSum := blake2b.Sum256(pub)
d := base58.Decode(addr)

fmt.Printf("d:%0x \n checksum:%0x", d, checkSum)

}

and the result is :
d:2aaa29cb6398bc70bfedbd4f7af69dd4c24b2d0b07ebb962699954c0e06d99112eaa97
checksum:c61d902e300e32c48fee79518f750976e8570df9faece77dc80f5271c2625e16

the decode of addr checksum is "aa97",not euqal the checksum.
what's wrong?

Decode Blocking Forever

I have the following defined in my pallet:

pub struct ProposalVotes<AccountId, Hash> {
    votes_for: Vec<AccountId>,
    votes_against: Vec<AccountId>,
    hash: Hash,
}

And a corresponding Go declaration:

type VoteState struct {
    VotesFor []types.AccountID
    VotesAgainst []types.AccountID
    Hash types.Hash
}

Calling Decode (inside DecodeFromBytes()) seems to never return. This only seems to be the case when the struct is non-empty.

Reproducible:

// From RPC state query 
enc := hexutil.MustDecode("0x16078eaf04151687736326c9fea17e25fc5287613693c912909cb226aa4794f26a4810000000")

var dec voteState
fmt.Println("Starting to decode")
err := types.DecodeFromBytes(enc, &dec)
fmt.Println("Finished decoding")
if err != nil {
	panic(err)
}

// Expected: VoteStatus {
//   VotesFor: []types.AccountID{types.NewAccountId(signature.TestKeyringPairAlice.PublicKey)},
//   VotesAgainst: []types.AccountID{},
//   Hash: prop.hash,
// } 

Check presence of all types in metadata at initialization, fail early

Currently, missing types are only detected when decoding. For events, this could mean that after several days, decoding might fail (e. g. a new term of the council).

Missing MetatData.AsMetadataV12.Modules

Awesome library 👍👍👍👍

But we are seeing an issue when building on v2.0.0 with same variable defined on two places.
It looks to be resolved here #108
but when we try upgrade to v2.0.0-rc6.2 we are missing the Metadata.AsMetadataV12.Modules

Please advise.
Thanks

Query historical system events on archive node

Hello,

I was wondering what would be the easies method to query historical system events from archive node. I understand that I can subscribe to get system events, but I was wondering, since I run archive node how could I get system events at specific heigh. Any help would be much appreciated.

unsupported metadata version 11

Remove hardcoded account info in author.go

SomeOffline Event parsing error

t=2020-07-10T14:18:55+0000` lvl=eror msg="Failed to process events in block" chain=substrate block=1516955 err="unable to decode 
field 1 event #0 with EventID [9 2], field ImOnline_SomeOffline: type *struct { ValidatorID types.AccountID; FullIdentification types.Exposure } 
does not support Decodeable interface and could not be decoded field by field, error: type *types.Exposure does not support 
Decodeable interface and could not be decoded field by field, error: Not supported: l>4 encountered when decoding a compact-encoded uint"

Add correct event parser

How to get extrinsic data from the block?

I'm trying to read details in the extrinsic call, but I'm not sure how to go about it.

So far, I have

• received the blocked based on the hash string

  var blockHashString = "0xd7a1c8e7fb89dcf72895a03d0a64b0aeb36e11aa7ab3e9bf036975b2d3b4a2c7";
  blockHash, err := types.NewHashFromHexString(blockHashString);
  if(err != nil) {
    panic(err)
  }

  // Get the block
  block, err := api.RPC.Chain.GetBlock(blockHash)
  if(err != nil){
    panic(err)
  }

• looping through extrinsics

  // Go through each Extrinsics
  for i, ext := range block.Block.Extrinsics {
    fmt.Println("EXT #  ", i)
    fmt.Println("EXT Details  ", ext)
  ...
  ...
  }

• located the Args as Extrinsics.Method.Args

  // inside the for loop described above
    fmt.Println("EXT Args Details  ", ext.Method.Args)

But the value of Args is an array of bytes ([] byte) (example below)

[220 188 40 21 96 60 227 172 245  ... 116 97 110 100 97 114 100 32 84 105 109 101 41]

I'm not sure how to get the full argument details out of this. Do I need to decode it further? It looks to be decoded already.

What should be next step? Looking forward to a nudge in the right direction.

How to extend EventRecords to match my custom substrate node

I have some custom events and types

Failed to sign with Subkey using ChainBridge

Subkey signing fails when running the chainbridge relayer to sign extrinsics, potentially because we are running a Substrate 2.0 chain. It looks like it fails at signing the extrinsic which goes into this library.

• Any idea how to debug or help if I receive EROR[10-21|15:50:49] Failed to execute extrinsic chain=edgeware err="failed to sign with subkey: exit status 1"?

Signature package could use JSON instead of regex

I believe that the version of subkey presently used supports JSON formatting which would likely be more reliable than regex and easier to upgrade.

Here's an example:

d@P51:~$ subkey --output=json inspect //Alice
{
  "accountId": "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d",
  "publicKey": "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d",
  "secretKeyUri": "//Alice",
  "secretSeed": "0xe5be9a5092b81bca64be81d212e7f2f9eba183bb7a90954f7b76361f6edb5c0a",
  "ss58Address": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
}

Verification Error: Execution: Could not convert parameter `tx` between node and runtime: No such variant in enum MultiSignature when submitting to Polkadot with v2.0.0-rc6-0

I'm using the example code from the godoc here, but am unable to actually submit the signed transaction.

It fails with

Verification Error: Execution: Could not convert parameter `tx` between node and runtime: No such variant in enum MultiSignature

which should already be solved according to #65 and #68.

Do we need to update anything from the example code for it to work?

events decode error

I get a storage raw 0x0c0000000000000080e36a09000000000200000001000000000000000000000000000200000002000000000000ca9a3b00000000020000 with key System Events. And this 55 bytes hex could not be decode with error unable to decode field 2 event #1 with EventID [1 0], field Indices_IndexAssigned: unexpected EOF.

Update to use Go Modules