aphoh/go-substrate-gen

https://github.com/Aphoh/go-substrate-gen

Go Substrate Code Generator

A tool that generates boilerplate code for substrate-based chains (calls, storage access, events, etc) using go-substrate-rpc-client.

This uses the metadata Json RPC provided by substrate-based chains, which you can get by doing:

curl -L -X POST -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getMetadata"}' https://rpc.polkadot.io > polkadot-meta.json

Just replace rpc.polkadot.io with your own server.

Installation

Clone the repo, run go install ./... and make sure it's on your path.

Using go generate

Make a submodule in your go project. Place a metadata.json and the following mymodule.go file in it

//go:generate go-substrate-gen metadata.json "github.com/my/package/mymodule"

package mymodule

Then run go generate ./... and it'll generate code for each pallet in the mymodule directory.

Calling manually

go-substrate-gen meta.json "github.com/my/package/submodule/for/code" 

Getting Metadata

There is code included under json-gen to fetch a human-readable version of the json from a locally running substrate node in dev mode. View the readme for instructions.

Architecture and Design

Please read the documentation, which describes the received metadata's structure, and also gives an overview of the code's structure.

Calling code

Example for pallet_balances

// Transfer some liquid free balance to another account.
//
// `transfer` will set the `FreeBalance` of the sender and receiver.
// If the sender's account is below the existential deposit as a result
// of the transfer, the account will be reaped.
//
// The dispatch origin for this call must be `Signed` by the transactor.
//
// # <weight>
// - Dependent on arguments but not critical, given proper implementations for input config
//   types. See related functions below.
// - It contains a limited number of reads and writes internally and no complex
//   computation.
//
// Related functions:
//
//   - `ensure_can_withdraw` is always called internally but has a bounded complexity.
//   - Transferring balances to accounts that did not exist before will cause
//     `T::OnNewAccount::on_new_account` to be called.
//   - Removing enough funds from an account will trigger `T::DustRemoval::on_unbalanced`.
//   - `transfer_keep_alive` works the same way as `transfer`, but has an additional check
//     that the transfer will not kill the origin account.
// ---------------------------------
// - Origin account is already in memory, so no DB operations for them.
// # </weight>
func MakeTransferCall(dest0 *MultiAddress, value1 *types.UCompact) (types.Call, error) {...}
...

Storage code

// Make a storage key for Account
//  The Balances pallet example of storing the balance of an account.
//
//  # Example
//
//  
//   impl pallet_balances::Config for Runtime {
//     type AccountStore = StorageMapShim<Self::Account<Runtime>, frame_system::Provider<Runtime>, AccountId, Self::AccountData<Balance>>
//   }
//  
//
//  You can also store the balance of an account in the `System` pallet.
//
//  # Example
//
//  ```nocompile
//   impl pallet_balances::Config for Runtime {
//    type AccountStore = System
//   }
//  ```
//
//  But this comes with tradeoffs, storing account balances in the system pallet stores
//  `frame_system` data alongside the account data contrary to storing account balances in the
//  `Balances` pallet, which uses a `StorageMap` to store balances data only.
//  NOTE: This is only used in the case that this pallet is used to store balances.
func MakeAccountStorageKey(byteArray0 [32]byte) (types.StorageKey, error) {...}

func GetAccount(state *state.State, bhash types.Hash, byteArray0 [32]byte) (ret AccountData, err error) {...}
...

Types

// Generated pallet_balances_AccountData
type AccountData struct {
	// Field 0 with TypeId=6
	Free types.U128
	// Field 1 with TypeId=6
	Reserved types.U128
	// Field 2 with TypeId=6
	MiscFrozen types.U128
	// Field 3 with TypeId=6
	FeeFrozen types.U128
}

// Generated SpRuntimeMultiaddressMultiAddress with id=188
type MultiAddress struct {
	IsId              bool
	AsIdField0        [32]byte
	IsIndex           bool
	AsIndexField0     struct{}
	IsRaw             bool
	AsRawField0       []byte
	IsAddress32       bool
	AsAddress32Field0 [32]byte
	IsAddress20       bool
	AsAddress20Field0 [20]byte
}

func (ty MultiAddress) Encode(encoder scale.Encoder) (err error) {...}

func (ty *MultiAddress) Decode(decoder scale.Decoder) (err error) {...}

go-substrate-gen's Issues

Get around `WrapperKeepOpaque`

This type currently has garbage metadata. It should either be a []byte or a pointer to it's second field element. It has custom scale encoding, so that needs to be replicated here.

Can you show an example of decoding events?

Hey, this library seems to be a godsend if you actually want to use gsrpc. Is it possible you could show an example of how to decode all the events in a block. Can't seem to figure it out. Thanks!

*sorry if this is a dumb question

Cant get historical metadata of a chain.

When I'm trying use this on historical metadata of a chain, it does not work:

error generating calls for pallet System: no call type found. Expected a path like *_runtime::RuntimeCall

How to reproduce:

1. Get historical metadata (from block 10000000): curl -L -X POST -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getMetadata", "params": ["0x6c51377632106ddba379416f2ef5e24450e90cee8c78a70daa262b2ae6f27e93"]}' https://rpc.polkadot.io > polkadot-meta.json
2. Run substrate-gen

Note: I've tried this on two chains already.

Avoid embedding Variants

Currently, variants are constructed by embedding every variant into a single parent struct.

In the case of the runtime calls, this can result in some incredibly large structs, as the runtime call struct contains every single field of every single possible call that could be constructed.

As a future TODO, update this struct embedding to store booleans for what variety it is, and a pointer to each different struct for memory efficiency.

Encode variants like Rust for maximal space-efficiency

Currently, go-substrate-gen encodes variants pretty inefficiently by having an empty entry in every variant struct. This could be solved by having methods that return a struct containing each variant's information, and only storing a uint8 and the actual information of the variant in go. It would look something like the below (omitting the err != nil checks for brevity)

type VariantXYZ struct {
  index uint8
  data interface{}
}

func (ty VariantXYZ) Encode(encoder scale.Encoder) (err error) {
  err = encoder.PushByte(ty.index)
  switch ty.index {
  case 0:
    err = encode.Encode(ty.data.(X))
    return err
  case 1:
    err = encode.Encode(ty.data.(Y))
    return err
  case 2:
    err = encode.Encode(ty.data.(Z))
    return err
  }
  return fmt.Errorf("Unrecognized variant")
}

func (ty *VariantXYZ) Decode(decoder scale.Decoder) (err error) {
  variant, err := decoder.ReadOneByte()
  ty.variant = variant
  switch variant {
  case 0:
    var tmp X
    err = decoder.Decode(&tmp)
    ty.data = &tmp
    return
  case 1:
    var tmp Y
    err = decoder.Decode(&tmp)
    ty.data = &tmp
    return
  case 2:
    var tmp Z
    err = decoder.Decode(&tmp)
    ty.data = &tmp
    return
  }
}

func (ty *VariantXYZ) IsX() bool {
  return ty.index == 0
}

func (ty *VariantXYZ) IsY() bool {
  return ty.index == 1
}

func (ty *VariantXYZ) IsZ() bool {
  return ty.index == 2
}

// Optionally these could just use go semantics and return the zero struct if it fails for easier chaining
func (ty *VariantXYZ) X() (d X, err error) {
  if !ty.IsX() {
    err = fmt.Errorf("Variant is not X")
  }
  return ty.data.(X), nil
}

func (ty *VariantXYZ) Y() (d Y, err error) {
  if !ty.IsY() {
    err = fmt.Errorf("Variant is not Y")
  }
  return ty.data.(Y), nil
}

func (ty *VariantXYZ) X() (d Z, err error) {
  if !ty.IsZ() {
    err = fmt.Errorf("Variant is not Z")
  }
  return ty.data.(X), nil
}