kilnfi/cosmos-validator-watcher

Real-time Cosmos-based chains monitoring

Cosmos Validator Watcher

Cosmos Validator Watcher is a Prometheus exporter to help you monitor missed blocks on any cosmos-based blockchains in real-time.

Features:

• Track when your validator missed a block (with solo option)
• Check how many validators missed the signatures for each block
• Track the current active set and check if your validator is bonded or jailed
• Track the staked amount as well as the min seat price
• Track pending proposals and check if your validator has voted (including proposal end time)
• Expose upgrade plan to know when the next upgrade will happen (including pending proposals)
• Trigger webhook when an upgrade happens

✨ Usage

Example for cosmoshub using 2 public RPC nodes and tracking 4 validators (with custom aliases).

Via compiled binary

Compiled binary can be found on the Releases page.

cosmos-validator-watcher \
  --node https://cosmos-rpc.publicnode.com:443 \
  --node https://cosmos-rpc.polkachu.com:443 \
  --validator 3DC4DD610817606AD4A8F9D762A068A81E8741E2:kiln \
  --validator 25445D0EB353E9050AB11EC6197D5DCB611986DB:allnodes \
  --validator 9DF8E338C85E879BC84B0AAA28A08B431BD5B548:9df8e338 \
  --validator ABC1239871ABDEBCDE761D718978169BCD019739:random-name

Via Docker

Latest Docker image can be found on the Packages page.

docker run --rm ghcr.io/kilnfi/cosmos-validator-watcher:latest \
  --node https://cosmos-rpc.publicnode.com:443 \
  --node https://cosmos-rpc.polkachu.com:443 \
  --validator 3DC4DD610817606AD4A8F9D762A068A81E8741E2:kiln \
  --validator 25445D0EB353E9050AB11EC6197D5DCB611986DB:allnodes \
  --validator 9DF8E338C85E879BC84B0AAA28A08B431BD5B548:9df8e338 \
  --validator ABC1239871ABDEBCDE761D718978169BCD019739:random-name

Available options

cosmos-validator-watcher --help

NAME:
   cosmos-validator-watcher - Real-time Cosmos-based chains monitoring tool

USAGE:
   cosmos-validator-watcher [global options] command [command options] [arguments...]

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --chain-id value                         to ensure all nodes matches the specific network (dismiss to auto-detected)
   --http-addr value                        http server address (default: ":8080")
   --log-level value                        log level (debug, info, warn, error) (default: "info")
   --namespace value                        namespace for Prometheus metrics (default: "cosmos_validator_watcher")
   --no-color                               disable colored output (default: false)
   --node value [ --node value ]            rpc node endpoint to connect to (specify multiple for high availability) (default: "http://localhost:26657")
   --no-gov                                 disable calls to gov module (useful for consumer chains) (default: false)
   --no-staking                             disable calls to staking module (useful for consumer chains) (default: false)
   --no-commission                          disable calls to get validator commission (useful for chains without distribution module) (default: false)
   --no-upgrade                             disable calls to upgrade module (for chains created without the upgrade module) (default: false)
   --denom value                            denom used in metrics label (eg. atom or uatom)
   --denom-exponent value                   denom exponent (eg. 6 for atom, 1 for uatom) (default: 0)
   --start-timeout value                    timeout to wait on startup for one node to be ready (default: 10s)
   --stop-timeout value                     timeout to wait on stop (default: 10s)
   --validator value [ --validator value ]  validator address(es) to track (use :my-label to add a custom label in metrics & ouput)
   --webhook-url value                      endpoint where to send upgrade webhooks (experimental)
   --x-gov value                            version of the gov module to use (v1|v1beta1) (default: "v1")
   --help, -h                               show help
   --version, -v                            print the version

❇️ Endpoints

• /metrics exposed Prometheus metrics (see next section)
• /ready responds OK when at least one of the nodes is synced (ie. .SyncInfo.catching_up is false)
• /live responds OK as soon as server is up & running correctly

📊 Prometheus metrics

All metrics are by default prefixed by cosmos_validator_watcher but this can be changed through options.

Metrics (without prefix)	Description
active_set	Number of validators in the active set
block_height	Latest known block height (all nodes mixed up)
commission	Earned validator commission
is_bonded	Set to 1 if the validator is bonded
is_jailed	Set to 1 if the validator is jailed
missed_blocks	Number of missed blocks per validator (for a bonded validator)
consecutive_missed_blocks	Number of consecutive missed blocks per validator (for a bonded validator)
node_block_height	Latest fetched block height for each node
node_synced	Set to 1 is the node is synced (ie. not catching-up)
proposal_end_time	Timestamp of the voting end time of a proposal
proposed_blocks	Number of proposed blocks per validator (for a bonded validator)
rank	Rank of the validator
seat_price	Min seat price to be in the active set (ie. bonded tokens of the latest validator)
skipped_blocks	Number of blocks skipped (ie. not tracked) since start
solo_missed_blocks	Number of missed blocks per validator, unless the block is missed by many other validators
tokens	Number of staked tokens per validator
tracked_blocks	Number of blocks tracked since start
transactions	Number of transactions since start
validated_blocks	Number of validated blocks per validator (for a bonded validator)
vote	Set to 1 if the validator has voted on a proposal
upgrade_plan	Block height of the upcoming upgrade (hard fork)

❓FAQ

Which blockchains are compatible?

Any blockchains based on the cosmos-sdk should work:

• cosmoshub
• celestia
• cronos
• dydx
• evmos
• injective
• kava
• osmosis
• persistence
• dymension
• zetachain
• ...

This app is using the CometBFT library (successor of Tendermint) as well as the x/staking module from the Cosmos-SDK.

How to get your validator pubkey address?

Option 1: use tendermint show-validator to get the pubkey and debug pubkey to convert to hex format.

CLI_NAME=gaiad
ADDRESS="$($CLI_NAME debug pubkey "$($CLI_NAME tendermint show-validator)" 2>&1 | grep "Address")"
ADDRESS="${ADDRESS##* 0x}"
ADDRESS="${ADDRESS##* }"
echo "${ADDRESS^^}"

(replace gaiad by the binary name or the desired chain, eg. evmosd, strided, injectived, …).

Option 2: use the cosmos-validator-watcher debug consensus-key sub command:

cosmos-validator-watcher debug validator \
  --node https://cosmos-rpc.publicnode.com:443 \
  cosmosvaloper1uxlf7mvr8nep3gm7udf2u9remms2jyjqvwdul2

Notes:

• the --node flag must be placed before the validator address)
• this doesns't work for consumer chains (neutron, stride) since they don't rely on the staking module

📃 License

MIT License.

cosmos-validator-watcher's Issues

Multiple Instances

I believe I am correct in that an instance of the cosmos-validator-watcher is required per monitored chain and that the exposed http port would need to be unique so that prometheus can hit each instance?

extract upgrade in MsgExecLegacyContent

Currently, only upgrade embedded in SoftwareUpgradeProposal and MsgSoftwareUpgrade are detected.

cosmos-validator-watcher/pkg/watcher/upgrade.go

Lines 218 to 234 in d6945ee

	switch content.TypeUrl {
	case "/cosmos.upgrade.v1beta1.SoftwareUpgradeProposal":
	var upgrade types.SoftwareUpgradeProposal
	err := cdc.Unmarshal(content.Value, &upgrade)
	if err != nil {
	return nil, fmt.Errorf("failed to unmarshal software upgrade proposal: %w", err)
	}
	return &upgrade.Plan, nil
	case "/cosmos.upgrade.v1beta1.MsgSoftwareUpgrade":
	var upgrade types.MsgSoftwareUpgrade
	err := cdc.Unmarshal(content.Value, &upgrade)
	if err != nil {
	return nil, fmt.Errorf("failed to unmarshal software upgrade proposal: %w", err)
	}
	return &upgrade.Plan, nil
	}

But it can also happen via a MsgExecLegacyContent.

Examples:

• https://www.mintscan.io/cosmos/proposals/914
• https://www.mintscan.io/evmos/proposals/303

Check if block missed was alone

ref: https://github.com/gaia/cosmosvalmon/blob/master/cosmosvalmon.py#L29

# When a lot of validators miss a block, it is not your fault. if you alone or only a handful missed,
# you should look into your setup. Set alertIfLessThan to the max number of validators missing a block you'd tolerate
# before you'd want to be alerted. If you set it to 0, you will never get an alert.

Expose total staked amount

• stake by validators
• min_stake for active set

fix panic

panic: runtime error: invalid memory address or nil pointer dereference	
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xf15db0]	
	
goroutine 45 [running]:	
github.com/kilnfi/cosmos-validator-watcher/pkg/rpc.(*Node).syncBlocks(0xc0005c0e10, {0x14403f0, 0xc0005cc2d0})	
	/go/src/app/pkg/rpc/node.go:273 +0xf0	
github.com/kilnfi/cosmos-validator-watcher/pkg/rpc.(*Node).Start(0xc0005c0e10, {0x14403f0, 0xc0005cc2d0})	
	/go/src/app/pkg/rpc/node.go:159 +0x798	
github.com/kilnfi/cosmos-validator-watcher/pkg/rpc.(*Pool).Start.func1()	
	/go/src/app/pkg/rpc/pool.go:35 +0x28	
golang.org/x/sync/errgroup.(*Group).Go.func1()	
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64	
created by golang.org/x/sync/errgroup.(*Group).Go	
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5

docker-compose with prometheus & grafana

Create a docker-compose to quickly set up prometheus demo with a default grafana dashboard

Watch block proposer

Expose validators via http API

expose all validators from x/staking via :8080/validators
goal: easily retrieve any validator pubkey

Add consecutive missed metric

Monitor available commission on owner wallet

Monitor proposals vote

auto detect modules version

so we don't need to specify --x-gov=xxx or --no-staking / --no-commission / etc

Include Grafana Dashboard Json export

Can you also include grafana json export for the dashboard which is shown in screenshot.
It would be helpful to start with

Expose validators rank in the active set

add metrics:

• validator_rank{address, name}
• total_active_validators

Block Polling

Fallback to block polling when websocket isn't available

Add option to disable call to x/staking validators

needed for consumer chains (eg. neutron, stride)

Ensure all nodes use the same chain-id

via the status endpoint

Detect upgrade proposals (hard forks)

detect when hard fork is coming + trigger webhook at given block
goal: auto merge upgrade PR (equivalent of cosmovisor but for gitops env)

Monitor proposals on cosmos consumer chains

Example with Neutron upgrade 2.0.0

neutrond q wasm cs smart neutron1436kxs0w2es6xlqpp9rd35e3d0cjnw4sv8j3a7483sgks29jqwgshlt6zh '{"list_proposals": {}}' --node https://rpc-kralum.neutron-1.neutron.org:443 --output json | jq

$ neutrond q wasm cs smart neutron1436kxs0w2es6xlqpp9rd35e3d0cjnw4sv8j3a7483sgks29jqwgshlt6zh '{"proposal": {"proposal_id": 25}}' --node https://rpc-kralum.neutron-1.neutron.org:443 --output json | jq ".data.proposal.msgs[0].custom.submit_admin_proposal.admin_proposal.software_upgrade_proposal.plan"

{
  "name": "v2.0.0",
  "height": 5416000,
  "info": "{\"binaries\": {\"linux/amd64\": \"https://github.com/neutron-org/neutron/releases/download/v2.0.0/neutrond-linux-amd64?checksum=sha256:ea63e50f7c6305cc87efb7b7b3010797f5a631f7f632ac2678a9bc2f8fcd660c\"}}"
}

Slack connector

Cosmos Validator Watcher is a Prometheus exporter to help you monitor missed blocks on any cosmos-based blockchains in real-time.

Track the staked amount

Could a Slack connector enter in the scope of this project?
For example send a notification when new delegations occur on tracked validators.

Crashloop while running as sidecar for initial minutes

I am running cosmos-validator-watcher as sidecar with my cosmos node. The cosmos-validator-watcher container crashloops till the node is synced with following error.

11:00PM WRN node http://localhost:26657/ is catching up at block 2688992
11:00PM WRN connected to http://localhost:26657/ (but node is catching up) chainID=athens_7001-1 height=2688992
11:00PM ERR error="no nodes synced"

This causes panic as the container is crashlooping. When a pod restarts, its understandable that it will take some time for the node to be synced. It would be better if cosmos-validator has a while condition check ( as a flag atleast ) which checks for the node to be available and then start the watcher rather than just crashlooping.