blinklabs-io/snek

Event-driven tool for tailing the Cardano blockchain

snek

snek is a tool for tailing the Cardano blockchain and emitting events for each block and transaction that it sees.

How it works

Input can be a local or remote Cardano full node, using either NtC (local UNIX socket, TCP over socat) or NtN to remote nodes.

Events are created with a simple schema.

{
    "type": "event type",
    "timestamp": "wall clock timestamp of event",
    "context": "metadata about the event",
    "payload": "the full event specific payload"
}

The chainsync input produces three event types: block, rollback, and transaction. Each type has a unique payload.

block:

{
    "context": {
        "blockNumber": 123,
        "slotNumber": 1234567,
    },
    "payload": {
        "blockBodySize": 123,
        "issuerVkey": "a712f81ab2eac...",
        "blockHash": "abcd123...",
        "blockCbor": "85828a1a000995c21..."
    }
}

rollback:

{
    "payload": {
        "blockHash": "abcd123...",
        "slotNumber": 1234567
    }
}

transaction:

{
    "context": {
        "blockNumber": 123,
        "slotNumber": 1234567,
        "transactionHash": "0deadbeef123...",
        "transactionIdx": 0,
    },
    "payload": {
        "blockHash": "abcd123...",
        "transactionCbor": "a500828258200a1ad..."
        "inputs": [
          "abcdef123...#0",
          "abcdef123...#1",
        ],
        "outputs": [
            {
                "address": "addr1qwerty123...",
                "amount":  12345687,
                "assets": [
                    {
                        "name": "Foo",
                        "nameHex": "abcd123...",
                        "amount": 123,
                        "fingerprint": "asset1abcd...",
                        "policyId": "54321..."
                    }
                ]
            }
        ],
        "metadata": {
            674: {
                "msg": [
                    "Test message"
                ]
            }
        },
        "fee": 1234567,
        "ttl": 123
    }
}

Each event is output individually. The log output prints each event to stdout using Uber's Zap logging library.

Configuration

snek supports multiple configuration methods for versatility: commandline arguments, YAML config file, and environment variables (in that order).

You can get a list of all available commandline arguments by using the -h/-help flag.

$ ./snek -h
Usage of snek:
  -config string
        path to config file to load
  -input string
        input plugin to use, 'list' to show available (default "chainsync")
  -input-chainsync-address string
        specifies the TCP address of the node to connect to
...
  -output string
        output plugin to use, 'list' to show available (default "log")
  -output-log-level string
        specifies the log level to use (default "info")

Each commandline argument (other than -config) has a corresponding environment variable. For example, the -input option has the INPUT environment variable, the -input-chainsync-address option has the INPUT_CHAINSYNC_ADDRESS environment variable, and -output has OUTPUT.

You can also specify each option in the config file.

input: chainsync

output: log

Plugin arguments can be specified under a special top-level key in the config file.

plugins:
  input:
    chainsync:
      network: preview

  output:
    log:
      level: info

Filtering

snek supports filtering events before they are output using multiple criteria. An event must match all configured filters to be emitted. Each filter supports specifying multiple possible values separated by commas. When specifying multiple values for a filter, only one of the values specified must match an event.

You can get a list of all available filter options by using the -h/-help flag.

$ ./snek -h
Usage of snek:
...
  -filter-address string
        specifies address to filter on
  -filter-asset string
        specifies the asset fingerprint (asset1xxx) to filter on
  -filter-policy string
        specifies asset policy ID to filter on
  -filter-type string
        specifies event type to filter on
...

Multiple filter options can be used together, and only events matching all filters will be output.

Example usage

Native using remote node

export INPUT_CHAINSYNC_NETWORK=preview
./snek 

Alternatively using equivalent commandline options:

./snek \
  -input-chainsync-network preview

In Docker using local node

First, follow the instructions for Running a Cardano Node in Docker.

docker run --rm -ti \
  -v node-ipc:/node-ipc \
  ghcr.io/blinklabs-io/snek:main

Filtering

Filtering on event type

Only output chainsync.transaction event types

$ snek -filter-type chainsync.transaction

Only output chainsync.rollback and chainsync.block event types

$ snek -filter-type chainsync.transaction,chainsync.block

Filtering on asset policy

Only output transactions involving an asset with a particular policy ID

$ snek -filter-type chainsync.transaction -filter-policy 13aa2accf2e1561723aa26871e071fdf32c867cff7e7d50ad470d62f

Filtering on asset fingerprint

Only output transactions involving a particular asset

$ snek -filter-type chainsync.transaction -filter-asset asset108xu02ckwrfc8qs9d97mgyh4kn8gdu9w8f5sxk

Filtering on a policy ID and asset fingerprint

Only output transactions involving both a particular policy ID and a particular asset (which do not need to be related)

$ snek -filter-type chainsync.transaction -filter-asset asset108xu02ckwrfc8qs9d97mgyh4kn8gdu9w8f5sxk -filter-policy 13aa2accf2e1561723aa26871e071fdf32c867cff7e7d50ad470d62f

Filtering on an address

Only output transactions with outputs matching a particular address

$ snek -filter-type chainsync.transaction -filter-address addr1qyht4ja0zcn45qvyx477qlyp6j5ftu5ng0prt9608dxp6l2j2c79gy9l76sdg0xwhd7r0c0kna0tycz4y5s6mlenh8pq4jxtdy

Filtering on a stake address

Only output transactions with outputs matching a particular stake address

$ snek -filter-type chainsync.transaction -filter-address stake1u9f9v0z5zzlldgx58n8tklphu8mf7h4jvp2j2gddluemnssjfnkzz

Push notifications

The example shows how push notification output can be used with filtering options. In this example, push notifications will be sent to the block events. Push notifications will be sent to the specified project_id in the serviceAccount.json file. Please refer to https://github.com/blinklabs-io/snek-mobile for more details on how to send push notifications to snek-mobile.

$ snek -filter-type chainsync.block -output push -output-push-serviceAccountFilePath /path/to/serviceAccount.json

snek's Issues

rework socket, host/port, and NtC vs. NtN selection for chainsync input

Instead of having the use of a UNIX socket or TCP host/port and the use of NtC vs. NtN as separately configurable things, we should instead have specifying a UNIX socket automatically use NtC and a TCP host/port automatically use NtN, with an extra setting to allow/force the use of NtC over TCP (when using socat and such)

Support webhooks

Create a webhook output plugin to send payloads as webhooks.

Add incremental backoff for reconnect in chainsync input

When there's a problem with the node, it will currently rapidly reconnect. We should have some sort of incremental backoff to avoid hammering an ailing node and blowing up the logs

Chainsync input should follow tip during bulk sync

We currently use BlockFetch and then start ChainSync from where we left off, but that results in the tip information sent to the consumer being stale during the bulk sync. We should start a sync and ignore blocks to update the tip info, and then we can cancel it and start a new sync after the bulk sync finishes.

Add Epoch number to ChainSyncStatus

This will allow downstream consumers to process per-epoch data more easily.

add support for "bulk" chain-sync mode in chainsync input

This involves using block-fetch with a large range of blocks. This also gets tricky when you want to start at chain genesis, because you have to actually start a chain-sync, receive the first RollForward message, and parse the block to get the block hash to use as the start of the range with block-fetch. The IntersectFound and RollBackward messages just give the empty "genesis" point, and you need the first actual block to get a slot and block hash.

Update Discord webhook links for known Cardano networks

Currently, we link to cexplorer.io for everything, but we need to link to the appropriate instance of the application for the network we're watching.

Get compliant with Windows

Windows claims Snek is a virus. Find out why and fix it.

show description for each plugin when listing

We currently just show a simple list of plug-in names when using -input list, but we should also have descriptions.

Icon image should be embedded

The icon only shows up if you're running from the repo, as the file path is relative and the file doesn't exist for binary users. Look at how to embed the icon into the binary.

Logging framework for plugins

We don't currently have any logging in plugins, since they are intended to be reusable. We should create a simple Logger interface (info/warn/debug/error) and allow specifying an optional logger in each plugin. The snek CLI should provide the logger to each plugin instance.

Expose era and block/tx type in chainsync input

We should probably expose the era name in the event JSON, but we can also expose the underlying block/tx type identifier for programmatic access

QR Code Generation and Scanning for FCM Token Submission

Scenario: Successfully generating and scanning QR code to send fcmToken

Given a QR code generator that produces codes in JSON format
When I generate a QR code with an API POST endpoint
Then the QR code should represent the correct JSON with the POST endpoint

Given the generated QR code
And a snek-mobile app ready to scan
When I scan the QR code with snek-mobile
Then the snek-mobile app should send the fcmToken to the API POST endpoint

persistent state support

This is mostly for storing the chainsync cursor, but it should be usable by all plugins. We should support multiple on-disk formats, such as JSON and sqlite

create base plugin object

Move the basic start/stop logic and channels to the base object

Support filter-pool argument

Let's support a filter-pool argument which should filter to:

• Blocks issued by the pool
• Delegation transactions to the pool

add transaction count to chainsync Block event

Chainsync input auto-reconnect fails if the connection handshake fails

{"level":"info","timestamp":"2024-01-20T14:57:17Z","caller":"chainsync/chainsync.go:215","msg":"connected to node at backbone.cardano-mainnet.iohk.io:3001","plugin":"input.chainsync"}
{"level":"info","timestamp":"2024-01-20T16:08:28Z","caller":"chainsync/chainsync.go:223","msg":"reconnecting to backbone.cardano-mainnet.iohk.io:3001 due to error: protocol error: keep-alive: timeout waiting on transition from protocol state Server","plugin":"input.chainsync"}
{"level":"info","timestamp":"2024-01-20T16:08:28Z","caller":"chainsync/chainsync.go:215","msg":"connected to node at backbone.cardano-mainnet.iohk.io:3001","plugin":"input.chainsync"}
{"level":"info","timestamp":"2024-01-20T16:10:38Z","caller":"chainsync/chainsync.go:223","msg":"reconnecting to backbone.cardano-mainnet.iohk.io:3001 due to error: protocol error: keep-alive: timeout waiting on transition from protocol state Server","plugin":"input.chainsync"}
{"level":"fatal","timestamp":"2024-01-20T16:10:44Z","caller":"snek/main.go:162","msg":"pipeline failed: handshake: timeout waiting on transition from protocol state Confirm","stacktrace":"main.main\n\t/home/runner/work/snek/snek/cmd/snek/main.go:162\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"}

create tests for filter plugins

The filter plugins can be used in isolation, so they're easily testable.

support for filtering by stake address

We currently do a "dumb" string match, but now we have support in gOuroboros for getting the stake address from any given ledger address

latest version GLIBC errors

ran the latest version but received this error

./snek-v0.11.0-linux-amd64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./snek-v0.11.0-linux-amd64)
./snek-v0.11.0-linux-amd64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./snek-v0.11.0-linux-amd64)

Support push notifications

Let's create a push notifications output and a simple receiver app.

Create framework for reporting plugin state to consumer

This is mostly needed to allow the chainsync input plug-in consumer to record the chainsync cursor for resuming after restart

make inclusion of CBOR in chainsync events optional

Turn it off by default and add a flag to turn it on

Switch to slog interface for plugin logging

We're currently using an ugly custom logging interface, but we should switch to slog. We don't want to do this until Go 1.22 is out, since slog is only available in Go 1.21+, and we want to support the 2 latest major versions

filter support

We want to be able to pass each event through a series of filter plugins in a pipeline. We should initially have an expression-based filter plugin, but we can also have plugins for things like executing an external command or executing a custom function (when using the Go API).

For UX reasons, we may want to not automatically register filter plugins and have an explicit config option for providing the filter expression via the CLI.

Fcm tokens are not properly updated

In the function AddNewFcmTokens, we are only adding new tokens. We don't remove tokens.
https://github.com/blinklabs-io/snek/blob/main/output/push/push.go#L192

The expected way is to add/remove tokens when they are added or removed through API.

filtering for chainsync events

This should include the ability to match on address and asset policy ID and/or name

Chainsync input auto-reconnect fails when it happens before any blocks are fetched

We use the last few cached blocks as the intersect point when reconnecting, but that list is empty when we haven't received any blocks after the first connect. We should instead use the original intersect points in this case.

reference inputs missing from transaction

Support for multiple values in filters

Each type of filter should accept multiple values that it will allow. This can be represented by comma-separated values on the commandline and a list in the code.

track/show version in compiled binary

We can pass a git hash or tag as the version at build time to incorporate in published binaries. We can then show it with snek -version

Add support to configure a port and host of the API server

ACTION REQUIRED: Changes to pulling Chainguard Images

Hey there Chainguard here.

We noticed that you are using Chainguard Images, thank you! We wanted to make you aware of an upcoming change that will impact your project.

Starting August 16, 2023 public users will no longer be able to pull images from our registry (cgr.dev/chainguard) by tags other than latest or latest-dev. Please see the announcement for more information.

You are currently using the following.

In https://github.com/blinklabs-io/snek/blob/7a265294440058a6caa68ea9674dcd2d328ef57a/Dockerfile:

• cgr.dev/chainguard/go:1.19

Our goal is to prevent your project from experiencing any disruptions. Please see the migration guide for options.

If there's more we can do to help please reply to this issue or email us at [email protected].

Thank you!

Allow configuring notification title

This will make it easy to differentiate between multiple instances.

synchronization issues in plugin goroutines on shutdown

We should close doneChan and then use a sync.WaitGroup to wait for all goroutines to exit before closing the rest of the channels

Use cobra for commandline handling

We'll still need to do much of our custom stuff, but we can potentially get better usage output and shell completion

Auto-reconnect in chainsync input plugin

This should be optional but enabled but default. We should reconnect with a number of retries and throw an error on failure. The chain sync should be setup again from scratch, but with our last known chain point(s) as the intersect point