pinax-network/substreams-sink-prometheus

Substreams Prometheus CLI Node.js

Substreams Prometheus CLI Node.js

substreams-sink-prometheus is a tool that allows developers to pipe data extracted metrics from a blockchain into a Prometheus time series database.

📖 Documentation

https://www.npmjs.com/package/substreams-sink-prometheus

Further resources

• Substreams documentation
• Prometheus documentation
• Substreams Sink Prometheus Rust

Protobuf

• pinax.substreams.sink.prometheus.v1.PrometheusOperations

Docker environment

Pull from GitHub Container registry

docker pull ghcr.io/pinax-network/substreams-sink-prometheus:latest

Build from source

docker build -t substreams-sink-prometheus .

Run with .env file

docker run -it --rm --env-file .env substreams-sink-prometheus run

Install globally via npm

$ npm install -g substreams-sink-prometheus

Run

Substreams Prometheus sink module

Options:
  -e --substreams-endpoint <string>    Substreams gRPC endpoint to stream data from (env: SUBSTREAMS_ENDPOINT)
  --manifest <string>                  URL of Substreams package (env: MANIFEST)
  --module-name <string>               Name of the output module (declared in the manifest) (env: MODULE_NAME)
  -s --start-block <int>               Start block to stream from (defaults to -1, which means the initialBlock of the first module you are streaming) (default: "-1", env: START_BLOCK)
  -t --stop-block <int>                Stop block to end stream at, inclusively (env: STOP_BLOCK)
  -p, --params <string...>             Set a params for parameterizable modules. Can be specified multiple times. (ex: -p module1=valA -p module2=valX&valY) (default: [], env: PARAMS)
  --substreams-api-token <string>      API token for the substream endpoint (default: "", env: SUBSTREAMS_API_TOKEN)
  --delay-before-start <int>           [OPERATOR] Amount of time in milliseconds (ms) to wait before starting any internal processes, can be used to perform to maintenance on the pod before actually letting it starts (default: 0,
                                       env: DELAY_BEFORE_START)
  --cursor-file <string>               Cursor lock file (default: "cursor.lock", env: CURSOR_FILE)
  --disable-production-mode            Disable production mode, allows debugging modules logs, stops high-speed parallel processing (default: true, env: PRODUCTION_MODE)
  --restart-inactivity-seconds <int>   If set, the sink will restart when inactive for over a certain amount of seconds (default: 60, env: RESTART_INACTIVITY_SECONDS)
  --hostname <string>                  The process will listen on this hostname for any HTTP and Prometheus metrics requests (default: "localhost", env: HOSTNAME)
  --port <int>                         The process will listen on this port for any HTTP and Prometheus metrics requests (default: 9102, env: PORT)
  --metrics-labels [string...]         To apply generic labels to all default metrics (ex: --labels foo=bar) (default: {}, env: METRICS_LABELS)
  --collect-default-metrics <boolean>  Collect default metrics (default: false, env: COLLECT_DEFAULT_METRICS)
  --verbose                            Enable verbose logging (default: false, env: VERBOSE)
  -h, --help                           display help for command

Open the browser at http://localhost:9102

Features

• Consume *.spkg from:
  • Load URL or IPFS
  • Read from *.spkg local filesystem
  • Read from substreams.yaml local filesystem
• Prometheus metrics
  • COUNTER
  • GAUGE
  • HISTOGRAM
  • SUMMARY
• Handle cursor on restart (saves cursor.lock file on disk)

substreams-sink-prometheus's Issues

Smaller images for container app

Smaller images for container app

Following the sprint meeting, @DenisCarriere commented on how big the docker images were.

I did some work on how to resolve this issue and this is the results.
The dockerfiles are under the docker-images branch.

Size of the images

Solution	Dockerfile	Size
Current	Dockerfile	1.12G
Alpine base	Dockerfile (replace node:current with node:20-alpine)	300M
Alpine multi-stage build	Dockerfile.alpine	274M
Distroless multi-stage	Dockerfile.Distroless	265M

Conclusion

Using a multi-staged build process should be the norm moving forward. It allows smaller images at practically no cost in complexity.

The use of a distroless image is more debatable. It is considered more secure (the production image doesn't ship with a shell, and other benefits...) but pinning a specific version for our use case is somewhat messy since no real stable version of these images exist.

We could pin a commit-SHA, but we would have no real guarantee that this will remain in the future. The difference in size is also not that significant.

I would love some input from others on this topic.