jvstein/bitcoin-prometheus-exporter

Prometheus exporter for Bitcoin Core nodes

bitcoin-prometheus-exporter's Issues

Consider getting rid of REFRESH_SECONDS

Most Prometheus exporters fetch the metrics immediately after being queried, possibly with some rate limiting. This way the refresh interval doesn't have to be configured in two places, i.e. scraper config as well as exporter config.

Synology, Docker, Bitcoin node, exporter

Hello.
Sorry for my English, I'm using a translator.
I'm working for Sinology, running a bitcoin node in docker (https://registry.hub.docker.com/r/lncm/bitcoind/) with your file: bitcoin.conf.

server=1
rpcauth=alice:8bfdd22c39bc8839b6c8e73659f4bbca$f46a37d99b29771a4b42dec9c7614dabe5400deff1ced0e98bb3d4aa643bed5e
rpcbind=0.0.0.0
rpcport=8332
rpcallowip=0.0.0.0/0

rpcconnect=bitcoind
rpcuser=alice
rpcpassword=password=123&z!=@e

The node is working.

I'm running bitcoin-prometheus-exporter in Synology in docker with a slightly modified docker-compose.yml file:

ash-4.4# cat docker-compose.yml
version: '3.3'
services:
exporter:
container_name: bitcoin-exporter
image: jvstein/bitcoin-prometheus-exporter:latest
build: .
ports:
- "9332:9332"
environment:
BITCOIN_RPC_HOST: bitcoind
BITCOIN_RPC_USER: "alice"
BITCOIN_RPC_PASSWORD: "password=123&z!=@e"
BITCOIN_CONF_PATH: /etc/bitcoin/bitcoin.conf
REFRESH_SECONDS: 1
LOG_LEVEL: "INFO"
volumes:
- /volumeUSB1/usbshare/bitcoin/bitcoin.conf:/etc/bitcoin/bitcoin.conf

I have an error in the logs:

2024/05/04 11:41:32 | stderr | socket.gaierror: [Errno -2] Name does not resolve
2024/05/04 11:41:32 | stderr | for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
2024/05/04 11:41:32 | stderr | for res in getaddrinfo(host, port, 0, SOCK_STREAM):
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
2024/05/04 11:41:32 | stderr | self.sock = self._create_connection(
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 922, in connect
2024/05/04 11:41:32 | stderr | self.connect()
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 951, in send
2024/05/04 11:41:32 | stderr | self.send(msg)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1011, in _send_output
2024/05/04 11:41:32 | stderr | self._send_output(message_body, encode_chunked=encode_chunked)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
2024/05/04 11:41:32 | stderr | self.endheaders(body, encode_chunked=encode_chunked)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1302, in _send_request
2024/05/04 11:41:32 | stderr | self._send_request(method, url, body, headers, encode_chunked)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1256, in request
2024/05/04 11:41:32 | stderr | self.__conn.request('POST', self.__url.path, postdata, headers)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/bitcoin/rpc.py", line 231, in _call
2024/05/04 11:41:32 | stderr | return self._call(service_name, *args)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/bitcoin/rpc.py", line 361, in call
2024/05/04 11:41:32 | stderr | result = rpc_client().call(*args)
2024/05/04 11:41:32 | stderr | File "/monitor/bitcoind-monitor.py", line 199, in bitcoinrpc
2024/05/04 11:41:32 | stderr | res = fn(*args, **kw)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 162, in _call
2024/05/04 11:41:32 | stderr | return self._call(fn, *args, **kw)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 288, in run
2024/05/04 11:41:32 | stderr | raise err
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 232, in _handle_error
2024/05/04 11:41:32 | stderr | self._handle_error(err)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 291, in run
2024/05/04 11:41:32 | stderr | return retrier.run(fn, *args, **_kw)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retry.py", line 129, in wrapper
2024/05/04 11:41:32 | stderr | uptime = int(bitcoinrpc("uptime"))
2024/05/04 11:41:32 | stderr | File "/monitor/bitcoind-monitor.py", line 267, in refresh_metrics
2024/05/04 11:41:32 | stderr | refresh_metrics()
2024/05/04 11:41:32 | stderr | File "/monitor/bitcoind-monitor.py", line 381, in refresh_app
2024/05/04 11:41:32 | stderr | self.result = application(self.environ, self.start_response)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/wsgiref/handlers.py", line 137, in run
2024/05/04 11:41:32 | stderr | Traceback (most recent call last):
2024/05/04 11:41:32 | stderr | 2024-05-04T06:41:32Z INFO Using config file: /etc/bitcoin/bitcoin.conf

what am I doing wrong?

If I go to the page:
http://192.168.x.x:9332/metrics

shows:
A server error occurred. Please contact the administrator.

Add logging

The script should have some logging for debugging purposes. A good example message is on #3. Make log level configurable via an environment variable.

REFRESH_SECONDS is 2x

We're sleeping twice:

• https://github.com/jvstein/bitcoin-prometheus-exporter/blob/master/bitcoind-monitor.py#L222
• https://github.com/jvstein/bitcoin-prometheus-exporter/blob/master/bitcoind-monitor.py#L257

the first one can be removed.

Also, before the sleep we can add logging, e.g.:

        print(
            (
                "Refresh took {} seconds, sleeping for {} seconds"
            ).format(
                duration,
                REFRESH_SECONDS
            )
        )

Example output:

Refresh took 0:00:01.965943 seconds, sleeping for 30.0 seconds
Refresh took 0:00:02.020055 seconds, sleeping for 30.0 seconds
Refresh took 0:00:02.032093 seconds, sleeping for 30.0 seconds
Refresh took 0:00:02.042147 seconds, sleeping for 30.0 seconds

Port could not be cast to integer

Running this either with or without docker, I get a type conversion error:

./bitcoind-monitor.py 
2022-01-12T07:53:19Z INFO Using config file: <default>
Traceback (most recent call last):
  File "/usr/lib/python3.8/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
  File "./bitcoind-monitor.py", line 362, in refresh_app
    refresh_metrics()
  File "./bitcoind-monitor.py", line 246, in refresh_metrics
    uptime = int(bitcoinrpc("uptime"))
  File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retry.py", line 129, in wrapper
    return retrier.run(fn, *args, **_kw)
  File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 291, in run
    self._handle_error(err)
  File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 232, in _handle_error
    raise err
  File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 288, in run
    return self._call(fn, *args, **kw)
  File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 162, in _call
    res = fn(*args, **kw)
  File "./bitcoind-monitor.py", line 207, in bitcoinrpc
    result = rpc_client().call(*args)
  File "./bitcoind-monitor.py", line 194, in rpc_client
    return rpc_client_factory()()
  File "./bitcoind-monitor.py", line 182, in <lambda>
    return lambda: Proxy(btc_conf_file=BITCOIN_CONF_PATH, timeout=TIMEOUT)
  File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/bitcoin/rpc.py", line 353, in __init__
    super(Proxy, self).__init__(service_url=service_url,
  File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/bitcoin/rpc.py", line 196, in __init__
    if self.__url.port is None:
  File "/usr/lib/python3.8/urllib/parse.py", line 174, in port
    raise ValueError(message) from None
ValueError: Port could not be cast to integer value as '28256:8332'

docker-compose.yml (relevant part)

    environment:
      BITCOIN_RPC_HOST: 172.28.1.5
      BITCOIN_RPC_PORT: 28256
      BITCOIN_RPC_USER: redacted
      BITCOIN_RPC_PASSWORD: redacted

bitcoin.conf (bitcoind is in a samourai dojo container)

[main]
rpcconnect=172.28.1.5:28256
rpcuser=redacted
rpcpassword=redacted

I've also tried removing the rpcconnect line from bitcoin.conf to no effect.

If anyone reading this has the exporter working, please share the pip list so I can see what library versions you're running. I would define versions in the Dockerfile to get a predictable outcome.

Docker images for arm64 platform

I'd like to use this for a bitcoin node running on a Raspberry Pi 4, but that requires an image for the arm64 platform. It should be possible to build multi-platform images using Buildx.

rust port

Thanks a lot for sharing this exporter !

I've just made a direct port of your work in rust: bitcoin-exporter

Also perhaps in the near future prometheus scrapping will be even more efficient and exhaustive bitcoind-observer

Regards

No logging output when running in k8s

The container starts but is a black box. Is there some trick I'm missing?

│   bitcoin-core-exporter:                                                                                                                                                                                                                  
│     Container ID:   containerd://7f772ef7adc36b27fa07655c714e6681193f17552907f33f3c735f619e0cde26                                                                                                                                         
│     Image:          jvstein/bitcoin-prometheus-exporter:v0.7.0                                                                                                                                                                            
│     Image ID:       docker.io/jvstein/bitcoin-prometheus-exporter@sha256:db9127a8c09325922c74acd39a24f20a3aea36690d461f6cd98f5cfa86b62cec                                                                                                 
│     Port:           9332/TCP                                                                                                                                                                                                              
│     Host Port:      0/TCP                                                                                                                                                                                                                 
│     State:          Running                                                                                                                                                                                                               
│       Started:      Thu, 10 Feb 2022 13:19:44 -0600                                                                                                                                                                                       
│     Ready:          True                                                                                                                                                                                                                  
│     Restart Count:  0                                                                                                                                                                                                                     
│     Limits:                                                                                                                                                                                                                               
│       cpu:     1                                                                                                                                                                                                                          
│       memory:  1Gi                                                                                                                                                                                                                        
│     Requests:                                                                                                                                                                                                                             
│       cpu:      1                                                                                                                                                                                                                         
│       memory:   1Gi                                                                                                                                                                                                                       
│     Liveness:   tcp-socket :9332 delay=5s timeout=1s period=5s #success=1 #failure=3                                                                                                                                                      
│     Readiness:  tcp-socket :9332 delay=10s timeout=1s period=10s #success=1 #failure=3                                                                                                                                                    
│     Environment:                                                                                                                                                                                                                          
│       REFRESH_SECONDS:       1                                                                                                                                                                                                            
│       LOG_LEVEL:             INFO                                                                                                                                                                                                         
│       BITCOIN_RPC_HOST:      bitcoin-core                                                                                                                                                                                                 
│       BITCOIN_CONF_PATH:     /home/bitcoin/.bitcoin/bitcoin.conf                                                                                                                                                                          
│       BITCOIN_RPC_USER:      rpcuser                                                                                                                                                                                                      
│       BITCOIN_RPC_PASSWORD:  rpcpassword                                                                                                                                                                                                  
│     Mounts:                                                                                                                                                                                                                               
│       /home/bitcoin/.bitcoin/bitcoin.conf from config (rw,path="bitcoin.conf")                                                                                                                                                            
│       /var/run/secrets/kubernetes.io/serviceaccount from default-token-brqnx (ro)

Unable to setup Grafana dashboard

When I try to import the Dashboard on Grafana I am asked to add the data source. When I try to add the exporter as a Prometheus data source in Graphana it shows a bad gateway

Am I missing something?

no bitcoin metrics

Hi,

I tried to run this but no bitcoin metrics are generated… almost all of them are 0…

Please help…

Advertise the project

Thanks for writing this nice exporter! I think it would be a good idea to add it to the list of Prometheus exporters so it's easier to find. There's another bitcoin exporter which is in the list but I think this one is superior.

There's also list of default port allocations.

handle nil metrics more gracefully

👋
as discussed here the bitcoin daemon will return

{
  "errors": [
    "Insufficient data or no feerate found"
  ],
  "blocks": 0
}

if you haven't yet downloaded enough transactions. because python throws a KeyError when accessing a key in a dict using square-bracket notation, pulling metrics will fail for lines 117-120, and stop execution (until you have seen enough transactions). If you switched to using the dict.get(key) notation instead it would at least return None as a default. I don't use Python often so i'm not sure if that affects something later on in your code but this is a major PITB to have to wait until the blockchain is seeded enough to run metrics, esp when running testnet or regtest. Based off this comment even if you started a node completely synced to the chain this may still not return values until you've been running the node a certain amount of time.

I understand you may not want to switch to using .get everywhere because it's basically a rescue nil & could obfuscate other issues but in the event of metrics (such as estimatesmartfee) where it's known the expected key won't exist, i'd consider different error handling.

Questions about bitcoin-python script

Dear Jeff,

I hope this message finds you well. I am writing to ask for some help with the bitcoin-python script. I have a bitcoind instance running and I would like to use the script to get information from it.

Firstly, I was wondering if you could clarify where the environment variables METRICS_ADDR and METRICS_PORT are located. I am not sure how to set these variables so that the script can connect to my bitcoind instance. I see the httpd is trying to bind to 8334 (already in use by bitcoind).

Besides RPC credentials, which other parts should I edit from the script to make it work?

Thank you very much for your time and help. I appreciate any guidance you can provide.

Best regards,
Federico.

Grafana dashboard for this exporter

Is there some dashboard available?
Could you share it please?

Option to disable banlist metrics

I'm using a banlist that's got ~5000 entries which make the exporter output huge. It's not fatal but it's not ideal either. I wonder if there's some more elegant solution than introducing "boolean" env variable to turn it off?

no metrics during sync progress

My node was down for a few days, after resuming it, there were no metrics while the sync was happening.

2020-08-22T23:58:25Z ERROR Retry after exception socket.timeout: timed out
2020-08-22T23:58:55Z ERROR Retry after exception socket.timeout: timed out
2020-08-22T23:59:26Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:00:06Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:00:37Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:01:08Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:01:39Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:02:11Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:02:43Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:03:17Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:03:53Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:04:31Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:04:44Z ERROR Refresh failed during retry. Cause: max timeout exceeded while retrying task: 300s
2020-08-23T00:04:44Z INFO Refresh took 0:06:57.398758 seconds, sleeping for 600.0 seconds

Misc:

• I increased all the timeouts / retry counts / sleep intervals that I could find, yet there was not a single data point going thru.
• I have magnetic disk.
• After sync was done metrics resumed as normal

Monitoring is most needed during unordinary conditions like this one. I don't think it makes sense to export any other metrics during sync, yet sync progress metric is important during sync. For example, it enables sync performance debugging.