jvstein / bitcoin-prometheus-exporter Goto Github PK
View Code? Open in Web Editor NEWPrometheus exporter for Bitcoin Core nodes
License: BSD 3-Clause "New" or "Revised" License
Prometheus exporter for Bitcoin Core nodes
License: BSD 3-Clause "New" or "Revised" License
Most Prometheus exporters fetch the metrics immediately after being queried, possibly with some rate limiting. This way the refresh interval doesn't have to be configured in two places, i.e. scraper config as well as exporter config.
Hello.
Sorry for my English, I'm using a translator.
I'm working for Sinology, running a bitcoin node in docker (https://registry.hub.docker.com/r/lncm/bitcoind/) with your file: bitcoin.conf.
server=1
rpcauth=alice:8bfdd22c39bc8839b6c8e73659f4bbca$f46a37d99b29771a4b42dec9c7614dabe5400deff1ced0e98bb3d4aa643bed5e
rpcbind=0.0.0.0
rpcport=8332
rpcallowip=0.0.0.0/0
rpcconnect=bitcoind
rpcuser=alice
rpcpassword=password=123&z!=@e
The node is working.
I'm running bitcoin-prometheus-exporter in Synology in docker with a slightly modified docker-compose.yml file:
ash-4.4# cat docker-compose.yml
version: '3.3'
services:
exporter:
container_name: bitcoin-exporter
image: jvstein/bitcoin-prometheus-exporter:latest
build: .
ports:
- "9332:9332"
environment:
BITCOIN_RPC_HOST: bitcoind
BITCOIN_RPC_USER: "alice"
BITCOIN_RPC_PASSWORD: "password=123&z!=@e"
BITCOIN_CONF_PATH: /etc/bitcoin/bitcoin.conf
REFRESH_SECONDS: 1
LOG_LEVEL: "INFO"
volumes:
- /volumeUSB1/usbshare/bitcoin/bitcoin.conf:/etc/bitcoin/bitcoin.conf
I have an error in the logs:
2024/05/04 11:41:32 | stderr | socket.gaierror: [Errno -2] Name does not resolve
2024/05/04 11:41:32 | stderr | for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
2024/05/04 11:41:32 | stderr | for res in getaddrinfo(host, port, 0, SOCK_STREAM):
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
2024/05/04 11:41:32 | stderr | self.sock = self._create_connection(
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 922, in connect
2024/05/04 11:41:32 | stderr | self.connect()
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 951, in send
2024/05/04 11:41:32 | stderr | self.send(msg)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1011, in _send_output
2024/05/04 11:41:32 | stderr | self._send_output(message_body, encode_chunked=encode_chunked)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
2024/05/04 11:41:32 | stderr | self.endheaders(body, encode_chunked=encode_chunked)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1302, in _send_request
2024/05/04 11:41:32 | stderr | self._send_request(method, url, body, headers, encode_chunked)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/http/client.py", line 1256, in request
2024/05/04 11:41:32 | stderr | self.__conn.request('POST', self.__url.path, postdata, headers)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/bitcoin/rpc.py", line 231, in _call
2024/05/04 11:41:32 | stderr | return self._call(service_name, *args)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/bitcoin/rpc.py", line 361, in call
2024/05/04 11:41:32 | stderr | result = rpc_client().call(*args)
2024/05/04 11:41:32 | stderr | File "/monitor/bitcoind-monitor.py", line 199, in bitcoinrpc
2024/05/04 11:41:32 | stderr | res = fn(*args, **kw)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 162, in _call
2024/05/04 11:41:32 | stderr | return self._call(fn, *args, **kw)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 288, in run
2024/05/04 11:41:32 | stderr | raise err
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 232, in _handle_error
2024/05/04 11:41:32 | stderr | self._handle_error(err)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retrier.py", line 291, in run
2024/05/04 11:41:32 | stderr | return retrier.run(fn, *args, **_kw)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/site-packages/riprova/retry.py", line 129, in wrapper
2024/05/04 11:41:32 | stderr | uptime = int(bitcoinrpc("uptime"))
2024/05/04 11:41:32 | stderr | File "/monitor/bitcoind-monitor.py", line 267, in refresh_metrics
2024/05/04 11:41:32 | stderr | refresh_metrics()
2024/05/04 11:41:32 | stderr | File "/monitor/bitcoind-monitor.py", line 381, in refresh_app
2024/05/04 11:41:32 | stderr | self.result = application(self.environ, self.start_response)
2024/05/04 11:41:32 | stderr | File "/usr/local/lib/python3.8/wsgiref/handlers.py", line 137, in run
2024/05/04 11:41:32 | stderr | Traceback (most recent call last):
2024/05/04 11:41:32 | stderr | 2024-05-04T06:41:32Z INFO Using config file: /etc/bitcoin/bitcoin.conf
what am I doing wrong?
If I go to the page:
http://192.168.x.x:9332/metrics
shows:
A server error occurred. Please contact the administrator.
The script should have some logging for debugging purposes. A good example message is on #3. Make log level configurable via an environment variable.
We're sleeping twice:
the first one can be removed.
Also, before the sleep we can add logging, e.g.:
print(
(
"Refresh took {} seconds, sleeping for {} seconds"
).format(
duration,
REFRESH_SECONDS
)
)
Example output:
Refresh took 0:00:01.965943 seconds, sleeping for 30.0 seconds
Refresh took 0:00:02.020055 seconds, sleeping for 30.0 seconds
Refresh took 0:00:02.032093 seconds, sleeping for 30.0 seconds
Refresh took 0:00:02.042147 seconds, sleeping for 30.0 seconds
Running this either with or without docker, I get a type conversion error:
./bitcoind-monitor.py
2022-01-12T07:53:19Z INFO Using config file: <default>
Traceback (most recent call last):
File "/usr/lib/python3.8/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "./bitcoind-monitor.py", line 362, in refresh_app
refresh_metrics()
File "./bitcoind-monitor.py", line 246, in refresh_metrics
uptime = int(bitcoinrpc("uptime"))
File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retry.py", line 129, in wrapper
return retrier.run(fn, *args, **_kw)
File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 291, in run
self._handle_error(err)
File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 232, in _handle_error
raise err
File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 288, in run
return self._call(fn, *args, **kw)
File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/riprova/retrier.py", line 162, in _call
res = fn(*args, **kw)
File "./bitcoind-monitor.py", line 207, in bitcoinrpc
result = rpc_client().call(*args)
File "./bitcoind-monitor.py", line 194, in rpc_client
return rpc_client_factory()()
File "./bitcoind-monitor.py", line 182, in <lambda>
return lambda: Proxy(btc_conf_file=BITCOIN_CONF_PATH, timeout=TIMEOUT)
File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/bitcoin/rpc.py", line 353, in __init__
super(Proxy, self).__init__(service_url=service_url,
File "/opt/bitcoind_exporter/.venv/prometheus_client/lib/python3.8/site-packages/bitcoin/rpc.py", line 196, in __init__
if self.__url.port is None:
File "/usr/lib/python3.8/urllib/parse.py", line 174, in port
raise ValueError(message) from None
ValueError: Port could not be cast to integer value as '28256:8332'
docker-compose.yml (relevant part)
environment:
BITCOIN_RPC_HOST: 172.28.1.5
BITCOIN_RPC_PORT: 28256
BITCOIN_RPC_USER: redacted
BITCOIN_RPC_PASSWORD: redacted
bitcoin.conf (bitcoind is in a samourai dojo container)
[main]
rpcconnect=172.28.1.5:28256
rpcuser=redacted
rpcpassword=redacted
I've also tried removing the rpcconnect
line from bitcoin.conf to no effect.
If anyone reading this has the exporter working, please share the pip list so I can see what library versions you're running. I would define versions in the Dockerfile to get a predictable outcome.
I'd like to use this for a bitcoin node running on a Raspberry Pi 4, but that requires an image for the arm64 platform. It should be possible to build multi-platform images using Buildx.
Thanks a lot for sharing this exporter !
I've just made a direct port of your work in rust: bitcoin-exporter
Also perhaps in the near future prometheus scrapping will be even more efficient and exhaustive bitcoind-observer
Regards
The container starts but is a black box. Is there some trick I'm missing?
│ bitcoin-core-exporter:
│ Container ID: containerd://7f772ef7adc36b27fa07655c714e6681193f17552907f33f3c735f619e0cde26
│ Image: jvstein/bitcoin-prometheus-exporter:v0.7.0
│ Image ID: docker.io/jvstein/bitcoin-prometheus-exporter@sha256:db9127a8c09325922c74acd39a24f20a3aea36690d461f6cd98f5cfa86b62cec
│ Port: 9332/TCP
│ Host Port: 0/TCP
│ State: Running
│ Started: Thu, 10 Feb 2022 13:19:44 -0600
│ Ready: True
│ Restart Count: 0
│ Limits:
│ cpu: 1
│ memory: 1Gi
│ Requests:
│ cpu: 1
│ memory: 1Gi
│ Liveness: tcp-socket :9332 delay=5s timeout=1s period=5s #success=1 #failure=3
│ Readiness: tcp-socket :9332 delay=10s timeout=1s period=10s #success=1 #failure=3
│ Environment:
│ REFRESH_SECONDS: 1
│ LOG_LEVEL: INFO
│ BITCOIN_RPC_HOST: bitcoin-core
│ BITCOIN_CONF_PATH: /home/bitcoin/.bitcoin/bitcoin.conf
│ BITCOIN_RPC_USER: rpcuser
│ BITCOIN_RPC_PASSWORD: rpcpassword
│ Mounts:
│ /home/bitcoin/.bitcoin/bitcoin.conf from config (rw,path="bitcoin.conf")
│ /var/run/secrets/kubernetes.io/serviceaccount from default-token-brqnx (ro)
Hi,
I tried to run this but no bitcoin metrics are generated… almost all of them are 0…
Please help…
Thanks for writing this nice exporter! I think it would be a good idea to add it to the list of Prometheus exporters so it's easier to find. There's another bitcoin exporter which is in the list but I think this one is superior.
There's also list of default port allocations.
👋
as discussed here the bitcoin daemon will return
{
"errors": [
"Insufficient data or no feerate found"
],
"blocks": 0
}
if you haven't yet downloaded enough transactions. because python throws a KeyError
when accessing a key in a dict using square-bracket notation, pulling metrics will fail for lines 117-120, and stop execution (until you have seen enough transactions). If you switched to using the dict.get(key)
notation instead it would at least return None
as a default. I don't use Python often so i'm not sure if that affects something later on in your code but this is a major PITB to have to wait until the blockchain is seeded enough to run metrics, esp when running testnet or regtest. Based off this comment even if you started a node completely synced to the chain this may still not return values until you've been running the node a certain amount of time.
I understand you may not want to switch to using .get
everywhere because it's basically a rescue nil
& could obfuscate other issues but in the event of metrics (such as estimatesmartfee
) where it's known the expected key won't exist, i'd consider different error handling.
Dear Jeff,
I hope this message finds you well. I am writing to ask for some help with the bitcoin-python script. I have a bitcoind instance running and I would like to use the script to get information from it.
Firstly, I was wondering if you could clarify where the environment variables METRICS_ADDR and METRICS_PORT are located. I am not sure how to set these variables so that the script can connect to my bitcoind instance. I see the httpd is trying to bind to 8334 (already in use by bitcoind).
Besides RPC credentials, which other parts should I edit from the script to make it work?
Thank you very much for your time and help. I appreciate any guidance you can provide.
Best regards,
Federico.
Is there some dashboard available?
Could you share it please?
I'm using a banlist that's got ~5000 entries which make the exporter output huge. It's not fatal but it's not ideal either. I wonder if there's some more elegant solution than introducing "boolean" env variable to turn it off?
My node was down for a few days, after resuming it, there were no metrics while the sync was happening.
2020-08-22T23:58:25Z ERROR Retry after exception socket.timeout: timed out
2020-08-22T23:58:55Z ERROR Retry after exception socket.timeout: timed out
2020-08-22T23:59:26Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:00:06Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:00:37Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:01:08Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:01:39Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:02:11Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:02:43Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:03:17Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:03:53Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:04:31Z ERROR Retry after exception socket.timeout: timed out
2020-08-23T00:04:44Z ERROR Refresh failed during retry. Cause: max timeout exceeded while retrying task: 300s
2020-08-23T00:04:44Z INFO Refresh took 0:06:57.398758 seconds, sleeping for 600.0 seconds
Misc:
Monitoring is most needed during unordinary conditions like this one. I don't think it makes sense to export any other metrics during sync, yet sync progress metric is important during sync. For example, it enables sync performance debugging.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.