When using very large fast-sync value: Out of memory during initial sync and now needs a full resync

[lvnilesh]

When using very large fast-sync value: Out of memory during initial sync and now needs a full resync

[cculianu]

Fast-sync is experimental and i guess temperamental. Do fast-sync = <some_other_value_that_is_smaller_than_what_you_already_tried_until_no_OOM_happens>.

[lvnilesh]

I will try with 1024 and report back.

[cculianu]

Also, are you using arm by any chance? On rpi? Or a real non-rpi machine?

[lvnilesh]

Thanks. It is a real machine:) an Intel NUC.

[cculianu]

Ok do let me know if it works. Note that even a modest fast-sync = 512 ends up working wonders in terms of speed. 5000 is a lot and I guess some of the internal data structures used end up being inefficient with large values. It's... experimental. Let me know if it works with a much smaller value.

[lvnilesh]

Finally it worked.

I used 1024. I kept watching memory. I restarted fulcrum service multiple times. UPS protected me from another crash during a short power outage in the middle of this fresh sync which took about 1.6 days.

Successfully connected to Sparrow.

Two questions:

1. What's the correct configuration for putting this behind an Nginx reverse proxy that already has lets encrypt capability?

2. What does this mean? These messages appear everytime Sparrow attempts to establish a connection to Fulcrum server at port 50002.

<SslSrv 0.0.0.0:50002> [Qt Warning] QSslSocket: cannot call unresolved function SSL_get_peer_certificate (:0, )
<SslSrv 0.0.0.0:50002> New SSL Client.12544 192.168.1.3:63034, 1 client total

Thanks.

Here are the finishing logs.

May 19 14:29:47 chromebox systemd[1]: Started Fulcrum.
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.335] [Qt Warning] QSslSocket: cannot resolve EVP_PKEY_base_id (:0, )
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.335] [Qt Warning] QSslSocket: cannot resolve SSL_get_peer_certificate (:0, )
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341] Loaded SSL certificate: Internet Widgits Pty Ltd  expires: Thu May 13 2032 04:20:03
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341] Loaded key type: private algorithm: RSA
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341] Enabled JSON parser: simdjson
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341] simdjson implementations:
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341]     haswell: Intel/AMD AVX2  [supported]
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341]     westmere: Intel/AMD SSE4.2  [supported]
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341]     fallback: Generic fallback implementation  [supported]
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.341] active implementation: haswell
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] jemalloc: version 5.2.1-0-gea6b3e9
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] Qt: version 5.15.2
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] rocksdb: version 6.14.6-ed43161
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] simdjson: version 0.6.0
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] ssl: OpenSSL 3.0.2 15 Mar 2022
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] zmq: libzmq version: 4.3.3, cppzmq version: 4.7.1
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] Fulcrum 1.6.0 (Release 5e95c0f) - Thu May 19, 2022 14:29:47.343 UTC - starting up ...
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] Max open files: 32767 (increased from default: 20000)
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.343] Loading database ...
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.780] DB memory: 512.00 MiB
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.780] Coin: BTC
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.780] Chain: main
May 19 14:29:47 chromebox Fulcrum[179714]: [2022-05-19 14:29:47.780] Verifying headers ...
May 19 14:29:50 chromebox Fulcrum[179714]: [2022-05-19 14:29:50.317] Initializing header merkle cache ...
May 19 14:29:51 chromebox Fulcrum[179714]: [2022-05-19 14:29:51.322] Checking tx counts ...
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.473] 734832435 total transactions
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.480] UTXO set: 81928024 utxos, 6881.954 MB
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.665] BitcoinDMgr: starting 3 bitcoin RPC clients ...
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.665] BitcoinDMgr: started ok
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.774] <Controller> Block height 737050, up-to-date
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.774] SrvMgr: starting 1 service ...
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.774] Starting listener service for SslSrv 0.0.0.0:50002 ...
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.775] Service started, listening for connections on 0.0.0.0:50002
May 19 14:30:03 chromebox Fulcrum[179714]: [2022-05-19 14:30:03.775] <Controller> Starting ZMQ Notifier (hashblock) ...
May 19 14:30:24 chromebox Fulcrum[179714]: [2022-05-19 14:30:24.898] <SslSrv 0.0.0.0:50002> [Qt Warning] QSslSocket: cannot call unresolved function SSL_get_peer_certificate (:0, )
May 19 14:30:24 chromebox Fulcrum[179714]: [2022-05-19 14:30:24.902] <SslSrv 0.0.0.0:50002> New SSL Client.12544 192.168.1.3:63034, 1 client total
May 19 14:30:49 chromebox Fulcrum[179714]: [2022-05-19 14:30:49.317] <SslSrv 0.0.0.0:50002> [Qt Warning] QSslSocket: cannot call unresolved function SSL_get_peer_certificate (:0, )
May 19 14:30:49 chromebox Fulcrum[179714]: [2022-05-19 14:30:49.317] <SslSrv 0.0.0.0:50002> New SSL Client.12779 192.168.1.3:63048, 1 client total
May 19 14:30:53 chromebox Fulcrum[179714]: [2022-05-19 14:30:53.416] <SslSrv 0.0.0.0:50002> [Qt Warning] QSslSocket: cannot call unresolved function SSL_get_peer_certificate (:0, )
May 19 14:30:53 chromebox Fulcrum[179714]: [2022-05-19 14:30:53.416] <SslSrv 0.0.0.0:50002> New SSL Client.12810 192.168.1.3:63054, 1 client total
May 19 14:30:54 chromebox Fulcrum[179714]: [2022-05-19 14:30:54.150] <SslSrv 0.0.0.0:50002> [Qt Warning] QSslSocket: cannot call unresolved function SSL_get_peer_certificate (:0, )
May 19 14:30:54 chromebox Fulcrum[179714]: [2022-05-19 14:30:54.150] <SslSrv 0.0.0.0:50002> New SSL Client.12820 192.168.1.3:63055, 1 client total
May 19 14:30:54 chromebox Fulcrum[179714]: [2022-05-19 14:30:54.199] <Controller> 12732 mempool txs involving 95216 addresses
May 19 14:30:54 chromebox Fulcrum[179714]: [2022-05-19 14:30:54.343] <SslSrv 0.0.0.0:50002> [Qt Warning] QSslSocket: cannot call unresolved function SSL_get_peer_certificate (:0, )
May 19 14:30:54 chromebox Fulcrum[179714]: [2022-05-19 14:30:54.343] <SslSrv 0.0.0.0:50002> New SSL Client.12832 192.168.1.3:63056, 1 client total
May 19 14:30:58 chromebox Fulcrum[179714]: [2022-05-19 14:30:58.030] <SslSrv 0.0.0.0:50002> [Qt Warning] QSslSocket: cannot call unresolved function SSL_get_peer_certificate (:0, )
May 19 14:30:58 chromebox Fulcrum[179714]: [2022-05-19 14:30:58.030] <SslSrv 0.0.0.0:50002> New SSL Client.12858 192.168.1.3:63057, 1 client total

[lvnilesh]

electrs provides this guidance. https://github.com/romanz/electrs/blob/master/doc/config.md#extra-configuration-suggestions

How should that be modified for Fulcrum?

[EchterAgo]

You can ignore the warning about SSL_get_peer_certificate, the server does not need peer certificates.

IIRC Fulcrum does not yet support any method to pass the real client IP from a reverse proxy like X-Forwarded-For, which means you will run into per-address limits faster in a reverse proxy setup because it will only see the proxy's address as the source (see #29). You could increase the limits for max_clients_per_ip, max_subs_per_ip and max_batch in the config file.

[lvnilesh]

Thanks @EchterAgo. I will wait for X-Forwarded-For support.

In the meanwhile, for a single client usage scenario, I modified fulcrum.conf like this

tcp = 0.0.0.0:50001
# ssl = 0.0.0.0:50002
# cert = /path/to/.fulcrum/cert.pem
# key = /path/to/.fulcrum/key.pem

Used this nginx config to get ssl://domain:50002 to work.

          stream {
            upstream electrumx {
              server 127.0.0.1:50001;
            }
            server {
              listen 50002 ssl;
              proxy_pass electrumx;

              ssl_certificate /path/to/certs/domain.com-fullchain.crt;
              ssl_certificate_key /path/to/certs/domain.com.pem;
              ssl_session_cache shared:SSL:1m;
              ssl_session_timeout 4h;
              ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
              ssl_prefer_server_ciphers on;
            }
          }

Thanks.

[cculianu]

X-Forward-For would only work with http.. Electrum protocol is not http. However, if using websockets that might work. But then I don't know if Sparrow supports the websocket interface. Or am I missing something?

[lvnilesh]

I may be mistaken about X-Forwarded-For and I need to read a bit more.

[lvnilesh]

It is considered safe to expose fulcrum port 50002 directly on the public internet without going through some thing like Nginx?

What are the best practices?