Wallet history browsing is broken with electrs

[eugenesan]

Wallet history browsing is broken with electrs

[craigraw]

If my understanding is correct, when Sparrow opens TX dialog it uses batched JSON-RPC to fetch all Inputs/Outputs while Electrum fetches only Inputs and only lists Outputs.

This is not correct wrt Sparrow. When a transaction tab is opened, Sparrow will retrieve the transactions for the first 50 inputs and related outputs. If the transaction was sent from the wallet, these transactions are retrieved from the wallet file, otherwise they are retrieved from the connected server (if available). User action must be taken to fetch the next 50 inputs or outputs by clicking the item in the transaction tree.

electrs starts to complain about multiple not optimized queries

This is an unfortunate log message, since it implies that there is something wrong with the client. There is nothing related to call order in the Electrum Server protocol specification (specifically, calling get_history for an unsubscribed scripthash), so really this is an Electrs issue and not a Sparrow one.

Can we limit pre-fetching of tx_hash only to inputs/outputs relevant to the wallet and fetch the rest upon request one by one?

This is really the nub of question - do I make Sparrow less usable for those with more powerful servers in order to reduce the load on one particular server implementation that has an architecture which scales poorly? I don't feel the answer is yes.

What I have done however is make Sparrow use any available wallet history to reduce the number of server calls made when opening a transaction tab. For those browsing their wallet history with "normal" sized transactions, this should make enough of a difference to keep Electrs from being overwhelmed. 02a0a32.

[craigraw]

I'm not sure what you mean. Sparrow uses the same call pattern when loading a wallet that Electrum. The main difference is that Sparrow uses batched JSON-RPC, which is more network efficient. It is impossible for it to be any more efficient/lightweight using the Electrum Server protocol, as far as I know.

I've covered the reason that Electrs can't scale well here: https://sparrowwallet.com/docs/server-performance.html#discussion. There is nothing Sparrow can do about this. I wouldn't call Fulcrum extremely unstable either, while noting it does have an open issue related to unclean shutdowns.

[eugenesan]

Thank you for explaining.

I realize that electrs is very limited but it does work with electrum why wouldn't it with Sparrow?
Perhaps there is a difference in how transaction data is fetched.

If my understanding is correct, when Sparrow opens TX dialog it uses batched JSON-RPC to fetch all Inputs/Outputs while Electrum fetches only Inputs and only lists Outputs.

Let's imaging a wallet with a transaction that resulted from the following 2 scenarios:

1. Coinbase (Chain) -> Pool Payout to 500 addresses
2. Coinbase (Exchange) -> Batched payout to 500 addresses
   Now if open that transaction in Sparrow it will try to fetch 501 I/O but we need only 1 input and 1 relevant to the wallet Output.
   Electrum fetches only the Inputs and shows Outputs as a simple list with no details.

To test my theory I did another test by restoring the same wallet that causes issues in Sparrow, in Electrum:

1. Electrum loads the wallet without any issues and I could not open any dialog while browsing the wallet to cause any issues with the server or even affect CPU usage.
2. Sparrow loads the wallet without any issues but the moment I open any transaction tab with more than a couple of Inputs/Output, I see spike in CPU usage and a electrs starts to complain about multiple not optimized queries. And depending on the count of Inputs/Outputs server hangs for a long time and Sparrow disconnects.

Regarding Fulcrum, I did 7 attempts to use it in last 2 years and it failed every single time during the initial sync or shorty after.
IMHO Until it is fixed, Fulcrum is not usable outside of dedicated and "plentiful" hardware.
While not ideal, Electrs it is the only alternative of aging ElectrumX for personal use and we should at least attempt to support it as Electrum does.

[PrinceOfEgypt]

Regarding Fulcrum, I did 7 attempts to use it in last 2 years and it failed every single time during the initial sync or shorty after. IMHO Until it is fixed, Fulcrum is not usable outside of dedicated and "plentiful" hardware. While not ideal, Electrs it is the only alternative of aging ElectrumX for personal use and we should at least attempt to support it as Electrum does.

I've been through the same exercise as you, and I ended up using fulcrum. It's currently running on a raspberry pi 4 that also runs bitcoin core, lnd, as well as several websites and other tools, and fulcrum works just fine.

I was not able to get electrs to work with sparrow even when running it on my most powerful machine. It just isn't designed to handle batched rpc calls or large wallets well. you can also see @craigraw 's extensive writeup of it here: https://sparrowwallet.com/docs/server-performance.html

I ran the initial sync on a more powerful machine then copied the data over to an ssd connected to the pi, and I have a cron job that stops fulcrum and backs up the data once a week just in case, but so far (knock on wood) I have not had a single issue with fulcrum. I'd be happy to share more details of my setup if you want.

[eugenesan]

Regarding Fulcrum, I did 7 attempts to use it in last 2 years and it failed every single time during the initial sync or shorty after. IMHO Until it is fixed, Fulcrum is not usable outside of dedicated and "plentiful" hardware. While not ideal, Electrs it is the only alternative of aging ElectrumX for personal use and we should at least attempt to support it as Electrum does.

I've been through the same exercise as you, and I ended up using fulcrum. It's currently running on a raspberry pi 4 that also runs bitcoin core, lnd, as well as several websites and other tools, and fulcrum works just fine.

I was not able to get electrs to work with sparrow even when running it on my most powerful machine. It just isn't designed to handle batched rpc calls or large wallets well. you can also see @craigraw 's extensive writeup of it here: https://sparrowwallet.com/docs/server-performance.html

I ran the initial sync on a more powerful machine then copied the data over to an ssd connected to the pi, and I have a cron job that stops fulcrum and backs up the data once a week just in case, but so far (knock on wood) I have not had a single issue with fulcrum. I'd be happy to share more details of my setup if you want.

I've read Craig's articles a few times and am aware of the limitations of Electrs.

My issue with Fulcrum is that I need it to run on a laptop and be able to sync it periodically when there is a suitable network connection. Unfortunately Fulcrum in it's current state cannot be trusted to be able to stop without DB corruption and dealing with DB backup/restore every time I need to sync is undue overhead I'd like to avoid.

[PrinceOfEgypt]

My issue with Fulcrum is that I need it to run on a laptop and be able to sync it periodically when there is a suitable network connection. Unfortunately Fulcrum in it's current state cannot be trusted to be able to stop without DB corruption and dealing with DB backup/restore every time I need to sync is undue overhead I'd like to avoid.

How are you running fulcrum? is it running on Linux, Windows, or MacOS? it will work best if you run it on linux and create a systemd service for it so it can be gracefully started / stopped.

FWIW I'll also point out that running any electrum service periodically isn't really useful - the whole point of an electrum server is that it is always watching / indexing the blockchain so you wallet can easily find transactions. You may want to consider dedicating some hardware to it or connecting directly to your node from sparrow if you can't dedicate hardware to constantly run the electrum server.

[eugenesan]

My issue with Fulcrum is that I need it to run on a laptop and be able to sync it periodically when there is a suitable network connection. Unfortunately Fulcrum in it's current state cannot be trusted to be able to stop without DB corruption and dealing with DB backup/restore every time I need to sync is undue overhead I'd like to avoid.

How are you running fulcrum? is it running on Linux, Windows, or MacOS? it will work best if you run it on linux and create a systemd service for it so it can be gracefully started / stopped.

FWIW I'll also point out that running any electrum service periodically isn't really useful - the whole point of an electrum server is that it is always watching / indexing the blockchain so you wallet can easily find transactions. You may want to consider dedicating some hardware to it or connecting directly to your node from sparrow if you can't dedicate hardware to constantly run the electrum server.

I am running on Linux and my setup served me well until recent deprecation of Python-RocksDB.
The other DB engine (LevelDB) ElectrumX is using is extremely slow that's why I migrated to Electrs.

Regarding "graceful" termination of Fulcrum, systemd does nothing special, it just sends kill -15 and Fulcrum breaks the DB from time to time regardless of how was stopped.

I'd argue that periodic sync makes a lot of sense in some situations.
For example if electrum server is needed when traveling frequently or without reliable network/power at home.
Imaging your dedicated server stops responding in the middle of a 2 week vocation and there is no one at home to diagnose/fix it.

The only downside is the need to sync before actual usage but if periodic sync happens let's say once a day, sync time is between 5-15 minutes.

The advantages are:

1. Less network traffic, less SSD wear and less CPU/Battery power since it processes the chain in batches and doesn't run all the time
2. Cost and simplicity, all you need is additional 1TB on your laptop SSD
3. No need to maintain dedicated hardware and network/power at home
4. With "portable" setup I can open a laptop, sync and have full node anywhere and anytime
5. I can setup TOR node on the laptop to allow sharing of the node with other mobile devices

[PrinceOfEgypt]

Regarding "graceful" termination of Fulcrum, systemd does nothing special, it just sends kill -15 and Fulcrum breaks the DB from time to time regardless of how was stopped.

I'll have to disagree with you on this point - you can absolutely set up the unit to gracefully shut down fulcrum:

[Service]
ExecStart=/usr/local/bin/Fulcrum /data/fulcrum/fulcrum.conf
ExecStop=/usr/local/bin/FulcrumAdmin -p 8000 stop

As far as the rest, I guess if that's your use case then we just have different approaches to solve the same problem. I run my fulcrum server at home and access it over ssl or tor to use it with my laptop or apps on my phone.