electrumsv/electrumsv-sdk

Libraries to facilitate development and testing of ElectrumSV and applications based on it

ElectrumSV - Lightweight Bitcoin SV client

Licence: MIT
Maintainers: Neil Booth, Roger Taylor, AustEcon
Project Lead: Roger Taylor
Language: Python (requires Python 3.9 later than 3.9.13. 3.10 and 3.11 not officially supported)
Homepage: https://electrumsv.io/

Getting started on Linux/MacOS

ElectrumSV is a Python-based application forked from Electrum Core.

If you are running from the Github repository, you are advised to use the latest release branch, which at this time is releases/1.3. The develop branch is used for the latest development changes and is not guaranteed to be as stable, or to have guaranteed long term support for some of the more advanced features we may have added and later remove. The master branch is frozen, out of date and will be overwritten by develop evenutally.

Ensuring you have at least Python 3.9.13

The ElectrumSV builds are created using Python 3.9.13 because these are the last release for Python 3.9 that the Python development team do binary releases for. This is the minimum allowed version of Python to use, we explicitly rule out running against earlier versions and we cannot guarantee later versions like 3.10 and 3.11 will work reliably due to breaking changes by the Python language developers.

You need to ensure you have Python 3.9.13 or later, the following command should look like this:

$ python3 --version
Python 3.9.16

You can use pyenv to install Python 3.9.16. First install pyenv:

curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash

Edit your .bashrc file as described, and then ensure the changes are put into effect:

$ source ~/.profile

Now you can install Python 3.9.16 using pyenv:

$ pyenv install 3.9.16

If you encounter errors during that process, you can refer to the pyenv common problems.

At this point, you can make Python 3.9.16 the default Python on your computer:

$ pyenv global 3.9.16

And you can check that your python3 version is indeed 3.9.16, by confirming the following command now looks like this:

$ python3 --version
Python 3.9.16

Ensuring you have at least Sqlite 3.31.1

ElectrumSV MacOS and Windows builds come with at least Sqlite version 3.31.1, but there are no Linux builds, and both Linux and MacOS users may wish to upgrade or make available the Sqlite version on their computer.

MacOS:

$ brew upgrade sqlite3
$ python3 -c "import sqlite3; print(sqlite3.sqlite_version)"
3.31.1

Linux:

$ python3 -m pip install -U pysqlite3-binary
$ python3 -c "import pysqlite3; print(pysqlite3.sqlite_version)"
3.31.1

You may see a different version displayed than 3.31.1, but as long as it is higher, this is fine.

Installing other dependencies

If you are running ElectrumSV from source, first install the dependencies:

MacOS:

brew install pyqt5
pip3 install --user -r contrib/deterministic-build/macos-py3.9-requirements-electrumsv.txt

Linux:

sudo apt-get install python3-pyqt5
pip3 install wheel
pip3 install cython==0.29.36
pip3 install --user -r contrib/deterministic-build/linux-py3.9-requirements-electrumsv.txt

Your should now be able to run ElectrumSV:

MacOS:

python3 electrum-sv

Linux:

python3 electrum-sv

You can also install ElectrumSV on your system. In order to do so, run the following command:

pip3 install . --no-dependencies

Problem Solving

If you choose to use Linux, you introduce complexity and uncertainty into the process. It is not possible to know all the unique choices you have made regarding it. The following tips may help work around problems you encounter.

Errors relating to "wheels"

If you encounter problems referring to wheels, make sure you have installed the wheel package:

pip3 install --user wheel

Errors relating to "libusb" installing the pip3 requirements

Install the following:

sudo apt-get install libusb-1.0.0-dev libudev-dev

Errors relating to "Python.h"

If you encounter problems referring to "Python.h", first check your Python version:

python3 --version

If it says "3.9", then install the following:

sudo apt install python3.9-dev

If it says a later version of Python, you should be able to figure out what to do.

Scanning QR codes

If you need to enable QR code scanning functionality, install the following:

sudo apt-get install zbar-tools

Getting started on Windows

The easiest way to run ElectrumSV on Windows, is to obtain an executable for the latest version from our website. This Git repository has a build-hashes.txt which should contain SHA-256 hashes for all our downloads. You can confirm that you have downloaded a valid file, by comparing it's SHA-256 hash to the hash we provide for the same file name.

You can also run from the Git repository directly, which is useful if you wish to customise or help us develop ElectrumSV.

You need to be sure that you are using a version of Python either 3.9.13 or higher. And that the version you are using has a version of Sqlite either 3.31.1 or higher. If you are for instance using a version of Python 3.8 that has a lower version of Sqlite, then update your Python 3.8 installation.

To run ElectrumSV from its top-level directory, first install the core dependencies:

py -3.9 -m pip install --user -r contrib/deterministic-build/win64-py3.9-requirements-electrumsv.txt

Then invoke it as so:

py -3.9 electrum-sv

You can also install ElectrumSV on your system. This will download and install most dependencies used by ElectrumSV. This is useful if you with to use the electrumsv Python library, perhaps for Bitcoin application development using ElectrumSV as a wallet server.

In order to do so, run these commands:

pip3 install . --no-dependencies

Extra development notes

Check out the code from Github:

git clone https://github.com/ElectrumSV/ElectrumSV
cd ElectrumSV

Run the pip installs (this should install dependencies):

pip3 install .

Create translations (optional):

sudo apt-get install python-requests gettext
./contrib/make_locale

Running unit tests (with the pytest package):

pytest electrumsv/tests

Running pylint:

pylint --rcfile=.pylintrc electrum-sv electrumsv

Running mypy:

mypy --config-file mypy.ini --python-version 3.9

Builds

Builds are created automatically for Git commits through the Azure Pipelines CI services which Microsoft and Github kindly make available to us.

The easiest way for you to create builds is to fork the project, and to link it to Azure Pipelines and they should also happen automatically. If you wish to look at the specific code that handles a given part of the build process, these will be referenced below for the various operating systems. To see how these are engaged, refer to the Azure Pipelines YAML files.

Source Archives

Run the following to create the release archives under dist/:

./contrib/make_source_archives.py

Mac OS X / macOS

See contrib/osx/.

Windows

See contrib/build-wine/.

electrumsv-sdk's Issues

Basic pushdata hash indexer

Basic miner ID implementation to run with the regtest node mining

...

Merchant payments payment page can better indicate payment event

As the current time, when an unpaid invoice payment page is displayed and the server marks the invoice paid and notifies the listening page, it just updates it with paid state as if it was visited later.

It should do something like show a "Payment received" or "Payment received ago" display of some kind and do whatever else makes sense.

The updated payment page to be paid:

Finish health check polling in status monitor

complete current model as per #57

reminder: Need to modify the electrumsv builtin plugin to accept an env var for changing the ElectrumX host (e.g. host.docker.internal)

ESV restapi - update the restapi so that the 'create_new_wallet' endpoint generates a random new seed

every time for testing of multiple instances

Add 32 bit wheel to release

As discussed - we will now do 32 bit windows, 64bit win, linux, mac

SDK CLI - the electrumsv cli interface for creating a new wallet works now so should use this internally

rather than the restapi (which requires the node + electrumx to be running) -

would greatly speed up resets and functional testing / azure pipelines etc...

Option1 (preferred for simplicity):

• add a flag (to electrumsv create_wallet <other options...> --no-password-check --walletpassword test )
  where --no-password-check means that it will pull in the --walletpassword rather than running 'getpass' to prompt the user to enter their password twice..
  electrumsv/electrumsv#532

Option2:

• work around it using Pexpect
  https://pexpect.readthedocs.io/en/stable/examples.html internally (to always set the password to "test")

Merchant payments payment page does not display pending expiration information

There is code to calculate when the invoice expires and to display and update it, but the expiry section is for some reason not getting displayed and updated.

Need a standardised approach to configuring plugins with both environment variables and commandline args

• It should be done in such a way that the env vars and cli args map onto each other (perhaps the env vars are all caps and cli args are all lower case (snake case)

and the SDK should have an automated way of producing a doc string describing the available configuration options on a per component basis (because every plugin will have totally different needs)

Merchant payments payment page has list display issues due to templates and dynamic population

The top of the recent list:

The top of the item list:

when electrumsv_node latest restapi change is pushed should add whatsonchain_api to CI/CD pipelines testing coverage

generic way of adding component-specific commandline flags and environment variables

Merchant payments page should show connectivity

Disconnection and reconnection to the websocket is only indicated in the page footer.

global_cli_flags can currently mutate globally which makes the code more complex than it should be

I'd like to assign the global_cli_flags to a named tuple as early as possible in the pipeline and pass it into each function call (namely the main plugin entrypoints)... which will alleviate any worries about mutating global state and causing bugs...

Given use of recursion, it feels like bugs are just waiting to happen due to this..

todo: install postgres directly using homebrew on MacOSX pipeline tests (currently doing a manual docker install as a workaround but it is very slow)

Need to add an "install" entrypoint to decouple installation from launch

This is most relevant for docker containers where you want to do all of the building / installing of packages in the image so that docker-compose up is snappy leaving a good impression for the user.

But the same goes for general use... if there is no need to reinstall packages, then the user should have the choice to skip it.

todo: find out why on Azure, the mAPI file log is empty (but is fine on linux and macos) - also find locally on windows

• have tried PYTHONUNBUFFERED=1 to promptly flush the logs but no luck

Need to make "reset" subcommand work for a single component at a time to match the "start" subcommand syntax

Avoid all subprocess calls that recurse back to the SDK calling itself on the commandline

Instead should be able to do all of these things programatically.

(This will make it much easier to manage sys.path and dynamic importing of plugins by avoiding loss of original calling context of the original process...)

Multiple instances of bitcoin nodes + reorg simulations / testing

Rough plan for this (mostly rehashing prior discussions)... Make it so that this is supported:

• electrumsv-sdk start --new --id=node1 node
• electrumsv-sdk start --new --id=node2 node
• electrumsv-sdk start --new --id=node3 node etc.
  (shouldn't be hard - this is how it is for electrumsv at present too):

We should also be able to reset each node via:

• electrumsv-sdk reset --id=node1
• electrumsv-sdk reset --id=node2
  ...

Then also modify the node->RPC subcommand i.e. (electrumsv-sdk node <RPC command> <optional args>)
So that it can be directed at each node instance (according to the --id flag for each respective node). So somthing like this:

electrumsv-sdk node --id=node1 generate 5
electrumsv-sdk node --id=node2 generate 6
electrumsv-sdk node --id=node3 generate 7

and other RPC commands to do with adding the other nodes as peers or banning them may come into play...

There might be two possible avenues for running the reorg simulation.

• If there's one ElectrumX instance, maybe we would want to shut it down (+/- reset the electrumx datadir) and start it back up again (but connected to a different node/RPC port)??
• if more than one ElectrumX instance paired to each node - maybe this provides a different flavor of reorg stress testing for the ElectrumSV wallet??

I think the former is closer to real-world conditions for us in that the ElectrumSV wallet has a main_server and notices the reorg via that pathway... So I'd favour that.

Merchant payments page footer needs polish

It currently has "XXX".

Merchant payments recent invoices list should monitor and update for expiration

Currently the expiring entries do not change state when the expiration time passes.

Standard set of test raw blocks and transactions that stress all edge cases

I have a fair idea of most of the pitfalls that need to be handled wrt indexing the chain.

Things like handling of malformed output scripts (which can legitimately happen where they do not comply with bitcoin script syntax rules). Or coinbase scripts do not get checked so can just find total junk in there... or weird stuff happens like the coinbase can make the txid be non-unique! (but only for coinbases as a special case)

But also the "happy path" of correctly parsing all pushdatas / opcodes.

So I think a set of hand-crafted raw blocks and raw transactions with expected output would be nice (but might be a bit tedious to make...)

I've often found myself wishing that something like this existed for a number of bitcoin-related projects.

For the wallet you really want a set of transactions that systematically map out all of the most common script types to stress all the edge cases too.

Merchant payments page table does not update row on server "invoice is paid" events

The web socket event should update the given table line without requiring a full table refresh.

get official SDK documentation hosted (readthedocs)

curio / trinket + dependencies failing to build wheels on windows azure...

As we are planning on replacing this with aiohttp anyway (for the status_monitor etc.) - it pushes us to complete this task sooner rather than later... at least for the status_monitor in the short-term

create_wallet - not sure how to suppress terminal output

ElectrumSV prints this:
Wallet saved in 'C:\Users\<username>\AppData\Local\ElectrumSV-SDK\component_datadirs\electrumsv\electrumsv1\regtest\wallets\worker1.sqlite'

and this:
New standard (bip32) account created for: 'C:\Users\<username>\AppData\Local\ElectrumSV-SDK\component_datadirs\electrumsv\electrumsv1\regtest\wallets\worker1.sqlite'

When I use all variations of

process = subprocess.Popen(line, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
                                   stdin=subprocess.DEVNULL)
process.wait()

And other subprocess commands... not sure how to do it at the moment.

I'm guessing it's because there are nested child processes being spawned at it can't control those? 🤷‍♂️

Update whatsonchain + whatsonchain_api plugins to take environment variables (for use inside of docker containers))

update docs for mAPI and provide a hyperlink to it in error messages

SDK CLI - make it so that if --id is new (not seen before) then --new flag is implied...

Remnants left over after pip uninstall

After running the electrumsv-sdk start command it clones the repositories to site-packages

But pip uninstall does not remove this so I am left with:

Not sure right at the moment how to deal with this other than perhaps making the first ever run of electrumsv-sdk purge all of this leftover cruft from prior installations... not ideal though because if I move those directories around in future it could get awkward.

I use the Manifest.in to include arbitrary things in site-packages...and seems to be cleaned up alright on pip uninstall...
But can't find any docs about removing things that are not present at the time of creating the build and dist for pypi...
in other words all of the extra stuff that gets generated at run-time of the SDK is not removed with a pip uninstall.

Maybe there is a way but haven't found any help for it yet.

Formalise the required set of APIs required for ElectrumSV operation

This may come from a variety of locations:

• Fee calclulation: Merchant API (delayed). REST.
• Broadcast a transaction: Merchant API (delayed). REST.
• Query paged pushdata hashes. REST.
• Blockchain headers. RPC of some sort.
• Transactions. REST.
• Merkle proofs. REST.

Basic merchant API for transaction broadcast / mining fees

allow switching of network for node, electrumx, electrumsv with --regtest --testnet, --scaling-testnet and --mainnet flags

I think a more generalized model for including this may need to have a generic way of adding component-specific commandline flags and environment variables (preferably coupling these two things together where command-line arguments always override environment variables).

Merchant server that can serve BIP270 invoices and process payments

...

stop unnecessary polling of status monitor

There is already a lock around the file that tracks state (component_state.json) so this file can be updated independently of http requests to/from the status monitor and will be much more efficient (will be a lot faster for starting/stopping components)...

The status_monitor can then be an optional component that only needs to be running if there is some other application (like a front-end website) that relies on this service (e.g. apps that are not SDK plugins cannot access the component_state.json file safely).

Potential bug - need to strictly couple datadir to --id. Currently there are possible edge cases where they could end up not matching

Sound lifetime management of components and up to date status monitoring information

The current for tracking the status of running components is (and will be when complete):

• a AppData/Local/ElectrumSV-SDK/component_state.json file (with a file lock) is updated -> ComponentState.Running
  on successful launch (determined by polling / health check)
• then the status monitor will continue polling the running components to ensure that ComponentState.Running is still true...
  if not then it would update component_state.json -> ComponentState.Failed for as long as the health check is failing. If the component comes online again then updates -> ComponentState.Running.
• However, if the component is explicitly stopped then component_state.json -> ComponentState.Stopped.
• Therefore the true status should be never any more than 1-2 seconds out of date at any given moment.

If the status_monitor is offline (it is optional). Then the ComponentState (Running / Failed / Stopped) are unreliable datapoints.

But the other contents of the component_state.json are always valid (such as the pid, component id, etc. etc.) and can be used to reliably stop / start / reset the components (the status monitor does not need to be online for this core functionality)

Alternative model as discussed with rt (Unsure if it is possible though).

• To have the status_monitor promoted to take on more responsibility. Namely, the launching of applications such that they can block in a thread (using subprocess.Popen() calls)...
• To make this workable we would need a way to pass environment variables to the subprocess.Popen() and a way to capture the return_code and stdout if the underlying server crashes or exits.
• Would also need a way to run in background OR spawn a new console window with the above still holding true
• Would also need a way to redirect the logs if running in the background.
• The advantage of this is that it allows immediate and robust feedback when a spawned application crashes and returns the traceback + update -> ComponentState.Failed immediately, rather than relying on (a potentially tenuous) polling / health check system over the network.

As discussed:

• We will stick with the current model for now but perhaps revisit this as at a later date.

electrumx, electrumsv and whatsonchain should error cleanly with nice error messages if git is not available and it's needed

failure to update status monitor on digital ocean server

Sometimes the status_monitor server (based on curio + trinket) starts thinking that requests to the update_status endpoint have a request.body that is empty and gives a json decode error... but the python requests.post() is definitely not sending an empty body...

I am having suspicions that this is a trinket / curio problem so have a low threshold for ditching curio/trinket in favour of the (much) more widely adopted aiohttp framework...

likely bug with --new flag detected on pipelines build:

https://dev.azure.com/electrumsv/ElectrumSV/_build/results?buildId=2672&view=results

Needs investigating / better testing at some point

mypy type checking

As discussed - need to add mypy type checking throughout codebase

More sophisticated electrumsv-sdk "stop" command

With the introduction of the StatusServer (and websocket) coming soon... I think the "stop" command needs to become more sophisticated to specify which ComponentType(s) it wants to stop.

By default (no args) it kills all processes that were started

I think instead it would leave the status server running (but kill the rest) until it is explicitly stopped via:

electrumsv-sdk stop status_server

or

electrumsv-sdk stop all

Currently the stop command just iterates through all of the ComponentTypes that it started and kills them all indiscriminately.

Instead each ComponentType (with pid attached) should be looked up and killed selectively depending on arguments to the "stop" command.

This allows the front-end to maintain the 'link' of being able to check the status (with the default usage pattern)

It should be possible to start multiple instances of ElectrumSV for an SDK run

• Maybe something like electrumsv-sdk start electrumsv --gui.
• All wallet data directories should be stored under the sdk data directory (now in AppData\Local\ElectrumSV-SDK or /home/.electrumsv-sdk).
• Each instance run should have a user-addressable name, whether the user provides it or it is allocated if none is given. This means that if they close a specific instance with it's wallet files and state, they should be able to start that instance again.

purge all useage of "sudo" wherever possible...

I found a workaround for the issue where I was unable to install the sdk without sudo priviledges:

By adding:

site.ENABLE_USER_SITE = "--user" in sys.argv[1:]

to the top of setup.py
ref: pypa/pip#7953

Whatever it takes really - need to find a way for obvious reasons. I can also remove the virtualenvironment setup for linux when this is solved (which was only there as a band-aid for this issue)

'node' command (for RPC requests) should allow "forcing" the host and port of the remote node to send requests to

this would render any --id flag irrelevant as a form of manual override. User must use one method or the other but not mix and match.

electrumsv-sdk node --host=host.docker.internal --port=18332

todo: assert that the status monitor is showing the expected process status in Azure

Explicitly enforce that if a component is single instance only - disallow --force-port option with error message.

SDK CLI - avoid starting and stopping ESV the first time merely to generate config files etc.

Instead could copy-paste the required files / create required directories (which should obviate a significant chunk of code and cut down on run time of pipelines tests.)