fusion44/blitz_api

A management backend for the RaspiBlitz project written in Python / FastAPI

Blitz API

A management backend for bitcoin and lightning node operators written in Python with FastAPI.

Beta Disclaimer

This software is still considered BETA and may contain bugs. Don't expose it to the open internet or use with a lot of funds.

• Blitz API
  • Beta Disclaimer
  • Configuration
    • Dependencies
  • Installation
    • Linux / macOS
    • Windows
  • Run the application
    • Linux / macOS
    • Windows
  • Development
    • Using the nix package manager
    • Installation
    • Sync changes to a RaspiBlitz
    • Debugging code running on a remote machine via VSCode
      • Prepare local machine
      • Prepare the RaspiBlitz
    • Unit / Integration testing
      • Run the tests with pytest
      • Run tests and generate a coverage
    • Client libraries
      • Generating client libraries
    • Before you commit
    • Swagger / OpenAPI
    • Useful cURL commands to test the API
  • Acknowledgements

Configuration

Create a .env file with your bitcoind and lnd configuration. See the .env_sample file for all configuration options.

Dependencies

• Python in version 3.7
• Redis
• Polar If you need an easy option to run a simple bitcoind & lnd client

Installation

⚠️ To setup a development environment for BlitzAPI skip to the Development section.

Linux / macOS

make install

or

python -m pip install -r requirements.txt

Windows

py -m pip install -r requirements.txt

Run the application

Linux / macOS

make run

or

python -m uvicorn app.main:app --reload

Windows

py -m uvicorn app.main:app --reload

Development

It is recommended to have python-poetry installed.

From within the blitz_api folder open a poetry shell via:

poetry shell

(To exit the poetry shell use: exit)

Using the nix package manager

Blitz API provides a Nix Flake file to create a development environment. Execute nix develop (make sure you have flakes enabled) to enter the environment.

In this environment a hidden folder .venv is created to install the python dependencies locally. If pyright can't find these dependencies create the following file in the root folder of the project:

{
  "venvPath": ".",
  "venv": ".venv"
}

Installation

poetry install
pre-commit install

or

make install-dev

If python dependencies have been changed it's necessary to freeze all requirements to requirements.txt:

poetry export -f requirements.txt --output requirements.txt

ℹ️ This will skip all dev dependencies by default.
This step is required to avoid having to install poetry for final deployment.

Sync changes to a RaspiBlitz

Create a file /script/sync_to_blitz.personal.sh (will be ignored by github) the SSH connection data to your RaspiBlitz.

localIP="192.168.178.61" sshPort="22" passwordA=""

Then you can run always make sync-to-blitz to copy your latest code over to your RaspiBlitz. The script automatically restarts the backend API with the new code on your RaspiBlitz and shows you the logs.

To test the backend API then call the SwaggerUI: http://[LOCALIP]/api/v1/docs - to call protected endpoints run the /system/login endpoint first with HTTP POST body:

{
  "password": "[PASSWORDA]"
}

and then copy the JWT Auth string returned to Authorize in the top section of the SwaggerUI.

You can also now test the RaspiBlitz WebUI against the API by running it locally on your dev laptop when you configure it to use the backend API of your RaspiBlitz.

Debugging code running on a remote machine via VSCode

To debug Python code that is running on another machine, like a RaspiBlitz, follow these steps.

Prepare local machine

• run make install-dev.
• open the .vscode/launch.json file and change the host to your remote machines IP

  •   "connect": {
        "host": "192.168.1.49",
        "port": 5678
      }

• open your .env_sample file and replace the line # remote_debugging=false with remote_debugging=true
  This is necessary because we're going to synchronize the local source with the remote node. The blitz_api service will be restarted on the remote node. Any changes to the `.env' file will be overwritten by the setup script on the Blitz. This script will use the `.env_sample` file as a base and fill it with data. This way we can trick the Blitz into enabling this setting every time we change something without the Blitz explicitly supporting it.
• make sure you follow the steps in the Sync changes to a RaspiBlitz section
• execute make sync-to-blitz
• Make sure to chose Attach in the RUN AND DEBUG windows of VSCode
• Hit F5 and voila you should be connected to your RaspiBlitz and debug code remotely

Prepare the RaspiBlitz

• SSH into the Blitz, and run sudo -i -u blitzapi
• cd blitz_api
• Finally run make enable-remote-debugging

Refer to this documentation to learn how to setup VSCode correctly.

Unit / Integration testing

Make sure to include tests for important pieces of submitted code.

Run the tests with pytest

make test

Run tests and generate a coverage

make coverage

This will run tests and generate a coverage html file in this folder: ./htmlcov

Client libraries

ℹ️ The client libraries live in an extra repository: https://github.com/fusion44/blitz_api_client_libraries

Generating client libraries

Install OpenAPI Generator and Java:

npm install @openapitools/openapi-generator-cli -g
sudo apt install default-jre

Clone https://github.com/fusion44/blitz_api_client_libraries next to the blitz_api folder.

make generate-client-libs

⚠️ The first run requires sudo as it must download a Java .jar file to the system npm package folder.

Before you commit

This project uses pre-commit to keep the source code structured. Please make sure to run either make pre_commit or pre-commit run --all-files. The CI pipeline will reject pull requests that fail this step. This step helps to ensures that the source code is formatted consistently and pull requests are as tidy as possible.

Swagger / OpenAPI

Once the API is running swagger docs can be found here:

http://127.0.0.1:8000/latest/docs

Useful cURL commands to test the API

curl -N -H "Authorization: Bearer JWT_TOKEN_HERE" http://127.0.0.1:8000/sse/subscribe

curl -N -H "Authorization: Bearer JWT_TOKEN_HERE" http://127.0.0.1:8000/v1/bitcoin/getblockchaininfo

curl -X POST -N http://127.0.0.1:8000/v1/setup/type/1

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"password":"12345678"}' \
  http://127.0.0.1:8000/system/login

Acknowledgements

Integrated Libraries:

• sse-starlette
• fastapi-versioning

blitz_api's Issues

SSE endpoint not properly versioned

See:
DeanWay/fastapi-versioning#56

Token expiry is in milliseconds, not seconds

https://github.com/fusion44/blitz_api/blob/main/.env_sample#L3 says expiry time in seconds.

Running a few tests, I conclude these are in fact milliseconds :)

Add App details to SSE messages

As a webUI I would like to have the information about the installed apps.

All Endpoints: add implementation for Core Lightning

@rootzoll important issue

Currently, the API only provides support for lnd. As a user I would like to have the webui handle c-lightning as well.

Add endpoint to receive funds on-chain

As a user I want to receive funds onchain so I can open some channels :)

Implement two factor authentication as an additional optional security layer.

More info

Round verification progress of SSE event "btc_info" to two digits

Bitcoin core calculates the blockchain verification by including unconfirmed transactions. This means that everytime there is a new transaction, a btc_info event is fired through the SSE channel. For clients this information is usually not useful beyond two digits.

Solution: round to two digits for SSE events.

The full floating point value can still be fetched through /bitcoin/get-blockhain-info

https://bitcoin.stackexchange.com/questions/93108/how-does-bitcoin-core-know-the-progress-of-blockchain-without-connecting-to-peer

Add endpoint to give me a list of recent transactions

E.g. like in the mock-backend

Add endpoint to fetch payment details

Implement /lightning/get-payment-details by payment_hash, payment_preimage or by a payment_request. This endpoint would return single payment request with all info available.

Usecase:
User lists all tx with "/lightning/list-all-tx" and wants to see the full details from one of them.

Let clients specify which messages should be pushed through the SSE stream

A new client can pass an array with message ids that will be included in the stream.
If no or an empty array is passed, all messages will be pushed unfiltered.

Probably requires #8 to be fixed first.

make install on OSX fails

Not able to install the requirements on a OSX system:

> blitz_api % make install
python -m pip install -r requirements.txt
Ignoring cffi: markers 'implementation_name == "pypy" and python_version >= "3.6"' don't match your environment
Ignoring colorama: markers 'python_version >= "3.6" and python_full_version < "3.0.0" and platform_system == "Windows" or platform_system == "Windows" and python_version >= "3.6" and python_full_version >= "3.5.0"' don't match your environment
Collecting fastapi-versioning@ git+https://github.com/fusion44/fastapi-versioning@master
ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
    fastapi-versioning@ git+https://github.com/fusion44/fastapi-versioning@master from git+https://github.com/fusion44/fastapi-versioning@master (from -r requirements.txt (line 151))
make: *** [install] Error 1

Implement /lightning/fee-report

Implement a way to get the current routing fees and routing forwarding history:

https://api.lightning.community/#feereport
https://api.lightning.community/#forwardinghistory

Add endpoint to unlock wallet

If the wallet is locked, I would like to be able to unlock it over the webUI.

Use "type" of MessageEvent for SSE Events instead of "lastEventId"

As discussed in Twitter DMs:

blitz_api sends the messages with an id which is then used in a MessageEvent as lastEventId(MDN documentation here.

In raspiblitz-web I'm differentiating between the type of the MessageEventthough, which not lets the web UI receive the messages, but it can't process them.

Changing the lines here works, but needs to be done for the "regular" events as well.

Add pre-commit hook to format source code

To avoid inconsistent source formatting add a pre-commit hook possibly using black or similar tools.

https://medium.com/gousto-engineering-techbrunch/automate-python-code-formatting-with-black-and-pre-commit-ebc69dcc5e03

Add endpoint to estimate fees

As a user I would like to know how much I need to pay for my on-chain transactions to get into the next X blocks to not overpay .

Add endpoint to list available apps

See mock-backend

Example Status Information

These are the key/value pairs in the raspiblitz.info that should be given by the backend to the frontend to run the setup dialogs on .... FYI @cstenglein & @fusion44

These are the relevant for the setup dialogs:

setupPhase='setup'
state=waitsetup
message='Please Login for Setup'
hddGotMigrationData=
online='1'
dnsworking=1
hddBlocksBitcoin=1
hddBlocksLitecoin=0

Those are not relevant for the setup process or frontend:

setupStep=0
baseimage=raspios_arm64
cpu=aarch64
localip='192.168.178.50'
fsexpanded=1
displayClass=lcd
displayType=
fundRecovery=0
hostname=
hddCandidate='sda'
board=
ramMB=
network=
chain=

Implement ideas from the minimal fastapi template

Especially token and session handling:
https://github.com/rafsaf/minimal-fastapi-postgres-template

Add endpoint to close a lightning channel

As a user I would like to close an existing lightning channel

list-all-tx: list correct state of invoices

As a user I would like to know if my invoice was expired. Currently it just displays "failed"

The listinvoices define the "settled" and the "state" for that.

https://api.lightning.community/#lnrpc-invoice

EDIT: if the amount paid was 0 mSat then it wasn't settled. So if the state was "failed" + 0 mSat amount then do I guess correctly that the invoice expired?

Finish implementation of /system/get-system-info

@rootzoll important issue

LND tlsCert as hex

At the moment the LND TLS cert in requested in the .env file as a text string. Because the cert file contains some escaped chars and line breaks I was not able to create a valid config from the raspiblitz install script.

blitz_api/.env

Line 27 in 66c5ea6

lnd_cert="\n-----BEGIN CERTIFICATE-----\ncertificate\n-----END CERTIFICATE-----\n"

It would be great if the TLS cert will be in HEX like the macaroon.

Build API for fetching Raspiblitz log

raspiblitz/raspiblitz-web#49

Add endpoint to pay on-chain

As a user I want to be able to transfer funds away from my RaspiBlitz.

Thus I would like to have an endpoint to send fund on-chain.

/bitcoin/sendpayment would come to mind, similar to /lightning/sendpayment

Break up lightning info object into multiple smaller objects

raspiblitz/raspiblitz-web#32

• Bitcoin API
• Lightning API
• Raspiblitz (System) API
• Push new objects via SSE to clients

Add endpoint to refresh token

As a user I don't want to be logged out when the token expires and I am still working in the UI.

Thus I would like to have a endpoint, so I can refresh the token before expiry.

Add endpoint to give me a list of mock transactions

Since #30 will take some more time, give me a list of mock transactions please :)

Example: https://github.com/cstenglein/raspiblitz-web/blob/master/backend-mock/transactions.js

Don't know if these fields are fine, will adapt to your solution 👍

Add Tests

• auth
• models
  • bitcoind
  • lightning
  • system
• repositories
  • ln_impl
    • clightning
    • lnd
  • apps
  • bitcoin
  • lightning
  • system
• routers
  • apps
  • bitcoin
  • lightning
  • setup
  • system
  • wallet
• main
• utils

blitz_api stuck at "Waiting for application startup" if redis is not installed

If redis is not installed, blitz_api waits forever(?) for it and it doesn't fail.

INFO:     Will watch for changes in these directories: ['/blitz_api']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [72907] using statreload
INFO:     Started server process [72909]
INFO:     Waiting for application startup.

Make shell_script_path more configurable

Currently it's more like hidden knowledge that you need to specify a path with shell_script_path where a config.scripts folder with a file called blitz.debug.sh in it.

See also my note here: https://github.com/cstenglein/raspiblitz-web#blitz-api

=> Make shell_script_path more configurable (maybe with an extra file property?)

Add endpoint to open a lightning channel

As a user I would like to open lightning channels to others.

Exception when shutting down application: 'NoneType' object is not callable

When shutting down the application with CTRL+C i get the following error stacktrace:

^CINFO:     Shutting down
INFO:     Finished server process [5244]
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/christophs/.local/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/routing.py", line 55, in app
    await response(scope, receive, send)
  File "/home/christophs/src/cloned/blitz_api/./app/sse_starlette.py", line 180, in __call__
    await run_until_first_complete(
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/concurrency.py", line 22, in run_until_first_complete
    (done, pending) = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 413, in wait
    return await _wait(fs, timeout, return_when, loop)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 529, in _wait
    await waiter
asyncio.exceptions.CancelledError
INFO:     ASGI 'lifespan' protocol appears unsupported.
Exception in callback _SelectorSocketTransport._call_connection_lost(None)
handle: <Handle _SelectorSocketTransport._call_connection_lost(None)>
Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 978, in _call_connection_lost
    super()._call_connection_lost(exc)
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 736, in _call_connection_lost
    self._protocol.connection_lost(exc)
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 117, in connection_lost
    self.on_connection_lost()
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/_handlers/http.py", line 42, in <lambda>
    on_connection_lost=lambda: connection_lost.set_result(True),
asyncio.exceptions.InvalidStateError: invalid state
Exception in callback handle_http.<locals>.retrieve_exception(<Task cancell...erver.py:104>>) at /home/christophs/.local/lib/python3.9/site-packages/uvicorn/_handlers/http.py:56
handle: <Handle handle_http.<locals>.retrieve_exception(<Task cancell...erver.py:104>>) at /home/christophs/.local/lib/python3.9/site-packages/uvicorn/_handlers/http.py:56>
Traceback (most recent call last):
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/server.py", line 107, in handler
    await handle_http(
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/_handlers/http.py", line 87, in handle_http
    await connection_lost
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/_handlers/http.py", line 58, in retrieve_exception
    exc = task.exception()
asyncio.exceptions.CancelledError
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-104' coro=<EventSourceResponse.stream_response() done, defined at /home/christophs/src/cloned/blitz_api/./app/sse_starlette.py:192> exception=TypeError("'NoneType' object is not callable")>
Traceback (most recent call last):
  File "/home/christophs/src/cloned/blitz_api/./app/sse_starlette.py", line 210, in stream_response
    await send({"type": "http.response.body", "body": b"", "more_body": False})
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/exceptions.py", line 68, in sender
    await send(message)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 148, in send
    await send(message)
  File "/home/christophs/.local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 156, in _send
    await send(message)
  File "/home/christophs/.local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 490, in send
    self.on_response()
TypeError: 'NoneType' object is not callable
INFO:     Stopping reloader process [5242]

Fix POST endpoints to receive args via json body, not via path query

For example: /lightning/send-payment

Add endpoint to change password while logged in

See https://github.com/cstenglein/raspiblitz-web/blob/master/backend-mock/index.js#L87-L90

Add endpoint to decode invoice

As a user I would like to see the amount etc. of a LN invoice before paying it.

Thus I would like to have an REST endpoint (no SSE event needed) to decode the LN invoice and show me the relevant information.

Send a complete current state through the SSE stream when a new client connects

Currently, only changes are sent and the client. The initial state must be fetched via the REST API.

A way to address each client individually is necessary.

Probably requires #8 to be fixed first.

Improve handling when multiple clients connect to the SSE stream

Currently, when multiple clients connect to the SSE stream, messages are send alternating but never all to all clients. All recurring tasks on the server are slowed too. Find the reason why this happens.

Version the API

Add API version to the server.

Should probably stick to major versions only to simplify the work necessary by clients.

FastAPI Plugin to automate this: https://pypi.org/project/fastapi-versioning/

Crash when processing payment request with route hints

Any handling of route hints with LND is crashing the backend.

Wrong balance for LN channel with "wallet_balance" message / lightning/get-balance endpoint

I got the following channel:

The endpoint & the wallet_balance message gives me the following result:

{
  "onchain_confirmed_balance": 742363,
  "onchain_total_balance": 742363,
  "onchain_unconfirmed_balance": 0,
  "channel_local_balance": 140950000,
  "channel_remote_balance": 100000000,
  "channel_unsettled_local_balance": 0,
  "channel_unsettled_remote_balance": 0,
  "channel_pending_open_local_balance": 0,
  "channel_pending_open_remote_balance": 0
}

140950000 => 140_950_000 are three zeros too much. for channel_local_balance & channel_remote_balance

The onchain_confirmed_balance is correct though:

Implement setting to set API to a read only mode

Users might want to disable all send funds functionalities for security reasons. This mode will then disable all send funds endpoints.

/system/login: expiry time is not a integer, but a float

I just logged in and decoded the token. In my .env I set the expiry to 30000s (30k)

The payload is the following:

{
  "user_id": "admin",
  "expires": 1637621221.8622634
}

As a developer I want a specific unix time to log out and not have to deal with a floating number.
Suggestion: round down :)

Lock dependencies via pip-compile

Pip can lock dependencies to a specific version with a specific hash. This ensures that the dev environment remains deterministic.
More Info
Example from Specter Wallet

Add endpoint / field to inform about the locked wallet

On startup when auto-unlock is not enabled, the Lightning wallet is locked.

I would like to have an extra endpoint or a field in ln_info_lite to see if the wallet is locked.

blitz_api doesn't respect the "chain" in raspiblitz.conf

changed "chain" from "main" to "test" in /mnt/hdd/raspiblitz.conf

blitz_api doesn't allow me to unlock the testnet wallet and only delivers mainnet information, not the testnet one

Reinstalled blitz_api after chain change & restart => shows new alias of testnet wallet, but still shows chain "mainnet"

Maybe likely a redis issue?

"ln_info_lite" does deliver "True" instead of "true" (invalid JSON)

See screenshot below

"system_info" does send invalid JSON in "health" and "health_messages" field

health and health_messages don't send a string, but instead a enum value wrapped in diamonds <> with a string in it, which is not valid JSON.

health field:

health_messages is the same:

lightning/list-all-tx: "failed to connect to all addresses" while node is syncing

While the node is syncing, lightning/list-all-tx returns a 500 status code with the detail message "failed to connect to all addresses".

Messages on connection are sent with single quotes (not valid JSON)

The messages that are sent on connecting are in single quotes: