actorforth/bch-devsuite

A quick and easy setup for Bitcoin Cash Node, SLP infrastructure and SmartBCH

bch-devsuite's Issues

Expose smartbch p2p port to external network when running testnet or mainnet

Port 26656 for smartbch needs to be externally exposed, without this the running node can only have outbound connections. With this port exposed the node might be able to serve other users behind NAT.

Do not reindex BCHN each time

When running Bitcoin Cash Node, the config file configs/bitcoincashnode.conf contains reindex=1

We should remove it, since this makes container restart incredibly slow. A reindex should only be used manually if the user knows the data got corrupted.

Support config file

User can supply a config file instead of having to go through init wizard, this will make deployment easier on their end.

Perform a full successful sync of smartbch(mainnet) and provide step-by-step instructions

Many users report issues with syncing smartbch on fresh install, someone needs to reproduce the issue and provide instructions.

Docs website

Create and deploy docs website for bch-toolkit

• generate from readme
• auto build and deploy using CI

Support SmartBCH mainnet

bch-devsuite can support deployment of smartbch

Init script improvement

Instead of using bash script, use Python with Jinja2 for templating and Typer (https://typer.tiangolo.com/) for command line interface builder. This changes should be compatibled with ./services start.

Proposed command:

Example

❯ ./bch-devsuite init
[?] Which components do you want to install?: 
 > X Local node (Bitcoin Unlimited node or Bitcoin Cash Node)
   X REST service (A rest.bitcoin.com-compatible or Bch-API)
   X SLP services (SLPDB, slpserve)

[?] Local node?: Bitcoin Unlimited Node
 > Bitcoin Unlimited Node
   Bitcoin Cash Node

[?] Network?: regtest
 > regtest
   testnet
   mainnet

[?] Please enter a username for the node RPC connection: tle
[?] Please enter a password for the node RPC connection: alpine
[?] Please enter number of initial blocks for regtest: 200
[?] REST Service?: bchrest
 > bchrest
   bchapi

SLPDB uses mongodb, a username and password is recommended.
[?] Please enter a username for mongodb: tle
[?] Please enter a password for mongodb: alpine
[?] docker network: bch-devsuite-network
[?] Exposed port? (Y/n): 
{
  "local_node": {
    "node": "bu",
    "network": "regtest",
    "rpc_conf": {
      "username": "tle",
      "password": "alpine"
    },
    "regtest_blocks": "200",
    "wait_time": 6
  },
  "rest_service": "bchrest",
  "slp": {
    "username": "tle",
    "password": "alpine"
  },
  "exposed_ports": true,
  "docker_network": "bch-devsuite-network"
}
Confirm? [y/N]: y

#########################
# VALIDATE DEPENDENCIES #
#########################
docker...✓✓
docker-compose...✓

##############
# SETTING UP #
##############
Creating docker network...ALREADY CREATED, SKIPPED
Setting up Bitcoin Unlimited node...✓
Generate docker-compose file...✓
Pull the Docker images...
Pulling bch-node      ... done
Pulling bchrest       ... done
Pulling opensight     ... done
Pulling mongodb-slpdb ... done
Pulling slpdb         ... done
Pulling slpserve      ... done
✓
Generating 200 initial blocks for regtest...
Creating bch-node ... done
[
  "20662918c049525e62ff72f84709981e6ea25ab6d4197256f94dfceb798ac2c2",
  ...,
  "1b52bf703cbdce2a8bcfbaabf0bfedf8599fff775143c51c9124337c882cb714"
]
✓
Stopping local node...Bitcoin server stopping
✓


Finished. Please use './bch-devsuite start' to run your new regtest node.

This version will be running on the local machine, not in Docker and we will improve by put it in a docker image later on.

Large transactions are not being broadcasted

Large transactions (tried with 43 and 47 UTXOs) are not being broadcasted by https://rest.bch.actorforth.org/#/rawtransactions/sendRawTransactionSingle.

The same transaction was broadcasted fine by this API: https://insomnia.fountainhead.cash/v1/tx/broadcast. See docs here.

bch-devsuite script Dockerization

To make it easier for users to adopt our devsuite, instead of having them install python3-venv, and pip install libraries, we ship everything in a docker image instead.

smartBCH does not sync anymore

re: #smartbch/smartbch#60

• Wait for official fix/release
• Apply the fix to devsuite

Proper docker image tag management

Create a landing page for DevSuite

Update the README's structure

I propose the following structure:

• Name of project (rebrand to DevSuite)
• Badges (Tests, container deployment, coding style and so on..)
• Getting started/quickstart
• Advanced usage (or just link to the docs after QuickStart)
• Architecture
• Contribution guide (maybe just link to CONTRIBUTING.md)

Add parameters to ./clean script to specify options

It would be quite useful to add argument parsing capabilities to the ./clean script so it can preform only specific operations that the users wishes. For example:

./clean wallet  # removes only user's wallet files
./clean node  # removes only the current bitcoind+nginx services
./clean all  # preforms every cleaning operation

Running the script without any arguments should display a usage message of possible arguments

Add support for using 'conda' python manager

Currently the included scripts expect that the virtualenv python package should be available. Anaconda manages its virtual environments in a slightly different way, and support can be added to the scripts so that users who prefer Anaconda can still run them.

Upgrade bitcoin unlimited to 2.0.0.0

Upgrade bitcoin unlimited to 2.0.0.0

Better shell interaction message

After setup is complete should have a message to show what endpoint to connect, summarize all the port.
Inspire by create-react-app

./clean does not cleanup smartbchd directory

Architecture docs

Make setup script option to deploy both rest.bitcoin.com and bch-api at the same time

Currently, the setup script does not support deploying both rest.bitcoin.com and bch-api at the same time.
Time estimate : 2 day

Support remote BCH RPC

Currently the devsuite need local BCH RPC deployed via devsuite script, it should be configurable to connect to any remote BCH node.

Dump and restore Bitcoin Cash Node

re: #22

Provide an automated way to bootstrap testnet and mainnet Bitcoin Cash Node from a block file instead of waiting on network.

Bitcoin Cash Node enables us to provide it with a block file bootstrap.dat which it uses for the initial sync instead of fetching the blocks over the network.

How to generate bootstrap.dat?

1. Prepare python scripts, specifically linearize-data.py and linearize-hashes.py from https://gitlab.com/bitcoin-cash-node/bitcoin-cash-node/-/tree/master/contrib/linearize
2. Patch linearize-data.py with this upstream commit
3. create linearize.cfg

# bitcoind RPC settings (linearize-hashes)
rpcuser=FIXME
rpcpassword=FIXME
host=<ip of bchd container>
port=<port>

max_height=<current height of blockchain, optionally round it to something nice>

netmagic=0b110907
genesis=000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943
input=/path/to/bch-toolkit/data/bitcoind/testnet3/blocks/
output_file=/tmp/bootstrap.dat
hashlist=/tmp/hashlist.txt

out_of_order_cache_sz = 100000000
rev_hash_bytes = False
file_timestamp = 0
split_timestamp = 0
debug_output = False

4. Generate hashlist

./linearize-hashes.py linearize.cfg > /tmp/hashlist.txt

5. Generate bootstrap.dat

./linearize-data.py linearize.cfg

This results in a 40GB file /tmp/bootstrap.dat

How to restore bootstrap.dat using?

1. Copy the file into correct place before starting bchd container

cp /tmp/bootstrap.dat data/bitcoind/testnet3/

2. Start bchd container
3. Once sync is complete, bchd renames the file to bootstrap.dat.old. This file is safe to delete.

It should take about 40 minutes(on my laptop) to sync testnet.

Action items.

• Figure out how and where do we store the dumps
• Figure out automation for restore process
• Figure out cleanup step to delete bootstrap.dat.old

Bitcoin Cash Node regtest Cli unable to update balance when transaction in mempool

Any RPC or CLI methods doesn't take into account what is in the mempool. It affects:
The node has spendzeroconfchange=1

Cant init without smartbch selected

use-case: Run devsuite without smartbch

In current master, if I build-config without selecting smartbch, I get following error

  File "/home/biggestfan/bch-devsuite/./bch-devsuite", line 349, in setup_docker_compose
    if config.smartbch.enabled:
AttributeError: 'NoneType' object has no attribute 'enabled'

Work-around is to select smartbch in wizard, but hand-edit the generated toml to say enabled=false in [smartbch] section.

The wizard should have generated a valid config in the first place.

Disable wallet functionality of bch node

Currently when I run bchn node, the wallet functions are enabled by default. This might lead me to expect that I can use this deployment for storing my BCH keys. If this is the intended usage then we should have ways for user to backup and preserve and provide their wallet file, if not then we should run the node with -disablewallet argument.

Upgrade bitcoincash node to v26.0.0

Upgrade bitcoincash node to v26.0.0

Make Electron Cash SLP wallet repository a git submodule pointing the private fork of it.

Upgrade smartBCH to v0.4.2

Smartbch recently released v0.4.2, bch-devsuite should update to use that version

Urgent: Upgrade Bitcoincash node and Bitcoin Unlimited nodes

Current versions of BCH nodes might stop working from 15th May onwards. Update the versions to

• BCH Unlimited release 1.10.0 with ElectrsCash 3.1
• v24.0.0

Also provide instructions for user to upgrade.

Dump and restore smartbch

re: #22

To make a dump (make sure the donor smartbchd is stopped)

sudo tar --exclude=priv_validator_state.json --use-compress-program=pigz -cvf /tmp/smartbch.tar.gz -C data/smartbchd data

To restore (make sure the recipient smartbchd is stopped and clean)

sudo tar --use-compress-program=pigz -xvf  /tmp/smartbch.tar.gz -C data/smartbchd/

BitcoinUnlimited cant sync testnet

The release version 1.9.2 has bug relating to checkpoints, but upstream dev branch has a fix.

Dump and restore SLPDB

re: #22

SLPDB is relatively straightforward

Need to run these from a machine/container that has mongodb installed, and can talk to mongodb-slpdb.

We should stop slpdb and slpserve containers prior to doing a restore. Not sure if there is any need to stop these while dumping.

Dump

mongodump -d slpdb_test --archive=/tmp/slpdb_test.mongo.gz --authenticationDatabase admin --gzip mongodb://mongouser:mongopass@mongodb-slpdb:27017

Restore

mongorestore  --drop  --authenticationDatabase admin --archive=/tmp/slpdb_test.mongo.gz --gzip   mongodb://mongouser:mongopass@mongodb-slpdb:27017

Potential security vulnerabilities in docs dependency

There is 2 package that needs to be upgraded.

• 1. markdown-it
• 2. nanoid

SmartBCH support Testnet

Dump and restore Bitcoin Unlimited

re: #22

Commands below require pigz to be installed. If its not then remove --use-compress-program=pigz from the tar commands and add z argument.

Generate dump

BU node must be stopped

sudo tar --exclude=debug.log --exclude=mempool.dat --exclude=wallet.dat --exclude=.lock --exclude=peers.dat --exclude=orphanpool.dat --use-compress-program=pigz -cvf /tmp/bu.tar.gz -C data/bitcoind testnet3

Restore dump

Put the files in the directory before creating the container

mkdir -p data/bitcoind/testnet3
tar --use-compress-program=pigz -xvf  /tmp/bu.tar.gz -C data/bitcoind/

Customizable data directory for blockchain data

Right now we hard code blockchain data storage directory.

resource specification

Adding documentation on how what resource they should expect and also the load test result.

Add ability to generate dumps of node, slpdb and fulcrum

Currently users of bch-toolkit need to wait long times (often several days) to fully sync node+slpdb to mainnet. We should provide dumps so that the bootstrap process can be made faster.