epiccurious/bitcoin-core-node-builder

Spin up a secure Bitcoin Core node with ease.

bitcoin-core-node-builder's Issues

Add acknowledgement for yeti.Bash

https://github.com/BenWestgate/yeti.Bash

offloadcoredata.sh should verify that RPC is running

In certain edge cases, such as when replaying blocks / rolling forward, bitcoin-qt is running but bitcoin-cli calls fail hard.

$ bitcoin-cli stop
error code: -28
error message:
Loading block index...

One option is to use bitcoin-cli -rpcwait stop or -rpcwaittimeout={num_seconds}

Another option is to check for an stderror error response from bitcoin-cli stop and handle the error with a clear message to the user and an exit 1.

Add a file header

https://google.github.io/styleguide/shellguide.html#file-header

Don't line break for PRESS ANY KEY messages

Use echo -en instead of echo -e

Offloading blocks/index/ and chainstate/ should be done intelligently

blocks/index/ contains an index of the validated blocks and is 113 megabytes.

chainstate/ contains the UTXO set derived from the block data and is 3.6 gigabytes.

When offloading Bitcoin Core data, these directories should be synced so only the diff is copied, rather that a full delete and copy.

Follow Google's shell style guide

https://google.github.io/styleguide/shellguide.html

This issue will not be part of the MVP milestone due to the significant refactoring and subsequent testing.

Improve the "syncing blockchain" message

Right now, it's 6 lines long including a blank line.

Shrink it to something 3-4 lines long.

After IBD completes, exit without user interaction

Remove read -rsn1 before exit 0 at the end of the script.

Make the path to the bitcoin binaries a variable

Stop hard-coding the path to the bitcoin binaries all over the script.

Instead, hard core it once at the start of the script.

Use shfmt to improve the formatting

offloadcoredata.sh fails hard when running out of free space on the target disk

Check at the start of the script that the disk usage du in the datadir blocks/ + chainstate/ is less than the disk free space df in the target directory.

Add option for compiling dependencies from source

[ -d build_dependencies ] || mkdir build_dependencies/
cd build_dependencies/

## INSTALL WGET

curl -O https://ftp.gnu.org/gnu/nettle/nettle-3.9.1.tar.gz
#curl -O https://ftp.gnu.org/gnu/nettle/nettle-2.5.tar.gz
tar -xf nettle-3.9.1.tar.gz
cd nettle-3.9.1/
./configure --enable-mini-gmp
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
sudo rm -rf nettle-3.9.1/
rm nettle-3.9.1.tar.gz

https://ftp.gnu.org/gnu/nettle/nettle-2.5.tar.gz

curl -O https://ftp.gnu.org/gnu/gnutls/gnutls-3.1.5.tar.lz
tar -xf gnutls-3.1.5.tar.lz
cd gnutls-3.1.5/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../

curl -O https://ftp.gnu.org/gnu/wget/wget-1.21.4.tar.lz
tar -xf wget-1.21.4.tar.lz
cd wget-1.21.4/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../


## INSTALL JQ

curl -O https://ftp.gnu.org/gnu/autoconf/autoconf-2.71.tar.xz
tar -xf autoconf-2.71.tar.xz
cd autoconf-2.71.tar.xz/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf autoconf-2.71/
rm autoconf-2.71.tar.xz


curl -O https://ftp.gnu.org/gnu/automake/automake-1.16.5.tar.xz
tar -xf automake-1.16.5.tar.xz
cd automake-1.16.5
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf automake-1.16.5/
rm automake-1.16.5.tar.xz

curl -O https://mirrors.ibiblio.org/gnu/libtool/libtool-2.4.6.tar.xz
tar -xf libtool-2.4.6.tar.xz
cd libtool-2.4.6/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf libtool-2.4.6/
rm libtool-2.4.6.tar.xz 

git clone https://github.com/jqlang/jq.git
cd jq/
git checkout $(git tag | grep -v rc | sort -V | tail -1)
git submodule update --init
autoreconf -fi
CPPFLAGS=-D_REENTRANT ./configure --with-oniguruma=builtin --disable-maintainer-mode
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf jq/

## INSTALL GPG TOOL

curl -O https://gnupg.org/ftp/gcrypt/gpgrt/libgpg-error-1.46.tar.bz2
tar -xf libgpg-error-1.46.tar.bz2
cd libgpg-error-1.46/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf libgpg-error-1.46
rm libgpg-error-1.46.tar.bz2

curl -O https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.9.4.tar.bz2
tar -xf libgcrypt-1.9.4.tar.bz2
cd libgcrypt-1.9.4/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf libgcrypt-1.9.4/
rm libgcrypt-1.9.4.tar.bz2

curl -O https://gnupg.org/ftp/gcrypt/libassuan/libassuan-2.5.5.tar.bz2
tar -xf libassuan-2.5.5.tar.bz2
cd libassuan-2.5.5/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf libassuan-2.5.5/
rm libassuan-2.5.5.tar.bz2

curl -O https://gnupg.org/ftp/gcrypt/libksba/libksba-1.6.3.tar.bz2
tar -xf libksba-1.6.3.tar.bz2
cd libksba-1.6.3/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf libksba-1.6.3/
rm libksba-1.6.3.tar.bz2

curl -O https://gnupg.org/ftp/gcrypt/npth/npth-1.6.tar.bz2
tar -xf npth-1.6.tar.bz2
cd npth-1.6/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf npth-1.6/
rm npth-1.6.tar.bz2

curl -O https://gnupg.org/ftp/gcrypt/gnupg/gnupg-2.4.2.tar.bz2
tar -xf gnupg-2.4.2.tar.bz2
cd gnupg-2.4.2/
./configure
make -j "$(($(sysctl -n hw.physicalcpu) + 1))"
sudo make install
cd ../
rm -rf gnupg-2.4.2/
rm gnupg-2.4.2.tar.bz2

user1s-MacBook-Pro:testdir user1$ jq --version
jq-1.6-dirty
user1s-MacBook-Pro:testdir user1$ gpg --version
gpg (GnuPG) 2.4.2
libgcrypt 1.9.4
Copyright (C) 2023 g10 Code GmbH
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /Users/user1/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Add a line break after pressing any key

Use master branch

Check for bitcoin binaries with the appropriate version

User should be able to re-run the script from any state and not have to repeat tasks (e.g. extracting the tarball).

Script should be able to handle recovering from any error that Bitcoin Core can recover from.

Don't require user confirmation to disable sleep

Turn the Bitcoin Core version into a variable

Build the URL dynamically based on the version number

Clarify in the README which files are temporary vs extract vs data directories

Check the disk free space

For now, hard code an arbitrary 600 GB as the minimum amount of disk free space.

If the user has less than 600 GB, offer to set a prune, with a default value of half the free space.

Add a condition if their system is below the minimum pruned block/ and chainstate/ size.

offloadcoredata.sh should verify that hardcoded paths exist

Ran into an issue where my data_directory_target was pointing to the wrong directory.

The first thing the script should do is check if the source and target paths are valid / exist.

If either the source or target are invalid, exit 1 the script.

Check for active internet

Try to ping 1.1.1.1.
If it fails, run the following:

• ping 1.0.0.1
• ping 8.8.8.8
• ping 8.8.4.4

Then test the DNS with ping cloudflare.com
If it fails, run the following:

• ping google.com
• ping wikipedia.com
• ping bing.com

Then try to access BitcoinCore.org:

• ping bitcoincore.org

Either of the stages fails, exit 1 with a relevant "no internet" error

Display sync progress as a percentage

Instead of displaying sync progress as 0.000000001, use 0.0000001%

The Linux user running Bitcoin Core should not have sudo permissions

We are currently launching bitcoin-qt as an administrator with super user permissions.

Need to create a separate user like "bitcoin" that has restricted permissions.

Also lock down what ports that user can communicate on.

Determine which parts of this guide video are relevant for all general use cases and if we want to use the admin to control services to control the users that control the binaries: https://m.youtube.com/watch?v=_Hrnls92TxQ

On low-RAM systems, set Bitcoin Core to run with reduced memory

Changes that will have little impact to performance:

1. After the IBD completes, reduce dbcache
2. Reduce the mempool with maxmempool=50
3. Possibly send a message to the user to avoid enabling listening mode
4. On Linux, set MALLOC_ARENA_MAX=1

If reboot is required after dist-upgrade, auto-start on boot

Relates to #1

Use either a service or a cron @reboot job. Need to understand more about the pros and cons of each option.

Wait for bitcoin-qt to stop before offloading the data directory

offloadcoredata.sh should shut down bitcoin-qt and wait for the process to stop before transferring data

Add optional Tor support

The node will keep the same tor hidden service (.onion address) when the system is rebooted.

Add optional additional hardening instructions to generate a new address after reboot.

shellcheck issue on offloadcoredata.sh, line 59

Issue raised by shellcheck. Need to come up with a solution to clear this.

user1@EPCMZXDK-ThinkPad:~/Documents/GitHub/bitcoin-core-node-builder$ shellcheck offloadcoredata.sh

In offloadcoredata.sh line 59:
rsync -ptgouq --partial "$blocks_source"/$highest_{blk,rev}_dat "$blocks_target"/
                                         ^-------^ SC2154 (warning): highest_ is referenced but not assigned.
                                         ^-------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
rsync -ptgouq --partial "$blocks_source"/"$highest_"{blk,rev}_dat "$blocks_target"/

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- highest_ is referenced but not as...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...

Clean up Terminal output for offloadcoredata.sh

Instead of echo'ing during every file copied, implement an updating "% complete" output.

For example,
1% complete
becomes
2% complete
on the same line.

https://stackoverflow.com/questions/11283625/overwrite-last-line-on-terminal

Add Bitnodes to Acknowledgements

For the bash script https://bitnodes.io/install-full-node.sh

Creator: https://github.com/ayeowch/

Gather dynamic list of Tor nodes

Planning to optionally disable clearnet and go Tor/I2P only after initial block download.

• Enable Tor + clearnet from the start
• Periodically, maybe every hour, scan debug.log for references to .onion addresses that we successfully connected to. Append the results to tor_peers in the Bitcoin Core data directory.
• When the initial block download completes, shut down Core, disable clearnet, and restart Core. Rely on tor_peers to bootstrap the initial connection via addnode.

Script throws an error is bitcoin core is closed during IBD

Sync progress:          0.03201991927044684
Blocks left to sync:    525726
Current chain tip:      Nov  8 06:42:01 AM EST 2013

Estimated size on disk: 13GB
Estimated free space:   839GB

Close this Terminal window by clicking on the "X".
This screen will refresh in 10 seconds...........error: timeout on transient error: Could not connect to the server 127.0.0.1:8332

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.

Identify the pre-headers and headers states

Syncing pre-headers occurs when:

• chain="main"
• initialblockdownload=true
• blocks=0
• headers=0
• time=1231006505 (the hard-coded genesis block)
• mediantime=1231006505 (the hard-coded genesis block)
• size_on_disk=293

Syncing headers occurs when:

• chain="main"
• initialblockdownload=true
• blocks=0
• headers>0
• time=1231006505 (the hard-coded genesis block)
• mediantime=1231006505 (the hard-coded genesis block)
• size_on_disk=293

Syncing the chain occurs when:

• chain="main"
• initialblockdownload=true
• blocks=0
• headers>0
• time>1231006505 (the hard-coded genesis block)
• mediantime>1231006505 (the hard-coded genesis block)
• size_on_disk>293

Install dependencies but not recommended packages

sudo apt install --no-install-recommends packagename

Include current blocks/ and chainstate/ directories in free space calculation

This ticket is pending the resolution of the "check disk space" issue/PR (#22).

Interesting scenario:

1. Start with 625 GB of free space and sync unpruned.
2. Sync the chain. There is now, say, 100 GB of free disk space.
3. Re-running node builder will incorrectly set a prune to 40 GB / 40%. Bitcoin Core will incorrectly delete 60% of the blocks.

When comparing the free space amount vs. the required disk space, subtract the disk usage in MiB of blocks/ and chainstate/.

Add a doc for testing changes to Node Builder code

Move support files out if root directory

Clean up the root directory so that the only files at that level are .gitignore, LICENSE.md, README.md, and nodebuilder.sh.

• Move support scripts to utils/ directory.

• Move bitcoin image to img/ directory

Improve error handling

https://betterprogramming.pub/best-practices-for-bash-scripts-17229889774d

Add acknowledgements section

Add a section to the README thanking other projects/guides that we used for inspiration.

• Yeti
• Parts 1 and 2 of Stop and Decrypt's guide
• find out who wrote the Tor docs for Bitcoin Core
• Youtube videos on Core from 402 Payment Required

Show disk used in MiB when less than 1 GiB

For nodes running with prune=550, the "estimated size on disk" metric is stuck at 0GB.
When the size is less than 1024 MiB, don't show in GiB because bash uses integers, not float.

Add line break after sleep

After the sleep, if the IBD has finished, the Terminal output appears on the same line as the sleep output.
Print a line break after sleeping

Reboot if required after dist-upgrade

https://askubuntu.com/questions/921541/how-to-determine-if-a-restart-is-needed-on-my-ubuntu-server#921546

Check if binary sha256sum is on PATH

If sha256sum exists, use sha256sum.

If shasum exists, use shasum -a 256.

If neither exists, exit 1 from the script.

https://github.com/babashka/babashka/pull/1108/files#diff-1e142e6277b12b7e1110478a24caee8f006a9349e86970c890203d6266209463R116-124

Use a variable for the guix.sigs clone directory

Speed up the IBD by increasing dbcache

Check if the user's system will support in increased dbcache

The risk of increasing dbcache too much is creating system instability and rolling blocks, so need to understand this issue better before making a change.

For example, if a user's system has 16GB and there are 6GB of RAM free, setting dbcache to 2 GB shouldn't be a problem. But if the user's system has 16GB and there are 2 GB free, setting dbcache to 2 GB will cause problems.

Need to verify checksum signatures

The process for verifying checksum signatures recently changed.

There is no contrib/builder-keys directory under the bitcoin/bitcoin.git repo.

That directory is now in the bitcoin-core/guix-sigs.git repo.

Additionally, there is no keys.txt file, only *.gpg files for each signer.

Need to figure out this new process, which is not yet documented on bitcoincore.org/en/download.

Use Bitcoin Core 25.0

Add desktop shortcut

Need two shortcuts on the desktop:
1. Relaunching the script (in case of power failure, etc.)
2. Running Bitcoin Core

Add Bitcoin to the Ubuntu 'Show Applications' list

Question on AskUbuntu

Question on SuperUser

shellcheck: -eq treats this as a variable on nodebuilder.sh, line 97

The match should be done with = or with ==, rather than -eq since it's a string match not arithmetic.

user1@EPCMZXDK-ThinkPad:~/Documents/GitHub/bitcoin-core-node-builder$ shellcheck nodebuilder.sh 

In nodebuilder.sh line 97:
while [[ $ibd_status -eq "true" ]]; do
                         ^----^ SC2154 (warning): true is referenced but not assigned.
                         ^----^ SC2309 (warning): -eq treats this as a variable. Use = to compare as string (or expand explicitly with $var).

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- true is referenced but not assign...
  https://www.shellcheck.net/wiki/SC2309 -- -eq treats this as a variable. Us...