libbitcoin/libbitcoin-blockchain

Bitcoin Blockchain Library

Libbitcoin

The Bitcoin Development Library

Documentation is available on the wiki.

License Overview

All files in this repository fall under the license specified in COPYING. The project is licensed as AGPL with a lesser clause. It may be used within a proprietary project, but the core library and any changes to it must be published online. Source code for this library must always remain free for everybody to access.

About Libbitcoin

The libbitcoin toolkit is a set of cross platform C++ libraries for building bitcoin applications. The toolkit consists of several libraries, most of which depend on the base libbitcoin-system library. Each library's repository can be cloned and built using common automake 1.14+ instructions. There are no packages yet in distribution however each library includes an installation script (described below) which is regularly verified in the automated build.

Installation

The master branch is a staging area for the next major release and should be used only by libbitcoin developers. The current release branch is version3. Detailed installation instructions are provided below.

• Debian/Ubuntu
• MacOS
• Windows

Autotools (advanced users)

On Linux and macOS libbitcoin is built using Autotools as follows.

$ ./autogen.sh
$ ./configure
$ make
$ sudo make install
$ sudo ldconfig

A minimal libbitcoin build requires boost and libsecp256k1. The libbitcoin/secp256k1 repository is forked from bitcoin-core/secp256k1 in order to control for changes and to incorporate the necessary Visual Studio build. The original repository can be used directly but recent changes to the public interface may cause build breaks. The --enable-module-recovery switch is required.

Debian/Ubuntu

Libbitcoin requires a C++11 compiler, currently minimum GCC 4.8.0 or Clang based on LLVM 3.5.

Install the build system (Automake minimum 1.14) and git:

$ sudo apt-get install build-essential autoconf automake libtool pkg-config git

Next download the install script and enable execution:

$ wget https://raw.githubusercontent.com/libbitcoin/libbitcoin/version3/install.sh
$ chmod +x install.sh

Finally install libbitcoin with recommended build options:

$ ./install.sh --prefix=/home/me/myprefix --build-boost --disable-shared

Libbitcoin is now installed in /home/me/myprefix/.

MacOS

The macOS installation differs from Linux in the installation of the compiler and packaged dependencies. Libbitcoin supports both Homebrew and MacPorts package managers. Both require Apple's Xcode command line tools. Neither requires Xcode as the tools may be installed independently.

Libbitcoin compiles with Clang on macOS and requires C++11 support. Installation has been verified using Clang based on LLVM 3.5. This version or newer should be installed as part of the Xcode command line tools.

To see your Clang/LLVM version:

$ clang++ --version

You may encounter a prompt to install the Xcode command line developer tools, in which case accept the prompt.

Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.0.0
Thread model: posix

If required update your version of the command line tools as follows:

$ xcode-select --install

Using Homebrew

First install Homebrew.

Next install the build system (Automake minimum 1.14) and wget:

$ brew install autoconf automake libtool pkgconfig wget

Next download the install script and enable execution:

$ wget https://raw.githubusercontent.com/libbitcoin/libbitcoin/version3/install.sh
$ chmod +x install.sh

Finally install libbitcoin with recommended build options:

$ ./install.sh --prefix=/home/me/myprefix --build-boost --disable-shared

Libbitcoin is now installed in /home/me/myprefix/.

Using MacPorts

First install MacPorts.

Next install the build system (Automake minimum 1.14) and wget:

$ sudo port install autoconf automake libtool pkgconfig wget

Next download the install script and enable execution:

$ wget https://raw.githubusercontent.com/libbitcoin/libbitcoin/version3/install.sh
$ chmod +x install.sh

Finally install libbitcoin with recommended build options:

$ ./install.sh --prefix=/home/me/myprefix --build-boost --disable-shared

Libbitcoin is now installed in /home/me/myprefix/.

Build Notes for Linux / macOS

The install script itself is commented so that the manual build steps for each dependency can be inferred by a developer.

You can run the install script from any directory on your system. By default this will build libbitcoin in a subdirectory named build-libbitcoin and install it to /usr/local/. The install script requires sudo only if you do not have access to the installation location, which you can change using the --prefix option on the installer command line.

The build script clones, builds and installs two unpackaged repositories, namely:

• libbitcoin/secp256k1
• libbitcoin/libbitcoin

The script builds from the head of their version7 and version3 branches respectively. The master branch is a staging area for changes. The version branches are considered release quality.

Build Options

Any set of ./configure options can be passed via the build script, for example:

$ ./install.sh CFLAGS="-Og -g" --prefix=/home/me/myprefix

Compiling with ICU (International Components for Unicode)

Since the addition of BIP-39 and later BIP-38 and Electrum mnemnoic support, libbitcoin conditionally incorporates ICU. To use passphrase normalization for these features libbitcoin must be compiled with the --with-icu option. Currently libbitcoin-explorer is the only other library that accesses this feature, so if you do not intend to use passphrase normalization this dependency can be avoided.

$ ./install.sh --with-icu --build-icu --build-boost --disable-shared

Building ICU and/or Boost

The installer can download and install these dependencies. ICU is a large package that is not typically preinstalled at a sufficient level. Using these builds ensures compiler and configuration compatibility across all of the build components. It is recommended to use a prefix directory when building these components.

$ ./install.sh --prefix=/home/me/myprefix --with-icu --build-icu --build-boost --disable-shared

Windows

Visual Studio solutions are maintained for all libbitcoin libraries. NuGet packages exist for all dependencies. ICU is integrated into Windows and therefore not required as an additional dependency when using ICU features.

The libbitcoin execution environment supports Windows XP Service Pack 2 and newer.

Supported Compilers

Libbitcoin requires a C++11 compiler, which means Visual Studio 2013 (with a pre-release compiler update) or later. Download and install one of the following free tools as necessary:

• Visual Studio 2017 Express
• Visual Studio 2015 Express
• Visual Studio 2013 Express
  • November 2013 CTP Compiler for Visual Studio 2013
  • November 2013 CTP Compiler installation issue

NuGet Repository

Dependencies apart from the libbitcoin libraries are available as NuGet packages:

• Packages maintained by sergey.shandar
  • boost
  • boost_atomic
  • boost_chrono
  • boost_date_time
  • boost_filesystem
  • boost_iostreams
  • boost_locale
  • boost_log
  • boost_log_setup
  • boost_program_options
  • boost_regex
  • boost_system
  • boost_thread
  • boost_unit_test_framework
• Packages maintained by evoskuil
  • secp256k1
  • libzmq [required for client-server repositories only]

The packages can be viewed using the NuGet package manager from the libbitcoin solution. The package manager will prompt for download of any missing packages.

The libbitcoin solution files are configured with references to these packages. The location of the NuGet repository is controlled by the nuget.config file repositoryPath setting and the NuGetPackageRoot element of each [project].props file.

Build Libbitcoin Projects

After cloning the the repository the libbitcoin build can be performed from within Visual Studio or using the build_all.bat script provided in the builds\msvc\build\ subdirectory. The script automatically downloads all required NuGet packages.

Tip: The build_all.bat script builds all valid configurations for all compilers. The build time can be significantly reduced by disabling all but the desired configuration in build_base.bat and build_all.bat.

The libbitcoin dynamic (DLL) build configurations do not compile, as the exports have not yet been fully implemented. These are currently disabled in the build scripts but you will encounter numerous errors if you build then manually.

Optional: Building External Dependencies

The secp256k1 and libzmq package above are maintained using the same Visual Studio template as all libbitcoin libraries. If so desired these can be built locally, in the same manner as libbitcoin.

• libbitcoin/secp256k1
• zeromq/libzmq

This change is properly accomplished by disabling the "NuGet Dependencies" in the Visual Studio properties user interface and then importing secp256k1.import.props, which references secp256k1.import.xml and libzmq.import.props, which references libzmq.import.xml.

See boost documentation for building boost libraries for Visual C++.

libbitcoin-blockchain's Issues

Move last checkpoint to construct and pass from node/server.

Implement block validation lock priority over tx validation.

http://stackoverflow.com/questions/11666610/how-to-give-priority-to-privileged-thread-in-mutex-locking

Regression in testnet validation, fails on block [4209].

Data population was broken by an inverted is_transaction_pool() test as part of the tx validation work in progress.

Verified block infinite loop

Quick question. I got into a state where it just infinitely verifies the same block and never moves forward. Is this a corrupt database issue in general?

Build break on OSX Mavericks (Travis)

CXX      src/database/libbitcoin_blockchain_la-mmfile.lo
src/database/mmfile.cpp:88:33: error: use of undeclared identifier 'MREMAP_MAYMOVE'
        data_, size_, new_size, MREMAP_MAYMOVE));
                                ^
1 error generated.
make: *** [src/database/libbitcoin_blockchain_la-mmfile.lo] Error 1
The command "if [[ $OSX   && $CLANG && $DYNAMIC ]]; then ./install.sh ; fi" exited with 2.

Master build configured for version3.

This was an error in the manual squashing of commits that were added to master and version3 after the version3 branch was created.

Create independent libbitcoin-database.

This will fork off the already well-isolated libbitcoin-blockchain/database subfolder into an independent repository and namespace as libbitcoin-database::database.

This will transition the file datase.hpp into the interface definition as bc::database::data_base reusing the technique of renaming the primary class/file to avoid conflict with its namespace, as with:

• bc::network::p2p
• bc::node::p2p:_node
• bc::server::p2p:_server
• bc::blockchain::block_chain

The interface will initially be:

class BCD_API data_base
{
public:
    typedef uint64_t handle;

    virtual bool create() = 0;
    virtual bool start() = 0;
    virtual bool stop() = 0;

    virtual handle start_read() = 0;
    virtual bool start_write() = 0;
    virtual bool end_write() = 0;
    virtual bool is_read_valid(handle handle) = 0;
    virtual bool is_write_locked(handle handle) = 0;

    virtual chain::block::ptr pop() = 0;
    virtual void push(chain::block::ptr block) = 0;

    virtual block_database& blocks() = 0;
    virtual spend_database& spends() = 0;
    virtual transaction_database& transactions() = 0;
    virtual history_database& history() = 0;
    virtual stealth_database& stealth() = 0;
};

The objective is to more formally isolate the database from blockchain validation, orphan and transaction pools, and public query semantics. This will allow for drop-in database replacement and facilitate independent evolution of the next generation (private query) database. It is anticipated that an interface layer will initially be added to generalize the current interface with private query requirements.

Block validator should test fetch_block result.

The presumption is that preceding blocks will always exist. This is based on tracking of current height in the validate_block::height_ member, which is obtained from a hash table search.

https://github.com/libbitcoin/libbitcoin-blockchain/blob/version2/src/validate_block.cpp#L418
https://github.com/libbitcoin/libbitcoin-blockchain/blob/version2/src/validate_block.cpp#L433

A database corruption scenario, in which the index lags the hash table, can produce an error which could be detected here.

Ensure at least 1 satoshi fee if fee/byte config is nonzero.

Parallelize block validation in several places.

A number of operations within the blockchain can easily execute in parallel. The most important is input validation. Currently this is executing on a strand and is therefore entirely sequential.

Chain state populate not safe for get allow collision height.

The chain state populator was modified to support generation of the tx pool chain state. This required avoiding of each query in which the branch would be empty (pool). One of the four such conditions was missed in the implementation, resulting in a segfault.

Cache top bock in block_chain.

libbitcoin/libbitcoin-database#122

Move fetch_block_locator from spin lock.

Add milestones.

Checkpoints currently force the chain through a specific block. This does not produce the desired behavior.

The objective of checkpoints was to prevent wasting time on weak chains especially at low hash power. Checkpoints also provide a performance boost in avoidance of validation for a chain segment that is encountered that has previously been verified.

The objectives can both be attained without the undesired forcing behavior. The checkpoint hash is also indicative of a certain amount of work. This can be treated as a minimum amount of work at the given height (i.e. not a min + max). The prevents inadvertently blocking a higher difficulty chain. The avoidance of validation can be achieved based on building a verified header chain that matches a specified point. In that case all blocks to that point can be assumed valid (based on previous validation by the owner), allowing the implementation to skip expensive aspects of the validation.

Renaming the configuration setting value will be helpful in preventing conflation with the original implementation.

Fetch locator block hashes/headers search is reversed.

There is a bug in the implementation of fetch_locator_block_headers and fetch_locator_block_hashes in that the block locator is searched in reverse order.

Optimize block header info retrieval

During validation we are retrieving the full block when we only need header info:

block_header_type validate_block_impl::fetch_block(size_t fetch_height)
{
    if (fetch_height > fork_index_)
    {
        const auto fetch_index = fetch_height - fork_index_ - 1;
        BITCOIN_ASSERT(fetch_index <= orphan_index_);
        BITCOIN_ASSERT(orphan_index_ < orphan_chain_.size());
        return orphan_chain_[fetch_index]->actual().header;
    }

    // TODO: This is over-requesting, can be optimized.
    // We only really need the bits and timestamp fields.
    auto result = interface_.blocks.get(fetch_height);
    BITCOIN_ASSERT(result);
    return result.header();
}

File remap during read can cause segfault.

This has not been observed, but is an issue. During any write a database file may be expanded and, in the process, remapped in memory. This invalidates any pointers held at the time. Given that the database uses sequence locking, reads may be ongoing during any write. Reads during write are later invalidated. But in the case of immediately following a memory remap the reads could easily follow invalidated pointers, resulting in a logical fault or segfault.

Priority thread pool must be closed before closing store.

I have seen two cases where the store is closed with incorrect file size. The opening guard prevents the store from opening, requiring a resync. It appears that this is the result of a shutdown race caused by failure to terminate the priority threadpool before closing the store.

Build breaks in linux/clang (static and dynamic)

-f 'test/htdb.cpp' || echo './'`test/htdb.cpp
test/htdb.cpp:33:10: error: no type named 'default_random_engine' in namespace
      'std'
    std::default_random_engine& engine, size_t size)
    ~~~~~^
test/htdb.cpp:62:32: error: use of undeclared identifier 'engine'; did you mean
      'inline'?
    std::default_random_engine engine;
                               ^~~~~~
                               inline
test/htdb.cpp:62:10: error: no type named 'default_random_engine' in namespace
      'std'
    std::default_random_engine engine;
    ~~~~~^
test/htdb.cpp:62:32: error: expected unqualified-id
    std::default_random_engine engine;
                               ^
test/htdb.cpp:65:50: error: use of undeclared identifier 'engine'
        data_chunk value = generate_random_bytes(engine, tx_size);
                                                 ^
test/htdb.cpp:96:32: error: use of undeclared identifier 'engine'; did you mean
      'inline'?
    std::default_random_engine engine;
                               ^~~~~~
                               inline
test/htdb.cpp:96:10: error: no type named 'default_random_engine' in namespace
      'std'
    std::default_random_engine engine;
    ~~~~~^
test/htdb.cpp:96:32: error: expected unqualified-id
    std::default_random_engine engine;
                               ^
test/htdb.cpp:99:50: error: use of undeclared identifier 'engine'
        data_chunk value = generate_random_bytes(engine, tx_size);
                                                 ^
9 errors generated.
make[1]: *** [test/test_libbitcoin_blockchain_test-htdb.o] Error 1
make[1]: Leaving directory `/home/travis/build/libbitcoin/libbitcoin-blockchain'
make: *** [check-am] Error 2
The command "if [[ $LINUX && $CLANG && $STATIC  ]]; then ./install.sh --disable-silent-rules --disable-shared --build-gmp --build-boost --prefix=$HOME/my-prefix CXXFLAGS='-Os -stdlib=libc++' LDLIBS='-lc++ -lc++abi -lm -lc -lgcc_s -lgcc'; fi" exited with 2.

also dynamic...

The command "if [[ $LINUX && $CLANG && $DYNAMIC ]]; then sudo CXX=$CXX CC=$CC ./install.sh --disable-silent-rules --disable-static --build-boost CXXFLAGS='-Os -stdlib=libc++' LDLIBS='-lc++ -lc++abi -lm -lc -lgcc_s -lgcc'; fi" exited with 2.

testnet blockchain not keeping up to date

On testnet my blockchain keeps falling behind but this doesn't happen on mainnet. It's a problem that I've tried many approaches to, until I noticed that many of the blocks on testnet seem to happen all at once:

http://tbtc.blockr.io/

And then I noticed these warnings:

00:42:14.256872 WARNING [poller]: Storing block 0000000013c7afcc1c74ff9a035ff195b6370a30c271ffd58c0c4b76af9f8684: Timestamp too far in the future
00:44:57.013515 WARNING [poller]: Storing block 0000000013c7afcc1c74ff9a035ff195b6370a30c271ffd58c0c4b76af9f8684: Timestamp too far in the future
00:55:27.380943 WARNING [poller]: Storing block 000000002fca961dc15d9db0bd7570673be740d3e54ebeb8d057152f07464adb: Timestamp too far in the future
00:57:32.134283 WARNING [poller]: Storing block 000000002fca961dc15d9db0bd7570673be740d3e54ebeb8d057152f07464adb: Timestamp too far in the future
00:59:33.896819 WARNING [poller]: Storing block 000000002fca961dc15d9db0bd7570673be740d3e54ebeb8d057152f07464adb: Timestamp too far in the future
01:01:31.168644 WARNING [poller]: Storing block 000000002fca961dc15d9db0bd7570673be740d3e54ebeb8d057152f07464adb: Timestamp too far in the future

So obviously this rule in check_block(), does not apply to testnet:

https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/src/validate.cpp#L376

const ptime block_time = from_time_t(blk_header.timestamp);
const ptime two_hour_future =
    second_clock::universal_time() + hours(2);
if (block_time > two_hour_future)
    return error::futuristic_timestamp;

I'm trying to find where in the Satoshi source code this rule is excluded but haven't found it yet.

Soft fork activation levels incorrect for testnet.

Fixed in commit: 49a5a00

Fetch block locator threshold not effective.

This was a trivial error in the generation of the locator whereby the threshold wasn't being honored. This resulted in redundant block requests, a waste of IBD time.

Optimize branch prevout and spent population.

These two functions are O(n * m^2) in block and output count, and get very costly with certain blocks. By committing valid transactions of a checked block with sufficient work, before continuing the block validation, we can leverage the store's hash table (and eventual transaction indexing) to achieve constant time for these populations without building redundant hash tables. This requires a dependency ordering of block txs and forgoing parallel tx validation as isolated from population :/.

[version3] segfault in chain_state (via server).

From mainnet1.libbitcoin.net.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000005cfd3d in libbitcoin::chain::chain_state::to_pool (top=..., version=<optimized out>) at src/chain/chain_state.cpp:474
474         return data;
(gdb) where
#0  0x00000000005cfd3d in libbitcoin::chain::chain_state::to_pool (top=..., version=<optimized out>) at src/chain/chain_state.cpp:474
#1  0x00000000005d215d in libbitcoin::chain::header::operator= (this=0x0, other=...) at src/chain/header.cpp:128
#2  0x0000000000000000 in ?? ()

Restarted after with good store because of write flushing.

Block height mismatch in coinbase [master]

mainnet, good block, regression:

WARNING [blockchain] Invalid block [00000000000004259387ca96016ca8e0d8b205db01424525796d076b86afbd96] block height mismatch in coinbase

Failure fetching input transaction

08:04:43.777374 WARNING [validate] Failure fetching input transaction.
08:04:43.778380 WARNING [validate] Invalid input [cb57c8ee6d29c8343bb3c783bd068edb6a3d75489b8ba16d1431b3884b389881:464]
08:04:43.791376 WARNING [validate] Invalid block [000000000000000005a91d4e72303f6b112af30ebc40262d404826c07feeb772] validation of inputs failed
08:04:43.791376 ERROR [poller] Error storing block [000000000000000005a91d4e72303f6b112af30ebc40262d404826c07feeb772] validation of inputs failed

VC++ 32 bit "pupshpop" test failure.

*************************unknown location(0): fatal error in "pushpop": memory access violation occurred at address 0xffffffff, while attempting to  write to an inaccessible (or protected) address
..\..\..\..\test\databases.cpp(222): last checkpoint
**************************

It's not clear whether 32 bit architecture support is intended.

Blockchain corruption appears to result in normal course.

After adding these exceptions to the slab and record implementations I've encountered a failure by hitting one of them during three different synchronizations on Windows. There's no reason to believe the problem is unique to Windows.

The problem is persistent as it represents an index corruption written to the database. The failure occurs on lookup of a transaction (in my cases during block validation, but it would occur in any case of the lookup).

The error is now cleanly reported and shutdown occurs. Previously it would result in an infinite loop on the particular thread.

https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/include/bitcoin/blockchain/impl/htdb_slab.ipp#L76

https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/include/bitcoin/blockchain/impl/htdb_slab.ipp#L128

https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/include/bitcoin/blockchain/impl/htdb_record.ipp#L71

https://github.com/libbitcoin/libbitcoin-blockchain/blob/master/include/bitcoin/blockchain/impl/htdb_record.ipp#L123

Hot backup and fault tolerance limitations

The blockchain store is intended to support hot backup and to be tolerant of hard shutdown (during write) faults. Due to the speed and limited amount of writes, this is generally the case, but it is not guaranteed by the implementation.

The two scenarios rely on the store always existing in a consistent state. However there are multiple indexes in distinct files. Writes across these files are not atomic and therefore these scenarios may result in a partial update. The implementation is unable to detect this corruption, as each file individually remains consistent.

Add configurable mempool timeout policy.

If the network is flooded with activity, unconfirmed transactions may be dropped in a few hours, but it the network is slow, it may be days. ... Is there any way to configure the mempool based on a timeout, so any transaction over X hours old will be dropped? - @swansontec

It would consist of a deadline timer on the tx pool, timeout config setting, per entry time stamp, and thread safe pool scan/update on timeout, with timer reinitialization.

Test run doesn't terminate in VC++.

Add mempool policy/DoS protection.

32 bits compatibility?

During my attempt to generate a debian package of libbitcoin-blockchain for i386 I discovered the following assert in src/database/mmfile.cpp:

// mmfile should be able to support 32 bit but because the blockchain 
// requires a larger file this is not validated or supported.
static_assert(sizeof(void*) == sizeof(uint64_t), "Not a 64 bit system!");

Just to know, isn't there any plan or trick at all to support 32bits?

Unblock safe store reads (no need for spin lock).

Many objects, including blocks and transactions, can be safely read even during write without possibility of corruption. The exceptions are metadata that is modified after the initial objects have been indexed. The file is trimmed to last logical size upon closing, but records are never trimmed while the store is open, they are only un-indexed.

• tx output spender heights, which is currently read only while write is precluded (in validation).
• tx height, which is used to store height and validation flags (so must be read/written atomically).
• tx position, which is used to indicate position and confirmation (so must be read/written atomically).
• fetch_transaction_position (requires above atomicity)
• fetch_output (see spender height issue above)
• last_height (safe)
• filter_blocks and filter_transactions (safe)
• fetch_locator_block_hashes and fetch_locator_block_headers (safe)
• fetch_spend (atomic addition to hash table)
• fetch_history (atomic addition to hash table, not across txs/blocks)
• fetch_stealth (atomic addition to array, not across txs/blocks)

Futures (these are safe now but won't be due to height overloading):

• header height, will be used to store height and to indicate chain (so must be read/written atomically).
• block and merkle_block (apart from header height) and eventually compact_block.
• fetch_block_height (requires above atomicity)

To implement atomicity without overextending locks, tx/block heights and positions must be read in construction of the transaction_result and block_result objects. This will be a slight performance hit in exchange for a major performance benefit (100% successful read while write).

Unsafe:

• fetch_block_locator (selection is by height/chain)

This can be resolved by caching the locator. It can be updated for each new block and incremented when it requires expansion (rarely). Ultimately this should be part of chain _state.

[master] flush_lock missing after stop with store open.

When set to flush_reorganizations = false there should exist a flush_lock file in the store directory any time following a hard shutdown of the store while it is open, until the store (or the file) is manually deleted. [Deletion of the file without deletion of the store ensures that the store is corrupt and will cause downstream failures.]

[blockchain]
# Flush each reorganization to disk, defaults to false.
flush_reorganizations = false

However there have been two observed occurrences of a flush_lock file not being found in this scenario (at least after the first attempted restart). The file must have been cleared on startup or on shutdown (or both). This should not be possible unless the database is closed properly. However in neither case would this have been true, and in the second there was an observed database integrity failure (file size incorrect).

Compute hashes in parallel.

Block check can be preceded with a parallel loop over transaction hash() calls to force parallel hash computation. This would apply only to whole blocks.

Hardcoded height causes perf drop at block 337,459.

This is the cause of the dramatic sync slowdown observed starting at block 337,459

BC_CONSTEXPR size_t block_validation_cutoff_height = 337459;

and

    // Skip non-essential checks if before last checkpoint.
    if (fork_index < block_validation_cutoff_height)
        return bc::error::success;

    // Perform strict but slow tests - connect_block()
    return validate.connect_block();

This value must be pulled out to configuration and set at the beginning of any lengthy sync.

Intermittent test termination by Travis host.

The dynamic build succeeds. Currently the test output is limited to global fail. Need to create an execution wrapper as in bx.

make  check-TESTS
make[1]: Entering directory `/home/travis/build/libbitcoin/libbitcoin-blockchain'
Running 12 test cases...
/bin/bash: line 5: 18736 Killed                  ${dir}$tst
FAIL: test/libbitcoin_blockchain_test
==================================
1 of 1 test failed
Please report to [email protected]
==================================
make[1]: *** [check-TESTS] Error 1
make[1]: Leaving directory `/home/travis/build/libbitcoin/libbitcoin-blockchain'
make: *** [check-am] Error 2
The command "if [[ $LINUX && $GCC   && $STATIC  ]]; then ./install.sh --disable-shared --build-gmp --build-boost --prefix=$HOME/my-prefix --build-dir=my-build CXXFLAGS='-Os -s'; fi" exited with 2.

Validation failures on two testnet blocks.

17:13:34.805351 WARNING [validate] Failure fetching input transaction.
17:13:34.805931 WARNING [validate] Invalid input [227ee87b49a652a23ae5c30de32d4dcedebdf442e7faced2789be7f7702667a9:189]
17:13:34.806059 WARNING [validate] Invalid block [0000000000cecfb76203a34e6d55eac4994ddfaad9d513f80efd216df888b4fd] validation of inputs failed
17:13:34.806179 ERROR [poller] Error storing block [0000000000cecfb76203a34e6d55eac4994ddfaad9d513f80efd216df888b4fd] validation of inputs failed

See: tx and block

17:13:08.067268 WARNING [validate] Failure fetching input transaction.
17:13:08.067858 WARNING [validate] Invalid input [3abacf2d0693eb7133b664e7e51f41dd0c5ec95cb6c481052bfd7585a76668b7:76]
17:13:08.067931 WARNING [validate] Invalid block [0000000025c5182db4ca42ab7a7d22f965ff42974215177ebb6b2d6dceac3de3] validation of inputs failed
17:13:08.067984 ERROR [poller] Error storing block [0000000025c5182db4ca42ab7a7d22f965ff42974215177ebb6b2d6dceac3de3] validation of inputs failed

See: tx and block

For input validation, tx-based validation (for mempool acceptance) validates first against mempool txs and then against blockchain txs:

https://github.com/libbitcoin/libbitcoin-blockchain/blob/version2/src/validate_transaction.cpp#L204

But block validation is limited to blockchain txs, and as one can see in this function, that is the source of the errors above:

https://github.com/libbitcoin/libbitcoin-blockchain/blob/version2/src/validate_block.cpp#L482

The implementation must consider a block's own transaction set as well as previously-accepted blocks.

Travis build break, missing include.

CXX    src/libbitcoin_blockchain_la-db_interface.lo
In file included from src/db_interface.cpp:20:0:
./include/bitcoin/blockchain/db_interface.hpp:28:65: fatal error: bitcoin/blockchain/database/history_scan_database.hpp: No such file or directory
 #include <bitcoin/blockchain/database/history_scan_database.hpp>
                                                                 ^
compilation terminated.
make: *** [src/libbitcoin_blockchain_la-db_interface.lo] Error 1

Build breaks in OSX and VC++ (CTP_Nov2013)

My-iMac:~ Eric$ clang --version
Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.0.0
Thread model: posix

  CXX      src/libbitcoin_blockchain_la-blockchain.lo
  CXX      src/libbitcoin_blockchain_la-transaction_pool.lo
warning: unknown warning option '-Wno-unused-local-typedefs' [-Wunknown-warning-option]
warning: unknown warning option '-Wno-unused-local-typedefs' [-Wunknown-warning-option]
src/blockchain.cpp:44:24: error: constexpr variable 'divisor' must be initialized by a constant expression
    constexpr uint64_t divisor = std::pow(2, 63);
                       ^         ~~~~~~~~~~~~~~~
src/blockchain.cpp:44:34: note: non-constexpr function 'pow<int, int>' cannot be used in a constant expression
    constexpr uint64_t divisor = std::pow(2, 63);
                                 ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/cmath:961:1: note: declared here
pow(_A1 __x, _A2 __y) _NOEXCEPT
^
1 warning and 1 error generated.
make: *** [src/libbitcoin_blockchain_la-blockchain.lo] Error 1

Chain state improperly promoted in case of reorg.

The change state caching optimization promotes the pool state to next block state in the case where the next block is not a reorg. However reorg was computed based on a mismatch between branch top height and pool height. This is incorrect in the case where the reorg branch reaches the same height as the pool.

Instead the branch must first be tested for single block size, in which case the height match is correct.

Store sequential lock implementation can cause failure

The store utilizes a seqlock to protect reads from corruption during writes. Concurrent writes are precluded by the use of an ordered strand.

The seqlock pattern allows in-progress reads to continue during writes, while blocking new reads until the write completes. This prevents writer starvation in scenarios where writes are fast and limited in quantity relative to reads. Any read that spans a write is invalidated and reattempted once the write is complete.

The presumption is that, while the read-while-write may produce invalid results, it will not cause a catastrophic failure. In a previous implementation a catastrophic failure did result from the ordering of pointer updates, where the reader ended up in an infinite loop. This was resolved, but other issues remain.

The writer places pointer values using a byte-based serialization. This is non-atomic - any part of a pointer update may be read by the reader during the writer update. In other words, the first 1-7 bytes of the pointer may be updated in the pointer read by the reader. This would lead to an invalid offset, resulting in a corrupt read and/or (ideally) segmentation fault.

A restart would clear the error until a subsequent occurrence, unless corrupted read data was used in a subsequent write. Reads invoked by the writer are not affected, as the writer maintains consistency within its operations.

bip34 activation test was used in bip30 context.

Fixed in commit: 9bc70b7

Deadlocks result from improper priority thread pool interaction.

With very high peer counts (stress testing) that are higher than network thread count we may see deadlock. However given the high volume of activity it may have just been backlog. The network is designed to run on a single thread which at typical levels has never been a problem.

Segfault with tx relay enabled.

This was the result of an inverted condition.

Mac OSX Build Error/Issue

Here are the results of a serialized makefile output:
.
.
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libbitcoin-blockchain.pc
config.status: executing depfiles commands
config.status: executing libtool commands
CXX src/libbitcoin_blockchain_la-blockchain.lo
CXX src/libbitcoin_blockchain_la-checkpoints.lo
CXX src/libbitcoin_blockchain_la-transaction_pool.lo
CXX src/libbitcoin_blockchain_la-validate.lo
CXX src/libbitcoin_blockchain_la-organizer.lo
In file included from ../libbitcoin-blockchain/src/organizer.cpp:20:
../libbitcoin-blockchain/include/bitcoin/blockchain/organizer.hpp:104:43: error: no member named 'max_size_t' in namespace 'libbitcoin'; did you mean
'vm_size_t'?
static constexpr size_t null_height = bc::max_size_t;
^~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/mach/i386/vm_types.h:112:20: note:
'vm_size_t' declared here
typedef uintptr_t vm_size_t;
^
1 error generated.
make: *** [src/libbitcoin_blockchain_la-organizer.lo] Error 1

Any recommendations?

OSX Build Issue

Getting MacBookPro results that differ from Travis

% git show

commit 012d98689e2fd66eceb7ed121ffe1cbe6698f02c
Merge: 9c8ee93 b0f36cf
Author: Eric Voskuil <[email protected]>
Date:   Fri Oct 2 01:52:02 2015 -0700

    Merge pull request #110 from evoskuil/no-testnet

    No testnet

./install.sh --prefix=/Users/XXX/Projects/BS/libbitcoin-blockchain/test --build-dir=/Users/XXX/Projects/BS/libbitcoin-blockchain/build --build-boost

...
...
  CXX      tools/show_records/tools_show_records_show_records-show_records.o
  CXX      tools/spend_db/tools_spend_db_spend_db-spend_db.o
  CXX      tools/stealth_db/tools_stealth_db_stealth_db-stealth_db.o
  CXX      tools/transaction_db/tools_transaction_db_transaction_db-transaction_db.o
tools/show_linked_records/show_linked_records.cpp:83:31: error: no viable conversion from 'const chain_item' to 'index_type' (aka 'unsigned int')
            chain_type chain{{new_item}};
                              ^~~~~~~~
  CXXLD    src/libbitcoin-blockchain.la
1 error generated.
make: *** [tools/show_linked_records/tools_show_linked_records_show_linked_records-show_linked_records.o] Error 1
make: *** Waiting for unfinished jobs....
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: src/.libs/libbitcoin-blockchain.a(src_libbitcoin_blockchain_la-mman.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: src/.libs/libbitcoin-blockchain.a(src_libbitcoin_blockchain_la-mman.o) has no symbols

Build breaks in VC++ (CTP_Nov2013)

The libbitcoin-blockchain repo is the only libbitcoin repo that doesn't support VS2013/Windows.

The primary issue is exemplified here:

template <typename IndexType, typename ValueType>
class disk_array
{
public:
    static constexpr ValueType empty = std::numeric_limits<ValueType>::max();
    ...
};

The use of this idiom:

constexpr T foo = std::numeric_limits<T>::max();

isn't allowed in VC Novermber 2013 CTP.

Flush lock stays open in case of validation failure and shutdown.

Flush lock should always close upon successful controlled shutdown.