bosagora/agora

POC Node implementation for CoinNet

Agora

Docker Hub image

Node implementation for BOA CoinNet

TestNet access

If you are looking for the instructions to access TestNet, please follow this link.

Docker usage

We provide a public build of this repository (see above). The easiest way to get agora is to run docker pull bosagora/agora.

This will pull the latest tag, which is the one our team deploys internally. Previous versions can be pulled via their version, e.g. docker pull bosagora/agora:v0.24.0.

The Dockerfile lives at the root of this repository, so one can run docker build -t agora . to build it. Note that you need to initialize submodules (git submodule update --init) before you first build agora.

Building on POSIX

Dependencies

You need a recent clang++ (with N4387 fixed), a recent (>=1.26.0) version of the LDC compiler, and dub.

Additionally, the following are dependencies:

• libsodium >= 1.0.18: Development library
• pkg-config: For DUB to find the correct sqlite3 and other system libraries
• openssl: Binary (to detect the version) and development library
• sqlite3: Development library
• zlib: Development library

Additionally, on OSX, PKG_CONFIG_PATH needs to be properly set to pick up sqlite3 and openssl. Provided you installed those packages via brew, you can run the following command prior to building:

export PKG_CONFIG_PATH="/usr/local/opt/sqlite/lib/pkgconfig:/usr/local/opt/[email protected]/lib/pkgconfig"

Since this setting does not persist, we recommend you follow Homebrew's instructions and add it to your .bashrc, .zshrc, etc...

The following can be used on Ubuntu 20.04 to get the required dependencies:

sudo apt-get update && sudo apt-get install build-essential clang libsodium-dev libssl-dev libsqlite3-dev zlib1g-dev

On older distributions (e.g. Ubuntu 18.04), libsodium might not be at version v1.0.18.

Build instructions

# First, make sure you have the package listed in the Dependencies section installed.
#
# Then, install the LDC compiler (you might want to use a newer version)
# This will also install dub, the D package manager / build tool
curl https://dlang.org/install.sh | bash -s ldc-1.28.1
# Add LDC to the $PATH
source ~/dlang/ldc-1.28.1/activate
# Clone this repository
git clone https://github.com/bosagora/agora.git
# Use the git root as working directory
cd agora/
# Initialize and update the list of submodules
git submodule update --init
# Build the application
dub build --skip-registry=all
# Build & run the tests
dub test --skip-registry=all

Running tests

Agora comes with plenty of tests. Take a look at the CI configuration for an exact list. At the moment, the three main ways to run the test are:

• dub test: Test the consensus protocol and runs all the unittests
• rdmd tests/runner.d: Run a serie of simple integrations tests
• ci/system_integration_test.d: Run a full-fledged system integration test, including building the docker image.

Source code organization

The code is divided into multiple parts:

• The "main" source code lives in source/agora. Consensus rules and node implementation lives there;
• The extracted SCP code lives in source/scpp. See the README for more information;
• Our Dlang binding for SCP's C++ implementation lives in source/scpd, along with some C++ helpers;
• The setup interface, Talos, is a React app living in the talos directory;

The directory source/agora/ is the root package. Sub-packages include:

• agora.api: Defines interfaces that describe the APIs exposed by different types of nodes (FullNode, Validator, Flash...);
• agora.cli: Contains various CLI tools used by Agora (see the dub.json sub-configuration to build them);
• agora.common: A leaf package that contains various general-purpose utilities which aren't Agora specific;
• agora.consensus: Implements Agora's consensus protocol, can be used as a standalone library;
• agora.crypto: Contains cryptographic utilities, such as the key type;
• agora.flash: Implementation of the Flash layer;
• agora.network: Manages a node's view of the network;
• agora.node: Implementation of the two main types of nodes (Full node and Validator) and related modules;
• agora.script: Implementation of the script engine;
• agora.stats: Holds helper modules for statistics exported by Agora and other tools;
• agora.test: Contains network tests for the consensus protocol. See the README for more details;
• agora.utils: Contains utilities that don't fit in other packages, such as a custom tracing GC, Tracy integration, TxBuilder...

Additionally, Agora's dependencies live in submodules and are managed via git submodule. Of those submodules, few are internally managed libraries (crypto, serialization, ...), and some may be forks of externally managed libraries (either because the library is unmaintained or because specific tweaks were needed). A README provides more details.

Running single test node using TestNet GenesisBlock

For a test of a full node connecting to testnet, try:

docker run -p 2826:2826 -p 9111:9111 -v `pwd`/devel/config-fullnode.yaml:/agora/etc/config.yaml bosagora/agora -c /agora/etc/config.yaml

The node API will be locally available. It can be accessed with:

curl http://127.0.0.1:2826/blocks/0

Likewise, the metrics endpoint will be exposed:

curl http://127.0.0.1:9111/metrics

agora's Issues

Dub + Coverage + LDC = Game over (DUB#1758)

dlang/dub#1758

Settlement Rules

Review SCP nomination protocol in order to keep it safe while opening the network

Currently the nomination protocol is handled by SCP, but we have to assess the changes required (if any) that come with making the network open.

One obvious approach would be to build a filter for transactions, so we could guarantee liveness, ultimately. See #407 for more informations.

Re-enable LDC testing

Make sure this is really temporary.
Depends on Geod24/localrest#1

Flash TestNet

Trust Contracts

Settlement TestNet (Validator)

getBlocksFrom should not use size_t

getBlocksFrom should not use size_t, change it to uint because this is external API for different architects

getBlocksFrom (ulong block_height, size_t max_blocks);

Define and implement block rewards

Premise

The white-paper defines a few things w.r.t. rewards: categorization, amount, randomness...
However, freezing rewards will be removed, as they are a form of passive income which would make our coin a security.
While the total amount of rewards is likely to stay the same, the distribution among the various categories will change to align incentives with the goals stated in the white-paper.

Incentive

One of the goal of the white-paper is to separate the political incentive from the economical incentive.
The direction we are heading towards is to have low rewards for simple validators, and increasingly higher rewards for L2 providers (validators with > 40k).
Since L2 liquidity cannot be slashed (that'd be double spend), we assume that every validator has exactly 40k.

Additionally we need to give incentive for validators to:

• Accept new validators in the validator set;
• Ensure that pre-images and other critical data are correctly propagated;

In the second case, the lack of propagation / cooperation (e.g. by withholding pre-images) could be used as an attempt to slash a well-behaving node.

prerequisite: #2245

Define and implement path finding algorithm for Flash transactions

Assess need for SDK and implement them

Block creation

Implement Schnorr signatures

As a developer, I need a signature scheme that is efficient and composable in order to sign my blocks and eventually my transactions.

Definition of Done:

• Signatures can be aggregated into one signature
• Signatures can be made and verified according to the well-known Schnorr algorithm

Payment Channel

Switch to submodules for dependencies

Because it's just much simpler to deal with.

Network manager sends a transaction even if a node already has the transaction

The node implements a hasTransactionHash API, but the network manager gossips the transaction to a node even if that node already contains the transaction.

The node should instead query the target node's hasTransactionHash(), and only send the transaction if hasTransactionHash() returns false.

getClient address argument type should be changed

    // TODO: move from string Address type to URL or abstract Address type
    // Currently it used both for "base URLs" and IP addresses
    protected API getClient (Address address, Duration timeout)

Simplified Payment Verification (SPV)

As a user, I want to be able to verify payment from a "simple device", such as a phone, without relying on having to download the full blockchain state and validate everything.

This will be addressed by making a block verifiable, then providing a Merkle Path within this block.

Need a way to cache a transaction's hash

Currently it's very had to make functions @nogc because any operation that's done on a transaction usually requires its hash, which is not typically passed around. For example, say you have a Transaction[] and you want to check if any of the transactions are contained in a container:

Transaction[] txs;

foreach (tx; txs)
    if (tx.hashFull() in map)  // allocation due to hashFull()
        ....

There might be a few ways of working around this:

• Store the Hash inside of the Transaction as a field, but don't serialize it when transferred over the network (to save on bandwidth). However this breaks contiguous serialization on disk (unless we are OK with serializing it to disk).
• Implement a wrapper struct that contains a hash of its wrapped value. Say Hashed!Transaction, which would contain the transaction as an alias this field. However this probably won't compose well with arrays..

There might be other ways too. If you know some other techniques please write your ideas here.

Track upstream dub issue 825 / PR 1002

This issue / PR are about globing in sourceFiles, which as explained here would be quite sweet for us.

DMD bug: Array comparison mismatch for `immutable[]` vs `[]`

See https://github.com/bpfkorea/agora/blob/9d2c958fc34f8cd4fcd3d4ef95e28b62aa8f4f36/source/agora/common/Deserializer.d#L114-L131

Genesis block/validator creation

As a developer, I need a genesis block and an initial validator set in order to build a chain.

Definition of Done:

• The genesis transaction is hardcoded
• There is at least one freezing transaction in genesis
• The private key of the freezing transaction is controlled by the foundation

Publicly deployed nodes

Implement inbound connection banning support in vibe.d

As a Developer, I need to prevent one form of DDoS attacks from other nodes, we need support for banning inbound connections from specific addresses.

Definition of Done:

• When a request reaches the server, it can be made to be rejected before any significant processing / memory allocation happens
• The functionality supports dynamically banning
• The functionality supports dynamically unbanning
• The functionality does not add much overhead on the request processing

Write system integration test for the timeout feature

Move SQLite initialization to a common module

Once the UTXO set (backed by SQLite) is implemented, we'll have two classes using it. So it would make sense to move the initialization out of the TransactionPool and into a common place.

See also #195

Add freezing capability to UTXO

As a node operator, I can create a transaction that marks my UTXO as “frozen” so that I can use it as collateral for running a validator node.

Definition of Done:

• An UTXO can be marked as "frozen"
• When a transaction spending a frozen UTXO comes in, it marks the UTXO as "melting" for 2016 blocks
• Transactions attempting to spend melting UTXO are invalid
• After the UTXO has melt, the new UTXO can be spent without delay
• The amount must be exactly 40,000 BOA (for the moment)

Implement a TCP, binary-based interconnection protocol

As a developer, I need to be able to use a protobuffer generator that uses simple TCP (UDP?) connections and protobuff for arguments in order to have efficient communication between nodes.

Definition of Done:

• The generator has a way to listen to a port and do "routing"
• "Routing" is the act of directing a message to one of the interface's method according to the first byte (or some other data, open to discussion)
• Once the route has been chosen, the arguments are deserialized according to the method chosen
• The act of routing does not allocate
• Routing is nothrow
• There is a client class that can be instantiated with an API as template parameter and an address / port as parameter
• The client class has support for timeouts
• The client class can perform asynchronously (either by itself or through runTask)
• Both client and server code are critical for performance, hence the code is performant.
• There is support to reject inbound peers (agora #202 )

Atomic wallet integration

As a user, I can use Atomic wallet to buy, sell, atomically swap, and pay in BOA in order to use my coins.

Definition of Done:

• I can send coins to another BOA holder
• I can atomically swap BOA for BTC
• I can atomically swap BOA for ETH

Integration tests: it must be part of code

Same as unittests preferably if possible in particular case.

runner.d - is no good at all, it is even compiled with other compilation directives/versions like ommited version (Have_vibe_core).

Full Node

Deploy monitoring tools

Once we have publicly deployed nodes in place (#249), we will need to put in place tools to monitor that everything works correctly. Things like Grafana will allow us to monitor metrics such as (those are examples):

• Machine metrics: memory & CPU usage, network bandwidth, ...;
• General node health status: Transaction pool size, disk errors, rejected transactions...
• General blockchain health data: Deviation from the expected time for blocks, average time from pool to block, etc...

Some example of what others do:

• https://www.lopp.net/bitcoin-information/statistics-metrics.html
• https://bitnodes.io/
• https://ethernodes.org/
• https://ethstats.net/

Tool Integration

Build a command line tool to send transactions

As a developer, I need a CLI tool to send a transaction to a node in order to quickly test things.

Definition of Done:

• There is a CLI tool that takes an IP/port (node address), an amount, an Outpoint (Tx hash + idx) and a private key
• The CLI tool builds a transaction from its parameters and send it to the node
• The CLI tool has a dump-only option

Publicly available block explorer

Random segfault in the tests

Observed on LDC 1.16, Mac OSX:

Target /Users/travis/build/bpfkorea/agora/source/scpp/build/libscp.o is up to date, nothing to do
Deserializing composite type ArgWrapper!() which has no serializable fields.
Serializing composite type ArgWrapper!() which has no serializable fields
Running ./build/agora 
0   agora                               0x000000011016b88e _D4core7runtime18runModuleUnitTestsUZ19unittestSegvHandlerUNbiPSQCk3sys5posix6signal9siginfo_tPvZv + 62
1   libsystem_platform.dylib            0x00007fff5baf2f5a _sigtramp + 26
0   agora                               0x000000011016b88e _D4core7runtime18runModuleUnitTestsUZ19unittestSegvHandlerUNbiPSQCk3sys5posix6signal9siginfo_tPvZv + 62
1   libsystem_platform.dylib            0x00007fff5baf2f5a _sigtramp + 26
Program exited with code -11
The command "dub test --skip-registry=all --compiler=${DC}" exited with 2.

https://travis-ci.com/bpfkorea/agora/jobs/220748509

Implement support for secp256k1

As a user, I need to be able to use my TokenNet key in CoinNet in order to use CoinNet without going through YAA (Yet Another Airdrop).

Definition of Done:

• There is a crypto_scalarmult_secp256k1 similar to crypto_scalarmult_ed25519 with secp256k1
• There is a crypto_scalarmult_secp256k1_base similar to crypto_scalarmult_ed25519_base with secp256k1
• There is a noclamp version of those scalarmult functions

Currently UTXOs aren't recovered if SQLite database loading fails

If DB construction fail we may want to just recover txes from the network instead

Integrate scpd code into Agora

As a developer, I need the SCP code to be integrated into Agora in order for the basic block of consensus to work.

Definition of Done:

• There is a subclass of SCPDriver
• Agreement is made with the quorum slice using SCP

Add automatic size checks between C++ and D types (structs, etc)

Long term goal: Make this check automatic

Originally posted by @Geod24 in #2 (comment)

BanManager lacks persistence

If a node gets banned, and we restart our node, well.. you know what happens

Dedicated foundation nodes

As a pre-requisite to deploy CoinNet, we will need to deploy some nodes operated by the Foundation.

Since those nodes will have access to a significant amount of BOA funds, they should each live in their own server.
Once we get a good enough view of the resource consumption of a BOA node (which TestNet will give us), we can estimate the hardware need for each node.

A few things:

• Those servers will most likely be Linux server;
• They will have full-disk encryption;
• They will be monitored by the monitoring tools ( #255 );
• The number of server is a policy decision to be taken by the Foundation: currently I'd say 6-8, but depends on the network;

BlockStorage does not work when block contains multiple transactions

Place this into tests/unit/BlockStorageMultiTx:

/*******************************************************************************

    Create blocks with multiple transactions and test that they are
    read properly.

    Copyright:
        Copyright (c) 2019 BOS Platform Foundation Korea
        All rights reserved.

    License:
        MIT License. See LICENSE for details.

*******************************************************************************/

module unit.BlockStorageMultiTx;

import agora.common.Amount;
import agora.common.Block;
import agora.common.BlockStorage;
import agora.common.crypto.Key;
import agora.common.Data;
import agora.common.Hash;
import agora.common.Transaction;
import agora.consensus.Genesis;

import std.algorithm.comparison;
import std.file;
import std.path;

/// The maximum number of block in one file
private immutable ulong MFILE_MAX_BLOCK = 100;

/// blocks to test
const size_t BlockCount = 300;

///
private void main ()
{
    string path = buildPath(getcwd, ".cache");
    if (path.exists)
        rmdirRecurse(path);

    mkdir(path);

    BlockStorage.removeIndexFile(path);
    BlockStorage storage = new BlockStorage(path);
    Block[] blocks;

    blocks ~= getGenesisBlock();
    storage.saveBlock(blocks[$ - 1]);

    Transaction[] last_txs;
    foreach (block_idx; 0 .. BlockCount)
    {
        // create enough tx's for a single block
        auto txs = makeChainedTransactions(getGenesisKeyPair(), last_txs, 1);

        auto block = makeNewBlock(blocks[$ - 1], txs);
        storage.saveBlock(block);
        blocks ~= block;
        last_txs = txs;
    }

    //// load
    Block[] loaded_blocks;
    loaded_blocks.length = BlockCount + 1;
    foreach (idx; 0 .. BlockCount + 1)
        storage.readBlock(loaded_blocks[idx], idx);
    size_t idx;

    assert(loaded_blocks == blocks);
}

/// Copied over from agora.common.Transaction
private Transaction[] makeChainedTransactions (KeyPair key_pair,
    Transaction[] prev_txs, size_t block_count)
{
    import agora.common.Block;
    import std.conv;

    assert(prev_txs.length == 0 || prev_txs.length == Block.TxsInBlock);
    const TxCount = block_count * Block.TxsInBlock;

    // in unittests we use the following blockchain layout:
    //
    // genesis => 8 outputs
    // txs[0] => spend gen_tx.outputs[0]
    // txs[1] => spend gen_tx.outputs[1]...
    // ..
    // tx[9] => spend tx[0].outputs[0]
    // tx[10] => spend tx[1].outputs[0]
    // ..
    // tx[17] => spend tx[9].outputs[0]
    // tx[18] => spend tx[10].outputs[0]
    // ..
    // therefore the genesis block and the 1st block are unique here,
    // as the 1st block spends all the genesis outputs via separate
    // transactions, and subsequent blocks have transactions which
    // spend the only outputs in the transaction from the previous block

    Transaction[] transactions;

    // always use the same amount, for simplicity
    const Amount AmountPerTx = 40_000_000 / Block.TxsInBlock;

    foreach (idx; 0 .. TxCount)
    {
        Input input;
        if (prev_txs.length == 0)  // refering to genesis tx's outputs
            input = Input(hashFull(getGenesisTx()), idx.to!uint);
        else  // refering to tx's in the previous block
            input = Input(hashFull(prev_txs[idx % Block.TxsInBlock]), 0);

        Transaction tx =
        {
            [input],
            [Output(AmountPerTx, key_pair.address)]  // send to the same address
        };

        auto signature = key_pair.secret.sign(hashFull(tx)[]);
        tx.inputs[0].signature = signature;
        transactions ~= tx;

        // new transactions will refer to the just created transactions
        // which will be part of the previous block after the block is created
        if (Block.TxsInBlock == 1 ||  // special case
            (idx > 0 && ((idx + 1) % Block.TxsInBlock == 0)))
        {
            // refer to tx'es which will be in the previous block
            prev_txs = transactions[$ - Block.TxsInBlock .. $];
        }
    }
    return transactions;
}

Allow smooth transition from TokenNet to CoinNet for service providers (e.g. exchanges)

Currently we have a bunch of "service providers" which are already integrated with TokenNet, e.g. a few exchanges, CoinMarketCap, and other reputation tools. Moving away from TokenNet and the well supported ERC20 infrastructure means a lot of integration work, some of which should be completed before we flip the switch.

Some of this will be handled by the API server we are currently (2020-07-01) developing.

Tags: token migration

Ensure that validators sign with a frozen UTXO

As a developer, I need to make sure that signatures by validators are done using a frozen UTXO in order to make sure the validator has stake in the system.

Definition of Done:

• There is a way to link a signature and a frozen UTXO

Design and implement quorum balancing

As a developer, I need to know what algorithm to follow for quorum assignment in order for the protocol to prevent censorship while being open.

Description

We can consider two degenerate cases for quorum balancing:

• When there is a single quorum with all nodes in it, safety is maximized, but so is communication overhead;
• When each node live in their own quorum, safety does not exists but there is no communication: a less degenerate version would be having a ring where every node belongs to two quorums, each with another node (or: two two nodes quorums).

The goal of quorum balancing is to produce a network graph with optimal safety & overhead. Note that the nodes that are quorum intersections are likely to be higher-staking nodes, as they have more to lose.

Example of rules:

• There is no quorum with < 3 nodes;
• There is no quorum with > 7 nodes;
• A quorum economic value must be >= 350,000 BOA;
• The top X (X=5?) stakers are placed in different quorums and used as 'seed';

Definition of done:

• A new validator is assigned a quorum immediately;
• Quorums get regularly shuffled (how often?);
• There is a sufficient (how much?) quorum intersection;
• Validators are distributed among quorums in an unpredictable manner;
• Validators are distributed among quorums in a fairly balanced way;
• Quorum distribution only depends on globally available state;

Tentative approach:

• Come up with a simple algorithm based on the example rules posted above;
• Build test case for working and non-working (byzantine) validators;
• Compute the economic penalty (low & high bounds) => If node A cheats, home much does it cost ? Is it aligned with the power this node has in the system ?;
• Optimize
• Repeat

Publicly available Network map (backend)

As a user, I need to be able to have a view of the network, geographically.

Some of that might come from the monitoring tools ( #255 ) but it could be interesting to integrate that in the advanced administrative interface, or a user-project (see the links in #255 ).
If this is part of the administrative interface, we should explore integrating geoip.

Define and implement time locks on transactions

Basic script execution engine

We need to come up with a set of primitives for our script execution that will satisfy the needs of Flash.
Obviously, that means we need to come up with a list of primitives required for Flash to work.

Related (sub) issues:

• #243
• #242
• #245

In the future, we might want to split this issue into smaller, more granular tasks once the initial research is completed.

Define and implement rules for transaction replacement

Transaction replacement is what happens when a new transaction is received that compete with another transaction in the MemPool.
In this case, some design could allow the transaction with higher fees to win.
We need to come up with our rule on this. Initially we will allow a replacement fee to occur if the fee is 20% higher than the previous one, rounded up to the next digit.