intersectmbo/ouroboros-network

Specifications of network protocols and implementations of components running these protocols which support a family of Ouroboros Consesus protocols; the diffusion layer of the Cardano Node.

Ouroboros-Network

This repository contains the core components of the network code for the Cardano node. It is a dependency when building the node from the cardano-node repository.

The following graph shows the dependency tree. The top-level package is ouroboros-consensus-diffusion which is part of ouroboros-consensus

  flowchart TD
    A[network-mux]                   --> B[network / Win32-network]
    M[cardano-ping]                  --> A
    D[ouroboros-network-framework]   --> A
    D                                --> E[ouroboros-network-api]
    E                                --> H[typed-protocols]
    G                                --> H[typed-protocols]
    G                                --> E
    F[ouroboros-network]             --> D
    F                                --> G[ouroboros-network-protocols]
    I[ouroboros-consensus-diffusion] --> F
    J[cardano-client]                --> F
    K[ntp-client]                    --> B

   click A "https://ouroboros-network.cardano.intersectmbo.org/network-mux/" _blank
   click M "https://ouroboros-network.cardano.intersectmbo.org/cardano-ping/" _blank
   click D "https://ouroboros-network.cardano.intersectmbo.org/ouroboros-network-framework/" _blank
   click E "https://ouroboros-network.cardano.intersectmbo.org/ouroboros-network-api/" _blank
   click F "https://ouroboros-network.cardano.intersectmbo.org/ouroboros-network/" _blank
   click G "https://ouroboros-network.cardano.intersectmbo.org/ouroboros-network-protocols/" _blank
   click I "https://github.com/intersectmbo/ouroboros-consensus/" _blank
   click J "https://ouroboros-network.cardano.intersectmbo.org/cardano-client/" _blank
   click K "https://ouroboros-network.cardano.intersectmbo.org/ntp-client/" _blank
   click H "https://github.com/input-output-hk/typed-protocols/" _blank

• network-mux - implementation of a general network multiplexer.
• ouroboros-network-api - shared API between network and consensus components.
• ouroboros-network-framework - low-level network components, e.g. snockets, connection manager, inbound governor, handshake mini-protocol, network simulator.
• ouroboros-network-protocols - implementation of all /node-to-node/ & /node-to-client/ protocols. Also contains a testing library which is implementing various applications for testing purposes.
• ouroboros-network- top-level integration of all network components also defines node-to-node and node-to-client API. It contains the implementation of the outbound governor.
• ouroboros-network-mock & ouroboros-network-testing - shared testing code.
• ntp-client - an ntp client (used by cardano-wallet).
• cardano-ping - a library which implements the core functionality of cardano-cli ping command.
• cardano-client - a subscription for node-to-client which wants to connect to a cardano-node.

Libraries:

• monoidal-synchronisation - a small standalone package which contains synchronisation primitives.

Ouroboros-Network Documentation

We have two documents which describe various levels of the networking layer of the Cardano Shelley implementation:

• Introduction to the design of Data Diffusion and Networking of Cardano Shelley

  This document explains the technical requirements and key constraints for the networking layer of the Cardano Shelley implementation of Ouroboros Praos. This is a design document.

• The Shelley Networking Protocol

  This document is a technical specification of the networking protocol. It includes serialisation formats, necessary details of the multiplexer and technical specifications of mini-protocols used by either node-to-node and node-to-client flavours of the protocol.

• Haddock documentation

• Official Cardano Documentation

  Specifically the section "Explore Cardano" is helpful, since it talks about the Cardano Architecture, Cardano Design Rationale, the Cardano Network, etc. Although the Cardano documentation is less detailed, it is a good place to start and refresh some more basic concepts about Cardano.

Work progress

On a biweekly basis, we publish updates on cardano-updates. We are also tracking our current work in the Ouroboros Network Project. Our quarterly goals are published in the [Cardano Node Product Backlog][cardano-node-backlog].

For Contributors

The contributing guide is available here. The style guide is available here. The code of conduct is available here.

Ouroboros-Network API

The API consists of three layers:

• mini-protocol APIs, which are GADTs for each mini-protocol under Ouroboros.Network.Protocol (defined in ouroboros-network-protocols package); this hides heavy type machinery of session types. One only needs the Peer or [PeerPipelined] type when one is using runPeer or [runPeerPipelined] function and each protocol exposes a function to create it (e.g. chainSyncClientPeer. There is also API which allows to run a Peer or [PipelinedPeer] with limits (i.e. per state timeouts & per message size limits).

• callback MiniProtocolCb. The callback is wrapped in OuroborosApplication GADT which allows to differentiate the initiator/responder (or client/server) callbacks.

• versioning which is a map from version numbers to the above callbacks and version data (the tricky part here is that version data type can be different between different versions; there is a simple way of building this map using a semigroup). You can use simpleSingletonVersion if your application does not depend on negotiated version data. However, Ouroboros.Network.NodeToNode and Ouroboros.Network.NodeToClient expose API which hides versioning from the caller.

Demo applications

• demo-chain-sync
• cardano-ping
• mux-demo
• demo-ping-pong
• demo-ntp-client

Instructions

To run a demo type:

cabal run <DEMO_NAME> --

After -- you will need to pass arguments, when a demo is run without arguments it will specify what arguments it needs.

ouroboros-network's Issues

Rewrite BlockFetch protocol using typed-protocols framework

Also implement a pipelined sender together with corresponding tests:

• using pipelined connect (in simuation in IO)
• over data channel and bytestring channel (in simulation and IO)
• over a local pipe

Document ReqResp pipeliened protocol

Document:

• the state machine
• transitions and messages
• include a short prose description of the protocol

A test fails conditionally if assertions are enabled

How to reproduce:

• Compile the code enabling assertions by bulding with ghc-options: -O0 the library component;
• Do a full rebuild with cabal new-build (or similar)
• Run dist-newstyle/build/x86_64-osx/ghc-8.2.2/ouroboros-network-0.1.0.0/t/tests/build/tests/tests --quickcheck-replay=720104 (or similar).

On my machine, on master the failure or success of the blockGenerator (IO) test is bound to whether or not I compile with assertions enabled. Perhaps there is a latent chain validation problem we are not catching?

Byron Diffusion System

net_impl_old_protocol
Proof of concept instantiation of the old diffusion system with some backend to store and serve blocks

Typed Framework for pipelined protocols

Develop a typed framework for pipelined protocols.

The criteria for submission are:

• type level description for protocols
• PingPong protocol with tests
  • using direct (in simulation and in IO)
  • connect and pipelined connect (in simulation and in IO)
  • using a channel over data channel, bytestring channel (in simulation and in IO)
  • using a local pipe
• codec, channels and a driver

Multiplexing Protocols

net_impl_multiplexing_protocols
Integrate and test the various protocols with the multiplexing, serialisation in simulation

This involves getting all the existing mini-protocols running together over the multiplexing mechanism either in simulation, or over a local TCP socket pair.

Completion critera for this work package:

• extension of each mini-protocol's pipe test to run over multiplexing over a pipe, in isolation
• run each mini-protocol's pipe test to run over multiplexing over a pipe, concurrently with ping-pong
• run all mini-protocols together concurrently

Failures in Peer Selection Policy

net_impl_peer_selection_failure
Incorporate failures into the peer selection policy and test in simulation

Block Download Mini Protocol

net_impl_protocol_block_download
Implement the mini-protocol for block body download (in sim)

This is the application level protocol for fetching ranges of blocks from another peer.

Ultimately this is a relatively simple protocol, however the final design of this protocol must be informed by the block fetching logic. See #66. Its expression relies on the framework for expressing mini-protocols. See #61.

The completion criteria for this work package is that:

• it has an interface that is deemed satisfactory for use by the block fetch logic. This relies on PoC work on the block fetch logic.
• tests in simulation that it can correctly fetch ranges of a chain, and correctly reports failure on ranges not in the current chain
• tests in simulation that multiple requests can be outstanding at once

Testing the mini protocol in isolation should cover multiple integration levels.

This epic is split into 3 tasks:

• #123 - block download protocol together with integration tests:

• direct,
• via typed Peer with connect

• #124 - codec for block download protocol messages, including coherence tests for that codec
• #125 - add more integration tests:

• codec in simulation
• via local pipe

Non-deterministic test failure in "pipe sync demo"

Travis failed an unrelated PR, and I can reproduce this on master:

tests -p 'pipe sync demo' --quickcheck-tests 10000
ouroboros-network
  Pipe
    pipe sync demo: FAIL
      Exception: ExceptionInLinkedThread ThreadId 39562 consumer end of input
      CallStack (from HasCallStack):
        error, called at src/Ouroboros/Network/Pipe.hs:82:50 in ouroboros-network-0.1.0.0-inplace:Ouroboros.Network.Pipe

1 out of 1 tests failed (197.14s)

The failure is non-deterministic and rare (yay ☹️), but quite reliably reproducible with a high enough number of tests.

It turns out that tasty does not give a seed in the case of an async failure. I experimented with some infrastructure to change that:

```data TestFailedException = TestFailedException SomeException
  deriving (Show)

instance Exception TestFailedException where
  displayException (TestFailedException e) = displayException e

catchTestExceptions :: IO a -> IO a
catchTestExceptions act = catch act (throwIO . TestFailedException)

ioProperty' :: Testable prop => IO prop -> Property
ioProperty' = ioProperty . catchTestExceptions

and that did give me a seed, but sadly the non-determinism seems to come from the test itself, not anything QC generated, because with a given seed it was still not reproducible.

Have not added the above code; not sure if it's worth adding somewhere or not.

Transaction Submission Mini Protocol

net_impl_protocol_tx_submission
Implement the mini-protocol for transaction submission (in sim)

This only has to cover transaction submission for the federated case initially. It can rely on the assumption that if B gets blocks from A, then B can submit transactions to A. This work package does not cover end to end transaction relay, just the point-to-point submission protocol.

Testing the mini protocol in isolation should cover multiple integration levels:

• direct,
• via typed Peer with connect
• via typed PeerSender with connectPipelined
• codec tests,
• via message channel,
• via byte channel (with codec),
• via local pipe

Port DNS Peer Discovery From Byron Codebase

net_impl_dns_peer_discovery
Port the DNS peer discovery code from the Byron codebase and fit it to new interface

Block Download: round trip tests which are using a codec

This is a sbutask of #61 and it depends on #124

Add tests:

• using codec in simulation
• via a local pipe

Implement PermBFT -> Praos transitional protocol

Document low level wire protocol, serialisation and frameing.

Write the necessary documentation for the wire protocol, serialisation and frameing as developed in #59.

Document BlockFetch pipeliend protocol

Document:

• the state machine
• transitions and messages
• include a short prose description of the protocol

Use Byron adapter to download blocks to new storage mechanism

Context

There is a new block storage layer. We'd like to populate it with some blocks.

Decision

Let's download some blocks from existing *nets like mainnet, and put them into the new block storage layer!

Acceptance criteria

1. We can download blocks from existing *nets like mainnet and put them into the new block storage layer.

PR

None as of yet

Consequences

We will be able to download blocks from existing *nets like mainnet, and put them into the new block storage layer.

Add a namespace for all the code.

At some point this code is going to be integrated with other parts of new code base that exists in other repositories. At that point, this code will need to be name spaced. Since this will need to be name spaced at some point I would suggest doing it sooner rather than later.

All code in the src directory should be prefixed with either Ouroboros, Cardano.Ouroboros or something similar. Similarly, test modules in the test directory should be name spaced with Test.X where X is whatever is chosen for the name space for code in the src directory.

Download Block Logic

net_impl_block_download_logic
Implement the logic for which blocks to download given the chains, peers and policy

This is a critical component of the network and consensus design that continuously monitors the node's current chain length, the available candidate chains of headers, fetched blocks and decides if and when to download block bodies, and which peers to download them from. It is critical to the header/body split within the consensus layer.

The function of the component is to look at the chains of headers that are available from each peer and decide which blocks to download. Downloaded blocks are handed back to the consensus layer to consider in extending the chain (ie to validate and run ledger rules). The download logic must take into account the maximum desired level of network concurrency to keep network resource use within acceptable levels. When resource constraints force choices, it must try to choose which peers to download blocks from to maximise the chances of fetching blocks within the deadlines imposed by the consensus layer. The policies embedded in this component

This work package involves creating a first working version of this component, but not the final version. The completion criteria for this work package are

• a first working PoC of the block download logic, using the block fetching protocol in simulation with test block providers
• a test demonstrating that it can succesfully fetch all the blocks for a longest chain
• document the design of this critical component and the policies it does or will need to embed and how these lead to the desired network level outcomes
• specify further system level simulation tests that will need to be done later to demonstrate that the network level outcomes
• specify further tests in simulation to exercise difficult cases in the block download logic, e.g. with slow or disconnecting peers, or peers that provide incorrect responses

Block Download Protocol

This is a subtask of #62 with a smaller scope:

• the block download protocol
• tests using direct and connect functions

Multiplexing, Versioning, Serialization and Framing Scheme

net_impl_multiplexing_serialization
Design and implement the multiplexing, versioning, serialisation and framing scheme for running the mini-protocols on a single bearer channel, in simulation

This takes a set of mini-protocol implementations (client and server sides) and runs them all over a single byte level channel by multiplexing at an appropriate level (either message or message chunk). This necessitates some design for framing (the units of multiplexing), and serialisation for frames. It also requires a minimal design for any opening messages including protocol version negotiation.

Consensus_Integrate with Network Layer

DeltaQ Measure

net_impl_deltaQ_measure
Implement measurement of delta-Q to be used in peer selection and block download decisions and test in sim with generators of known delta-Q

Rewrite ChainSync protocol using typed-protocols framework

Integrate and Test Protocols in Sim and TCP

net_impl_integrate_tcp
Integrate and test the combination of protocols in simulation and TCP, including timeouts and failures

Proxy to Sync Chains Between Byron and Shelley Nodes

net_impl_dns_peer_discovery
Implement a proxy to sync chains and send transactions between the old and new nodes

Integrate pre-Shelley era ledger.

Peer Selection Interface

net_impl_peer_selection
Design and implement a peer selection interface and initial peer selection policy for the federated setting and test it in simulation

Chain Synchronization Mini Protocol Tests

Sub task of: #61
It should include tests on all levels (slack discussion):

1. direct,
2. via typed Peer with connect
3. via message channel,
4. via byte channel (with codec),
5. via local pipe
6. codec tests

Consensus_Incorporate Network Layer (Header/Body Split)

Block Download: codec

Implement codecs for block download messages together with consistency tests for that codec.

Take another look at trying to remove the ExceptT from the HasFS stack

.. and ideally turning it into a record.

Document ChainSync protocol

Document:

• the state machine
• transitions and messages
• include a short prose description of the protocol

Chain Synchronization Mini Protocol

net_impl_protocol_chain_sync
Implement the mini-protocol for chain header synchronisation (in sim).

This is the application level protocol for continuously syncronising a chain of headers from one peer to another peer.

There have been several iteration of this protocol, along with interations in the framework for expressing all the "mini" protocols. The completion criteria for this work package is that

• we have the protocol expressed in a satisfactory framework
• tests in simulation that it can continuously synchronise a changing chain
• including basic binary encoding to/from a byte level channel
• it is used in working tests in the consensus layer

Testing the mini protocol in isolation should cover multiple integration levels:

• direct,
• via typed Peer with connect
• codec tests
• via message channel,
• via byte channel (with codec),

5. via local pipe

Interpret Bearer Channel in TCP

net_impl_tcp
nstantiate/interpret the bearer channel multiplexing on TCP and unix domain socket

Consensus_Exec Spec Ourborous Genesis

Integrate Header Syncing and Block Download with Block Download Logic

net_impl_sync_download
Integrate and test the header syncing and block download protocols with the block download logic

Fix demo playground

The demo playground has been updated in #118 to use the new consensus layer node, but it has a number of problems that need to be fixed/investigated:

• CLI parser is broken, and when using Praos, stake is not updated correctly at epoch boundary. This was fixed in a somewhat hap-hazard manner by Lars and Philipp in preparation for the demo. These fixes are available at philipp/demo-kludge and need to be ported.
• The demo does not seem to work when running with less than 3 nodes. This needs to be investigated.
• Cleanup of the named pipes needs to be fixed.
• We should make the protocol a CLI argument. This involves some type level hackery.
• We should make it possible to use the system RNG.
• Introduce "fake" quantified constraints Reintroduce use of QuantifiedConstraints

Integrate and Test Peer Selection

net_integrate_peer_selection
Integrate and test the peer selection

P2P Literature Review

net_design_p2p_literature_review

Review the literature on P2P peer discovery and select a least-bad approach.
Write a literature review report.

Make immutable DB epoch type configurable

In Ouroboros.Storage.Immutable.DB there is

-- This is just a placeholder as we don't have (yet) a proper 'Epoch' type in
-- this codebase.
type Epoch = Word

I suggest we make the index type a parameter of the immutable DB. For cardano-sl applications we can set it to be Word64, since that's the size of epoch indices there.

Fialures, Timeouts, Keepalive

net_impl_failures_timeouts_keepalive
Design and implement a coherent strategy for protocol and I/O failures, including the consistent use of timeouts and keepalive

Praos test failure

Alexander Vieth [3 days ago]
Got a test failure today

```Dynamic chain generation
  simple Praos convergence: FAIL (2.66s)
    *** Failed! Falsifiable (after 21 tests)
...
Use --quickcheck-replay=82375 to reproduce```


In case we're not already aware


Marc Fontaine [2 days ago]
fails with 750fcce53b7aa1fcb6da0f5075f0ae also```

Existing Desing and Prototype for Broadcast and Chain Selection

net_existing_design_and_proto
Existing design and prototype for broadcast and chain selection

This work package covers the prototype code and design concepts arising from the earlier design and prototyping phase done in Summer 2018. The prototype covered an iteration of the chain syncing protocol, and a sketch of how to use this to make a network of nodes that can relay a chain between them, all running in simulation.

The completion criteria was simply that the code was in a suitable working state to share and discuss with the consensus team. See chain_prototype_review_networking.

Document Wire Protocol

net_doc_wire_protocol
Document the initial design of the versioning, serialisation, framing and multiplexing for the protocol

Also includes the documentation for each mini protocol, including state transitions, prose descriptions and message serialisation formats.

This will be an addition to the general network design document in the github repo.

The definition of done for this work package is document review and sign-off from a majority of the document authors.

Rewrite ReqResp (tx-submission) protocol using typed-protocols framework

Also write a pipelined sender together with corresponding tests:

• using pipelined connect (in simuation in IO)
• over data channel and bytestring channel (in simulation and IO)
• over a local pipe

Consensus_On Disk Representation

DeltaQ and Block Download Logic

net_impl_dns_peer_discovery
Integrate the delta-Q measurements with the the block download logic and test delta-Q based decisions

Implement permissive BFT

Per discussion with @edsko, I'll implement an instance of the Ouroboros.Consensus.Protocol.Abstract.OuroborosTag class for the permissive BFT protocol.

net_impl_multiplexing_serialization

Description

Design and implement the multiplexing, versioning, serialization and framing scheme for running the mini-protocols on a single bearer channel, in simulation

Business value

Estimate

`MonadTimer` does not compile on windows

PR #156 fails with the following error:

Configure flags:
--prefix=/nix/store/gnr0mxv2hb3wymm8065w7sy5j1yn20bg-io-sim-classes-0.1.0.0-lib-io-sim-classes-x86_64-pc-mingw32 lib:io-sim-classes --package-db=clear --package-db=/nix/store/06g6xr27qb61mg8mkhl3gnp3n9852ga1-io-sim-classes-0.1.0.0-lib-io-sim-classes-config-x86_64-pc-mingw32/package.conf.d --with-ghc=x86_64-pc-mingw32-ghc --with-ghc-pkg=x86_64-pc-mingw32-ghc-pkg --with-hsc2hs=x86_64-pc-mingw32-hsc2hs --with-gcc=x86_64-pc-mingw32-cc --with-ld=x86_64-pc-mingw32-ld --with-ar=x86_64-pc-mingw32-ar --with-strip=x86_64-pc-mingw32-strip --enable-executable-stripping --enable-library-stripping --hsc2hs-option=--cross-compile --hsc2hs-option=--via-asm 
Configuring library for io-sim-classes-0.1.0.0..
building
Preprocessing library for io-sim-classes-0.1.0.0..
Building library for io-sim-classes-0.1.0.0..
[1 of 6] Compiling Control.Monad.Class.MonadFork ( src/Control/Monad/Class/MonadFork.hs, dist/build/Control/Monad/Class/MonadFork.o )
[2 of 6] Compiling Control.Monad.Class.MonadST ( src/Control/Monad/Class/MonadST.hs, dist/build/Control/Monad/Class/MonadST.o )
[3 of 6] Compiling Control.Monad.Class.MonadSTM ( src/Control/Monad/Class/MonadSTM.hs, dist/build/Control/Monad/Class/MonadSTM.o )
[4 of 6] Compiling Control.Monad.Class.MonadSay ( src/Control/Monad/Class/MonadSay.hs, dist/build/Control/Monad/Class/MonadSay.o )
[5 of 6] Compiling Control.Monad.Class.MonadTimer ( src/Control/Monad/Class/MonadTimer.hs, dist/build/Control/Monad/Class/MonadTimer.o )

src/Control/Monad/Class/MonadTimer.hs:19:1: error:
    Could not find module ‘GHC.Event’
    Use -v to see a list of the files searched for.
   |
19 | import qualified GHC.Event as GHC (TimeoutKey, getSystemTimerManager,
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
builder for '/nix/store/964iq5xmq8259gcl235x6vxh3hgfdmzx-io-sim-classes-0.1.0.0-lib-io-sim-classes-x86_64-pc-mingw32.drv' failed with exit code 1