nasa/hdtn

High-rate Delay Tolerant Network (HDTN) Software

High-rate Delay Tolerant Network

Overview

Delay Tolerant Networking (DTN) has been identified as a key technology to facilitate the development and growth of future space networks. DTN is an overlay network that uses the bundle protocol to connect once disparate one-to-one links. Bundles are the primary unit of data in a DTN, and can be of essentially any size. Existing DTN implementations have operated in constrained environments with limited resources resulting in low data speeds, and cannot utilize more than a fraction of available system capacity. However, as various technologies have advanced, data transfer rates and efficiency have also advanced. To date, most known implementations of DTN have been designed to operate on spacecraft.

High-rate Delay Tolerant Networking (HDTN) takes advantage of modern hardware platforms to substantially reduce latency and improve throughput compared to today’s DTN operations. HDTN maintains compatibility with existing deployments of DTN that conform to IETF RFC 5050. At the same time, HDTN defines a new data format better suited to higher-rate operation. It defines and adopts a massively parallel pipelined and message-oriented architecture, allowing the system to scale gracefully as its resources increase. HDTN’s architecture also supports hooks to replace various processing pipeline elements with specialized hardware accelerators. This offers improved Size, Weight, and Power (SWaP) characteristics while reducing development complexity and cost.

The HDTN team has completed an extensive review of the major DTN protocols and specifications. Our requirements analysis is captured in E-20225TM_Cover.indd (nasa.gov). Also see the HDTN Wiki and the HDTN User Guide for more information.

Architecture

HDTN is written in C++, and is designed to be modular. These modules include:

• Ingress - Processes incoming bundles.
• Storage - Stores bundles to disk.
• Router - Calculates the next hop for the bundle and sets links up/down based on contact plan.
• Egress - Forwards bundles to the proper outduct and next hop.
• Telemetry Command Interface - Web interface that displays the operations and data for HDTN.

Build Environment

Tested Platforms

• Linux
  • Ubuntu 20.04.2 LTS
  • Debian 10
  • RHEL (Red Hat Enterprise Linux) 8
  • ARM64
• Windows
  • Windows 10 (64-bit)
  • Windows Server 2022 (64-bit)
  • Windows Server 2019 (64-bit)
• macOS
  • Apple Silicon M2 on Ventura
  • Intel x64 on Ventura
• FreeBSD Intel x64
• OpenBSD Intel x64
• Raspbian
• ARM on x86

Dependencies

HDTN build environment requires:

• CMake version 3.12 minimum
• Boost library version 1.66.0 minimum, version 1.69.0 for TCPCLv4 TLS version 1.3 support
• ZeroMQ (tested with version 4.3.4)
• OpenSSL (recommended version 1.1.1). If OpenSSL is not available, disable OpenSSL support via the CMake cache variable ENABLE_OPENSSL_SUPPORT:BOOL
• On Linux: gcc version 9.3.0 (Debian 8.3.0-6)
• On Windows: see the Windows Build Instructions
• Target: x86_64-linux-gnu
• Tested platforms: Ubuntu 20.04.2 LTS, Debian 10, Windows 10 (Visual Studio), Windows Server 2019 (Visual Studio), Windows Server 2022 (Visual Studio), and Mac

These can be installed on Linux with:

• On Ubuntu

sudo apt-get install cmake build-essential libzmq3-dev libboost-dev libboost-all-dev openssl libssl-dev

• On RHel

sudo dnf install epel-release
sudo dnf install cmake boost-devel zeromq zeromq-devel gcc-c++ libstdc++-devel

macOS Dependencies

• ZeroMQ

On macOS, ZeroMQ needs to be built from source and installed in the /usr/local directory which can be done with:

sudo git clone https://github.com/zeromq/libzmq /usr/local/libzmq
cd /usr/local/libzmq
git checkout v4.3.4
sudo mkdir build
sudo chmod -R 777 .
cd build
cmake ..
make -j4
make install

• OpenSSL and gnutls

First, install Homebrew, which is a package manager for macOS:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Finally, install OpenSSL and gnutls using Homebrew:

brew install openssl gnutls

Known issue

• Ubuntu distributions may install an older CMake version that is not compatible
• Some processors may not support hardware acceleration or the RDSEED instruction, both ON by default in the cmake file

Notes on C++ Standard Version

HDTN build environment sets by default the CMake cache variable HDTN_TRY_USE_CPP17 to ON.

• If set to ON, CMake will test if the compiler supports C++17. If the test fails, CMake will fall back to C++11 and compile HDTN with C++11. If the test passes, CMake will compile HDTN with C++17.
• If set to OFF, CMake will compile HDTN with C++11.
• With Visual Studio 2017 version 15.7 and later versions, if C++17 is enabled, CMake will set the compile option /Zc:__cplusplus to make the __cplusplus preprocessor macro accurate for Visual Studio. (Otherwise, without this compile option, by default, Visual Studio always returns the value 199711L for the __cplusplus preprocessor macro.)
• Throughout the HDTN codebase, C++17 optimizations exist, usually within a #if(__cplusplus >= 201703L) block.

Optional X86 Hardware Acceleration

HDTN build environment sets by default two CMake cache variables to ON: USE_X86_HARDWARE_ACCELERATION and LTP_RNG_USE_RDSEED.

• If you are building natively (i.e. not cross-compiling) then the HDTN CMake build environment will check the processor's CPU instruction set as well as the compiler to determine which HDTN hardware accelerated functions will both build and run on the native host. CMake will automatically set various compiler defines to enable any supported HDTN hardware accelerated features.
• If you are cross-compiling then the HDTN CMake build environment will check the compiler only to determine if the HDTN hardware accelerated functions will build. It is up to the user to determine if the target processor will support/run those instructons. CMake will automatically set various compiler defines to enable any supported HDTN hardware accelerated features if and only if they compile.
• Hardware accelerated functions can be turned off by setting USE_X86_HARDWARE_ACCELERATION and/or LTP_RNG_USE_RDSEED to OFF in the CMakeCache.txt.
• If you are building for ARM or any non X86-64 platform, USE_X86_HARDWARE_ACCELERATION and LTP_RNG_USE_RDSEED must be set to OFF.

If USE_X86_HARDWARE_ACCELERATION is turned on, then some or all of the following various features will be enabled if CMake finds support for these CPU instructions:

• Fast SDNV encode/decode (BPv6, TCPCLv3, and LTP) requires SSE, SSE2, SSE3, SSSE3, SSE4.1, POPCNT, BMI1, and BMI2.
• Fast batch 32-byte SDNV decode (not yet implemented into HDTN but available in the common/util/Sdnv library) requires AVX, AVX2, and the above "Fast SDNV" support.
• Fast CBOR encode/decode (BPv7) requires SSE and SSE2.
• Some optimized loads and stores for TCPCLv4 requires SSE and SSE2.
• Fast CRC32C (BPv7 and a storage hash function) requires SSE4.2.
• The HDTN storage controller will use BITTEST if available. If BITTEST is not available, it will use ANDN if BMI1 is available.

If LTP_RNG_USE_RDSEED is turned on, this feature will be enabled if CMake finds support for this CPU instruction:

• An additional source of randomness for LTP's random number generator requires RDSEED. You may choose to disable this feature for potentially faster LTP performance even if the CPU supports it.

Storage Capacity Compilation Parameters

HDTN build environment sets by default two CMake cache variables: STORAGE_SEGMENT_ID_SIZE_BITS and STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB.

• The flag STORAGE_SEGMENT_ID_SIZE_BITS must be set to 32 (default and recommended) or 64. It determines the size/type of the storage module's segment_id_t Using 32-bit for this type will significantly decrease memory usage (by about half).
  • If this value is 32, The formula for the max segments (S) is given by S = min(UINT32_MAX, 64^6) = ~4.3 Billion segments since segment_id_t is a uint32_t. A segment allocator using 4.3 Billion segments uses about 533 MByte RAM), and multiplying by the minimum 4KB block size gives ~17TB bundle storage capacity. Make sure to appropriately set the totalStorageCapacityBytes variable in the HDTN json config so that only the required amount of memory is used for the segment allocator.
  • If this value is 64, The formula for the max segments (S) is given by S = min(UINT64_MAX, 64^6) = ~68.7 Billion segments since segment_id_t is a uint64_t. Using a segment allocator with 68.7 Billion segments, when multiplying by the minimum 4KB block size gives ~281TB bundle storage capacity.
• The flag STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB must be set to an integer of at least 1 (default is 1 and recommended). It determines the minimum increment of bundle storage based on the standard block size of 4096 bytes. For example:
  • If STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB = 1, then a 4KB * 1 = 4KB block size is used. A bundle size of 1KB would require 4KB of storage. A bundle size of 6KB would require 8KB of storage.
  • If STORAGE_SEGMENT_SIZE_MULTIPLE_OF_4KB = 2, then a 4KB * 2 = 8KB block size is used. A bundle size of 1KB would require 8KB of storage. A bundle size of 6KB would require 8KB of storage. A bundle size of 9KB would require 16KB of storage. If STORAGE_SEGMENT_ID_SIZE_BITS=32, then bundle storage capacity could potentially be doubled from ~17TB to ~34TB.

For more information on how the storage works, see module/storage/doc/storage.pptx in this repository.

Logging Compilation Parameters

Logging is controlled by CMake cache variables:

• LOG_LEVEL_TYPE controls which messages are logged. The options, from most verbose to least verbose, are TRACE, DEBUG, INFO, WARNING, ERROR, FATAL, and NONE. All log statements using a level more verbose than the provided level will be compiled out of the application. The default value is INFO.
• LOG_TO_CONSOLE controls whether log messages are sent to the console. The default value is ON.
• LOG_TO_ERROR_FILE controls whether all error messages are written to a single error.log file. The default value is OFF.
• LOG_TO_PROCESS_FILE controls whether each process writes to their own log file. The default value is OFF.
• LOG_TO_SUBPROCESS_FILE controls whether each subprocess writes to their own log file. The default value is OFF.

Getting Started

Build HDTN

If building on Windows, see the Windows Build Instructions. To build HDTN in Release mode (which is now the default if -DCMAKE_BUILD_TYPE is not specified), follow these steps:

• export HDTN_SOURCE_ROOT=/home/username/HDTN
• cd $HDTN_SOURCE_ROOT
• mkdir build
• cd build
• cmake ..
• make -j8
• make install

Note: By Default, BUILD_SHARED_LIBS is OFF and hdtn is built as static To use shared libs, edit CMakeCache.txt, set BUILD_SHARED_LIBS:BOOL to ON and add fPIC to the Cmakecache variable: CMAKE_CXX_FLAGS_RELEASE:STRING=-03 -DNDEBUG -fPIC

Note: If building HDTN from a cloned git repo (with a .git directory in the source root), HDTN's CMake will get the Git SHA-1 hash so that not only will HDTN's logger print the HDTN version, but also it will print HDTN's latest commit hash. However, if building HDTN from zip, tar, tar.bz2, or tar.gz, HDTN's CMake can detect the commit hash within the archive file's comment header by running cmake with the following argument that points to the archive file from where HDTN was extracted from:

• cmake -DHDTN_ARCHIVE_FILE=/path/to/hdtn.zip ..

ARM Platforms

HDTN has been tested on various ARM platforms such as Raspberry Pi, Nvidia Jetson Nano and an Ampere Altra Q64-30 based server. To build HDTN in a native ARM environment add the -DCMAKE_SYSTEM_PROCESSOR flag to the cmake command. This flag removes x86 optimizations and the x86 unit test. Shared libraries are disabled for ARM builds by default.

• cmake .. -DCMAKE_SYSTEM_PROCESSOR=arm

Run HDTN

Note: Ensure your config files are correct, e.g., The outduct remotePort is the same as the induct boundPort, a consistant convergenceLayer, and the outducts remoteHostname is pointed to the correct IP adress.

You can use tcpdump to test the HDTN ingress storage and egress. The generated pcap file can be read using wireshark.

• sudo tcpdump -i lo -vv -s0 port 4558 -w hdtn-traffic.pcap

In another terminal, run:

• ./runscript.sh

Note: The contact Plan which has a list of all forthcoming contacts for each node is located under module/router/contact_plans/contactPlan.json and includes source/destination nodes, start/end time and data rate. Based on the schedule in the contactPlan the Router sends events on link availability to Ingress and Storage. When the Ingress receives Link Available event for a given destination, it sends the bundles directly to egress and when the Link is Unavailable it sends the bundles to storage. Upon receiving Link Available event, Storage releases the bundles for the corresponding destination and when receiving Link Available event it stops releasing the budles.

There are additional test scripts located under the directories test_scripts_linux and test_scripts_windows that can be used to test different scenarios for all convergence layers.

Run Unit Tests

After building HDTN (see above), the unit tests can be run with the following command within the build directory:

• ./tests/unit_tests/unit-tests

Run Integrated Tests

After building HDTN (see above), the integrated tests can be run with the following command within the build directory:

• ./tests/integrated_tests/integrated-tests

Routing

The Router module runs by default Dijkstra's algorithm on the contact plan to get the next hop for the optimal route leading to the final destination, then sends a RouteUpdate event to Egress to update its Outduct to the outduct of that nextHop. If the link goes down unexpectedly or the contact plan gets updated, the router will be notified and will recalculate the next hop and send RouteUpdate events to egress accordingly. An example of Routing scenario test with 4 HDTN nodes was added under $HDTN_SOURCE_ROOT/test_scripts_linux/Routing_Test

Bundle Protocol Security (BPSec)

BPSec library module interacts with Ingress and Egress modules and HDTN utility applications such as BPGen/BPSendFile and BPSink/BPReceiveFile to apply and process integrity and confidentiality security services based on the security role. BPSec is enabled by default with Bundle Protocol Version 7 and it requires OpenSSL FIPS module. It can be disabled by setting the CMakeCache.txt variable ENABLE_BPSEC:BOOL to OFF.

Examples of tests with BPSec enabled using intergity and confidentiality services were added under $HDTN_SOURCE_ROOT/test_scripts_linux/BPSec_Tests. BPSec configuration files which include security policy rules and the security operation failure events handling are located under $HDTN_SOURCE_ROOT/config_files/bpsec.

TLS Support for TCPCL Version 4

TLS Versions 1.2 and 1.3 are supported for the TCPCL Version 4 convergence layer. The X.509 certificates must be version 3 in order to validate IPN URIs using the X.509 "Subject Alternative Name" field. HDTN must be compiled with ENABLE_OPENSSL_SUPPORT turned on in CMake. To generate (using a single command) a certificate (which is installed on both an outduct and an induct) and a private key (which is installed on an induct only), such that the induct has a Node Id of 10, use the following command:

• openssl req -x509 -newkey rsa:4096 -nodes -keyout privatekey.pem -out cert.pem -sha256 -days 365 -extensions v3_req -extensions v3_ca -subj "/C=US/ST=Ohio/L=Cleveland/O=NASA/OU=HDTN/CN=localhost" -addext "subjectAltName = otherName:1.3.6.1.5.5.7.8.11;IA5:ipn:10.0" -config /path/to/openssl.cnf

Note: RFC 9174 changed from the earlier -26 draft in that the Subject Alternative Name changed from a URI to an otherName with ID 1.3.6.1.5.5.7.8.11 (id-on-bundleEID).

• Therefore, do NOT use: -addext "subjectAltName = URI:ipn:10.0"
• Instead, use: -addext "subjectAltName = otherName:1.3.6.1.5.5.7.8.11;IA5:ipn:10.0"

To generate the Diffie-Hellman parameters PEM file (which is installed on an induct only), use the following command:

• openssl dhparam -outform PEM -out dh4096.pem 4096

Audio and Video Streaming

HDTN has media streaming applications BpSendStream and BpRecvStream. BpSendStream intakes an RTP stream from an outside source, bundles RTP packets, and sends the bundles to a BP network. BpRecvStream performs the inverse operation by intaking a stream of bundles, extracting the RTP packets, and forwarding the RTP stream to an IP network. Once built, the bpsend_stream executable can be used to ingest media such as RTP packets and mp4 files, and send them to a DTN network as bundles over any supported convergence layer. Similarly, the bprecv_stream executable can be used to receive bundles from a DTN and reassemble the media.

Streaming is disabled by default. It can be enabled by setting the CMakeCache.txt variable ENABLE_STREAMING_SUPPORT:BOOL to ON.

Examples of File Streaming tests were added under $HDTN_SOURCE_ROOT/test_scripts_linux/Streaming. Before running the tests set the HDTN_SOURCE_ROOT the hdtn root directory and LD_LIBRARY_PATH to /usr/local/lib/x86_64-linux-gnu

Streaming dependencies need also to be installed. See below example of Gstreamer dependencies installation steps.

Streaming Dependencies installation (Gstreamer)

1. Install OS dependencies

sudo apt-get update -y && sudo apt-get install -y pkg-config \
    flex bison git python3-pip ninja-build libx264-dev \
    cmake build-essential libzmq3-dev libboost-dev libboost-all-dev openssl libssl-dev

2. Install meson and the directory where it was installed to PATH

sudo pip3 install meson

3. Clone, build, and install Gstreamer

git clone https://github.com/GStreamer/gstreamer.git 

cd gstreamer && meson setup build 

meson configure build && meson compile -C build 

ninja -C build install 

An alternative way to install all these dependencies:

sudo aptitude install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio

Viewing Multimedia Streams

The following command can be used to view video for H264 Stream:

gst-launch-1.0 -v -e udpsrc address=127.0.0.1 port=8989 ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! queue ! h264parse ! decodebin ! autovideosink

The following command can be used to view video with audio for H264 Stream:

gst-launch-1.0 -v -e udpsrc address=127.0.0.1 port=8989 ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)MP2T, payload=(int)96" ! rtpmp2tdepay ! queue ! tsparse ! decodebin ! autovideosink

Displaying received File Stream

The following command can be used to play the received video file to compare it to the original file

ffplay <name of file>

Web User Interface

This repository comes equiped with code to launch a web-based user interface to display statistics for the HDTN engine. It relies on a dependency called Boost Beast which is packaged as a header-only library that comes with a standard Boost installation. The web interface will use OpenSSL (if found by CMake) since the web interface supports both http as well as https, and hence both ws (WebSocket) and wss (WebSocket Secure). If OpenSSL is not found, the web interface will only support http/ws. The web user interface is enabled by default at compile time. If the web user interface is not desired, it can be turned off by setting the CMakeCache.txt variable USE_WEB_INTERFACE:BOOL to OFF. The web interface can be built with CivetWeb (statically linked without SSL support) instead of Boost Beast in order to reduce HDTN binary file size. The HDTN CMake will automatically download and build CivetWeb and link to it statically. Simply set CMake cache option WEB_INTERFACE_USE_CIVETWEB to ON (or run cmake with argument -DWEB_INTERFACE_USE_CIVETWEB:BOOL=ON). If you are building without access to the internet, you can download version 1.16 source zip file of CivetWeb from GitHub and set the CMake cache variable CIVETWEB_PREDOWNLOADED_ZIP_FILE to point to the filesystem path of the downloaded zip file.

Now anytime that HDTNOneProcess runs, the web page will be accessible at http://localhost:8086 and an alternative "system view" gui based on D3.js will be accessible at http://localhost:8086/hdtn_d3_gui/

Beta 2.0 Web User Interface

There is a new Web User Interface under development which leverages a modern, single-page-app web framework. The existing Web User Interface is in the process of being migrated over to the new framework and can be used in a Beta capacity if desired. The following are functional components in the new GUI:

• Configuration - Allows you to generate HDTN configuration files in a web form

Building

Dependencies

The new Web User Interface leverages Node Package Manager for handling dependencies and building the web files to host. Download Node before attempting to build.

Build Steps

HDTN needs to be configured to host the new UI. First set the GUI version environment variable:

export HDTN_GUI_VERSION=2

At this point HDTN needs to be rebuilt, re-run the cmake and make commands. If Node Package Manager is installed correctly and HDTN_GUI_VERSION is set to 2, cmake will automatically build and host the new GUI. After HDTN is rebuilt, simply start HDTN as normal, the GUI will continue to be accessible via a web browser at http://localhost:8086.

Simulations

HDTN can be simulated using DtnSim, a simulator for delay tolerant networks built on the OMNeT++ simulation framework. Use the "support-hdtn" branch of DtnSim which can be found in the official DtnSim repository. Currently HDTN simulation with DtnSim has only been tested on Linux (Debian and Ubuntu). Follow the readme instructions for HDTN and DtnSim to install the software. Alternatively, a pre-configured Ubuntu VM is available for download here (the username is hdtnsim-user and password is grc). More Details about the installation steps can be found here

Docker Instructions

First make sure docker is installed.

apt-get install docker

Check the service is running

systemctl start docker

There are currently two Dockerfiles for building HDTN, one for building an Oracle Linux container and the other for building an Ubuntu. This command will build the Ubuntu one:

docker build  -t hdtn_ubuntu containers/docker/ubuntu/.

The -t sets the name of the image, in this case hdtn_ubuntu. Check the image was built with the command:

docker images

Now to run the container use the command:

docker run -d -t hdtn_ubuntu

Check that it is running with:

docker ps

To access it, you'll need the CONTAINER_ID listed with the ps command

docker exec -it container_id bash

Stop the container with

docker stop container_id

The same container can either be restarted or removed. To see all the containers Use:

docker ps -a

These can still be restarted with the run command above. To remove one that will no longer be used:

docker rm container_id

Docker Compose Instructions

Docker compose can be used to spin-up and configure multiple nodes at the same time. This is done using the docker compose file found under HDTN/containers/docker/docker_compose

cd containers/docker/docker_compose

This file contains instructions to spin up two containers using Oracle Linux. One is called hdtn_sender and the other hdtn_receiver. Start them with the following command:

docker compose up

On another bash terminal these can be accessed using the command:

docker exec -it hdtn_sender bash
docker exec -it hdtn_receiver bash

This setup is perfect for running a test between two hdtn nodes. An example script for each node can be found under HDTN/tests/test_scripts_linux/LTP_2Nodes_Test/. Be sure to run the receiver script first, otherwise the sender will have nowhere to send to at launch.

Kubernetes Instructions

Download the dependencies

sudo apt-install docker microk8s

The first step is to create a docker images to be pushed locally for kubernetes to pull:

docker build docker/ubuntu/. -t myhdtn:local

Check that it was built:

docker images

Next we build the image locally and inject it into the microk8s image cache

docker save myhdtn > myhdtn.tar
microk8s ctr image import myhdtn.tar

Confirm this with:

microk8s ctr images ls

Now we deploy the cluster, the yaml must reference the injected image name

microk8s kubectl apply -f containers/kubernetes/hdtn_10_node_cluster.yaml 

There should now be ten kubernetes pods running with HDTN. See them with:

microk8s kubectl get pods

To access a container in a pod, enter the following command:

microk8s kubectl exec -it container_name -- bash

When you're finished working with this deployment, delete it using:

microk8s kubectl delete deployment hdtn-deployment

Use the get pods command to confirm they've been deleted

microk8s kubectl get pods

hdtn's Issues

LtpEngine has no way to correlate a session source with its session ID

The API for LtpEngine currently has callbacks to provide status on outgoing transfer sessions, all indexed by a Ltp::session_id_t object. But there is no way to correlate any of those session ID with the data which originated the outoging session (from any of the TransmissionRequest... functions). I added an outside synchronization on the user side to guarantee each TransmissionRequest correlates to a single TX session ID, but this will not scale to high TX session rates and is unnecessary wait time.

It would be useful if the TransmissionRequest... functions allowed either the user to define a correlation object (just a object-identity token would do, either a std::shared_ptr<SessionToken>) or if possible you could make use of C++ promise/future infrastructure to have the TransmissionRequest... functions return an std::future<Ltp::session_id_t> which would the be fulfilled in the work thread when the session ID is assigned.

I can prototype this last idea and prove it out if it seems like a good path to take.

bpsink core dumps when fails to bind

In case the bind port is already used, boost::system::system_error is thrown which leads to terminate and a coredump:

[ bpsink   ][ info ]: This is HDTN version 1.0.0
[ bpsink   ][ info ]: starting..
[ bpsink   ][ info ]: All LTP UDP engines can receive a maximum of 65535 bytes per packet
terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
what():  bind: Address already in use
Aborted (core dumped)

Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f562805a859 in __GI_abort () at abort.c:79
#2  0x00007f56282e5911 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f56282f138c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f56282f13f7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f56282f16a9 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00005642b00fec2b in void boost::throw_exception<boost::system::system_error>(boost::system::system_error const&) ()
#7  0x00005642b00f8451 in boost::asio::detail::do_throw_error(boost::system::error_code const&, char const*) ()
#8  0x00005642b00f8332 in boost::asio::detail::throw_error(boost::system::error_code const&, char const*) ()
#9  0x00005642b017404a in boost::asio::basic_socket_acceptor<boost::asio::ip::tcp, boost::asio::executor>::basic_socket_acceptor<boost::asio::io_context>(boost::asio::io_context&, boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const&, bool, std::enable_if<std::is_convertible<boost::asio::io_context&, boost::asio::execution_context&>::value, void>::type*) ()
#10 0x00005642b0180b75 in TcpclV4Induct::TcpclV4Induct(boost::function<void (std::vector<unsigned char, PaddedMallocator<unsigned char> >&)> const&, induct_element_config_t const&, unsigned long, unsigned long, boost::function<void (unsigned long, Induct*, void*)> const&, boost::function<void (unsigned long, Induct*, void*)> const&) ()
#11 0x00005642b0194cd8 in boost::enable_if_<!boost::is_array<TcpclV4Induct>::value, std::unique_ptr<TcpclV4Induct, std::default_delete<TcpclV4Induct> > >::type boost::make_unique<TcpclV4Induct, boost::function<void (std::vector<unsigned char, PaddedMallocator<unsigned char> >&)> const&, induct_element_config_t const&, unsigned long const&, unsigned long const&, boost::function<void (unsigned long, Induct*, void*)> const&, boost::function<void (unsigned long, Induct*, void*)> const&>(boost::function<void (std::vector<unsigned char, PaddedMallocator<unsigned char> >&)>const&, induct_element_config_t const&, unsigned long const&, unsigned long const&, boost::function<void (unsigned long, Induct*, void*)> const&, boost::function<void (unsigned long, Induct*, void*)> const&) ()
#12 0x00005642b0193d21 in InductManager::LoadInductsFromConfig(boost::function<void (std::vector<unsigned char, PaddedMallocator<unsigned char> >&)> const&, InductsConfig const&, unsigned long, unsigned long, unsigned long, boost::function<void (unsigned long, Induct*, void*)> const&, boost::function<void (unsigned long, Induct*, void*)> const&) ()
#13 0x00005642b011e555 in BpSinkPattern::Init(std::shared_ptr<InductsConfig>&, std::shared_ptr<OutductsConfig>&, bool, cbhe_eid_t const&, unsigned int, unsigned long, unsigned long) ()
#14 0x00005642b00dd20f in BpSinkAsyncRunner::Run(int, char const* const*, bool volatile&, bool) ()
#15 0x00005642b00d87fb in main ()

This should probably be handled nicer.

No visibility into whether the LtpUdpEngineManager is actually listening

I ran into a configuration problem that caused the LtpUdpEngineManager to fail to bind to the desired port, but other than logging the failure the application has no visibility into the failure to do anything about it (i.e. fail out with an error exit code).

HDTN/common/ltp/src/LtpUdpEngineManager.cpp

Line 64 in 9f543f2

m_udpSocket.bind(boost::asio::ip::udp::endpoint(boost::asio::ip::udp::v4(), M_MY_BOUND_UDP_PORT)); // //if udpPort is 0 then bind to random ephemeral port

Having too much dynamic behavior in an object constructor poses problems like this because a constructor has no return value (and it appears that you are restricting the use of exceptions in HDTN, which is the only way for a constructor to fail).

Instead of having the bind happen in the constructor and the constructor call its own Start() method, it would be easier to manage if the bind actually happened in the Start() method which could return a boolean "success, it worked" status value. The application could then use that return value to decide how to handle failures to bind.
If this is changed, there may need to be a symmetric change to the Stop() method to close (unbind) the socket if necessary.

Magic value present in many SDNV codec functions

The value 10 appears many times in Ltp.cpp related to the maximum size of an encoded SDNV. This makes understanding the purpose of the "magic value" more difficult.

I recommend to make a global declaration of the semantics of that value in Sdnv.h such as:

/// The largest possible encoding of a 64-bit value
constexpr int SdnvMaximumEncodedSize = 10;

and then search-and-replace "10" with "SdnvMaximumEncodedSize" (and where applicable "11" with "1 + SdnvMaximumEncodedSize" and similar).

JsonSerializable class is missing virtual base destructor

The class JsonSerializable is a virtual base class but it is missing its virtual destructor function. This can cause derived classes to fail to clean up properly because of unspecified behavior.

Security block parameter/result decoding is fragile

The current decoding behavior in Bpv7AbstractSecurityBlock::DeserializeIdValuePairBpv7() requires that the implementation have type-specific handling for all possible security context IDs, which is an incredibly difficult ask of HDTN. Even if #51 is implemented, there are still situations where the BPA needs to know that a BCB is targeting specific blocks even if the BCB has a security context which itself cannot be handled.

For example, the proposed BPSec COSE context uses a CBOR map for the AAD Scope parameter.

Unfortunately, the handling of generic ASB parameter or result values requires a recursive CBOR decoder. Such a decoder is available from QCBOR or others on the cbor.io listing, but this would add a build dependency to HDTN.

Since the parameters and results are at the end of the BTSD, it would also be possible for the HDTN decoder to have an indication on the ASB that it was able to decode up to the parameters successfully, but not parameters or results. This would allow BPSec processing to be able to use the target list and context ID without needing to actually use any of the parameters or results.

Graceful handling of Memory Full Conditions

Add functionality to gracefully handle memory full conditions that may occur during operations. If the amount of data being received by HDTN cannot be handled, the suggested operation is to push back on the data senders to send less data or a less desirable option is to lose some data. The desire is to avoid catastrophic or lock up conditions of HDTN requiring operator restart.

Unrecognized Option --dest-uri-eid=ipn:2.1

I am currently working with the support-hdtn branch of DTNSim and encountered an issue when trying to run the hdtn_demo example. When I execute the router command as specified in the documentation, I receive an error message indicating that the option --dest-uri-eid=ipn:2.1 is unrecognized.

The specific error message is:

[ router ][ info ]: This is HDTN version 1.0.0
[ router ][ error]: unrecognised option '--dest-uri-eid=ipn:2.1'

Testing Platform

Operating System: Ubuntu 20.04
OMNeT++ Version: 5.7

HDTN GUI Enhancement - Pending Bundles Table

It would be very useful if the web UI had a way to list what bundles are "pending" in storage. Maybe something like add a button that would trigger a popup near the storage UI that would show

Destination | Bundle Count | Storage Usage | Next Expiration
ipn:123.123 | 15 | 15MB | 00:00:04:12 remaining
ipn:123.40 | 10 | 148MB | 00:00:00:05 remaining
ipn:124.123 | 108 | 1.23GB | 00:00:00:34 remaining
ipn:132.1 | 9 | 57KB | 00:06:13:57 remaining

Allow changing a peer MTU at runtime

Unless there is a technical limitation preventing it, it would be helpful to allow changing the MTU (for segments and for reports) for a single peer at runtime. Similar to how the delay times can now be changed, it could be an option of whether the MTU change affects current sessions or just future sessions.

It appears that there is already an LtpEngine::SetMtuReportSegment() but nothing similar for data segment MTU. And the current function implicitly only applies to future sessions.

Error Linking CXX executable ltp-file-transfer

I am building HDTN on a raspberry pi using the raspberry pi os terminal. I am currently in the HDTN/build directory and have executed cmake .. -DCMAKE_SYSTEM_PROCESSOR=arm
After doing so, I executed make -j2 and it started building. However once it got to 41% it produced this error:

[ 40%] Building CXX object common/ltp/CMakeFiles/ltp-file-transfer.dir/apps/ltp_file_transfer/src/LtpFileTransferRunner.cpp.o
[ 41%] Linking CXX executable ltp-file-transfer
[ 41%] Built target ltp-file-transfer
make: *** [Makefile:149: all] Error 2

How can I resolve this issue?
Thank you.

Defer data retransmission with out-of-order report segments

When the network is causing out-of-order segment reception it is possible that one or more synchronous reception reports are received either out-of-order or within a short time window, possibly followed by an asynchronous reception report (see #23) indicating that the full red part was received. To avoid unnecessary data retransmission the sending engine should defer sending gap-filling data segments until some small window of time after the last reception report for that session.

Upon receiving a reception report the sending engine should (in addition to sending a report ack segment):

1. Add (as a set union) the report's claims to the total claims seen for the session.
2. If there are gaps in the claims, start a retransmission timer for the session
3. Otherwise, if there is a retransmission timer running stop it (there is no need to retransmit in this case)

When the retransmission timer expires (i.e. there are still gaps to send) then send data segments to cover the remaining gaps for the session.

The result of this procedure is that the sending engine will not send either duplicate data segments to cover gaps in earlier-sent reports which are claimed in later-sent reports. In the case of no loss but highly out-of-order this will result in no unnecessary data retransmission to occur.

Allow changing peer time delays at runtime

Similar to #27 for address changes, it is necessary to be able to update the OWLT and margin for a peer. I have no need to modify these separately, so the interface would be something like:

void LtpEngine::SetDelays(const boost::posix_time::time_duration & oneWayLightTime, const boost::posix_time::time_duration & oneWayMarginTime)

It looks like this is a straightforward change to LtpEngine and LtpTimerManager classes to propagate the change down. This would affect only newly started sessions, but for my use case that's fine because the sessions are very short lived. A more extensive change may be needed if the goal is to change member variable state and to update running timers.

Release bundles using bundle metadata

Currently, bundles are released only by the final destination. This can cause issues for scheduling algorithms that could schedule different routes for bundles in storage that have the same final destination, as bundles could be sent down different routes. For example, consider a scenario with 4 nodes, A, B, C, and D. For two distinct sets of bundles generated at A with final destination D, our scheduler allocates two different routes to ensure deliverability, such as route 1: A->B->D, and route 2: A->C->D. When A->B is executed for route 1, it is possible for bundles originally intended for route 2 to be sent along this route. These bundles may have different requirements that could make them undeliverable if sent to the wrong next hop/contact.

Requested additions:

• The ability to explicitly mark bundles, and release bundles by certain markings.

• And/Or, releasing bundles based on a combination other metadata, such as source, creation time ranges, final destination.

RFC 9174 certificate profile SAN

The certificate profile in RFC 9174 changed from the earlier -26 draft in that the Subject Alternative Name changed from a URI to an otherName with ID 1.3.6.1.5.5.7.8.11 (id-on-bundleEID).

To generate CSR/certificate with this other name form using OpenSSL the syntax changes from subjectAltName = URI:ipn:10.0 to subjectAltName = otherName:1.3.6.1.5.5.7.8.11;IA5:ipn:10.0. The encoded form is still an IA5String. On the verifying side, the entity needs to match the otherName type and specific id-on-bundleEID OID, then extract the IA5String value and compare with the expected URI.

BPSec decoding mishandles larger parameter and result sizes

For some reason there is a case where the size of a decoded chunk of BTSD is cast down to an 8-bit value, when the size could actually be arbitrarily large. This causes later parts of the block to fail to decode and/or indicates that decoding failed even if it did not.

A patch to remove the cast and fix this problem is below.

--- a/common/bpcodec/src/codec/Bpv7BpSecExtensionBlocks.cpp
+++ b/common/bpcodec/src/codec/Bpv7BpSecExtensionBlocks.cpp
@@ -597,7 +597,7 @@ bool Bpv7AbstractSecurityBlock::DeserializeIdValuePairBpv7(uint8_t * serializati
         }
     }
 
-    numBytesTakenToDecode = static_cast<uint8_t>(serialization - serializationBase);
+    numBytesTakenToDecode = (serialization - serializationBase);
     return true;
 }
 bool Bpv7AbstractSecurityBlock::IsEqual(const id_value_pairs_vec_t & pVec1, const id_value_pairs_vec_t & pVec2) {

Access metadata for incoming bundles through API

We would like to be able to see bundle metadata for incoming bundles through API. Fields include:

• Final destination
• Source
• Creation time
• Lifespan
• Priority
• Bundle size

LTP/UDP rate limiting

Neither LTP nor UDP provide intrinsic rate limiting behavior. When operating LTP-over-UDP over a known-rate link it is helpful for the LTP engine itself to rate limit segments to individual peer engines. The limit could be as simple as a Token Bucket for each peer that decrements by the segment size when one is sent and increments at a desired rate (the configure parameter) and refuses to send a segment until its size is available. This is really to limit data segments, so the other segment types could be counted in decrementing but ignored for send gating (to avoid unnecessary report or ACK delays).

I'm planning on implementing a simple rate limit for prototyping, so this ticket is just to inform that this work is happening outside of the normal HDTN project. I'd be happy to coordinate changes later on if it is working properly.

No way to bind UDPCL to a specific source port

The current outduct_element_config_t has a way to ensure a specific LTP local port with ltpSenderBoundPort but there is no way to do the equivalent thing with the outgoing UDP source port.

The current behavior uses a fixed bind port of "0" (meaning choose an ephemeral port) here

HDTN/common/udp/src/UdpBundleSource.cpp

Line 242 in 73e30db

m_udpSocket.bind(boost::asio::ip::udp::endpoint(boost::asio::ip::udp::v4(), 0)); //bind to 0 (random ephemeral port)

but this could be made a configuration from e.g. an option udpSenderBoundPort.

Additionally, the current open() and bind() within UdpBundleSource::OnResolve() doesn't actually need to wait for the peer address resolution. Both open and bind can happen (and catch errros) in that class constructor.

If source port selection is allowed, the following should be enabled just before the bind() call to enable multiple sockets to use the same sort port

m_udpSocket.set_option(boost::asio::socket_base::reuse_address(true));

CMake header install path and collisions

The change for #8 does install headers but on a POSIX environment it clutters (and potentially collides with existing headers in) the main include directory (/usr/local/include or /usr/include) with all of the HDTN headers together.

Convention is to install headers under a library-named prefix directory to avoid potential collisions. Could this ticket be reopened and a solution worked out to install under include/HDTN or similar when UNIX is set? This could be a top-level logic to set some variable (e.g. HDTN_INCLUDEDIR) based on the platform and then use that variable for the install() destination.

There is also a way to use the PUBLIC_HEADER property of a target to associate includes with a specific target, but this is not necessary here because this project isn't using export targets.

Allow changing a peer engine address at runtime

LTP itself does not bind a transfer session to a specific network address or UDP port; sessions are independent of how the segments arrive at the engine.

Currently it's not possible to alter the UDP endpoint of an LtpUdpEngine so to switch the network address of a peer engine it requires to remove the old LtpUdpEngine objects and create new ones. But this leaves a gap of service over time and also interrupts any sessions already in progress.

It would be better if the LtpUdpEngine provided a public interface to modify the m_remoteEndpoint member. It seems like that does not need to be synchronized with anything, since that endpoint is used only when sending outgoing datagrams.

As a prototype I attempted to add a public interface

void LtpUdpEngine::setEndpoint(const boost::asio::ip::udp::endpoint & remoteEndpoint)

as well as adding a public endpoint resolution function to the manager to be consistent with the engine construction

boost::asio::ip::udp::endpoint LtpUdpEngineManager::resolveEndpoint(const std::string & remoteHostname, const uint16_t remotePort)

and that seems to do the job. But I didn't do any extensive testing on this technique.

Defer synchronous reception report with out-of-order data segments

When red part data is segmented and delivered to the receiving engine out-of-order, the checkpoint(s) and EORP can be received before the earlier-in-block data segments. If a synchronous report is sent immediately upon receiving the checkpoint there will be data segments in-flight and about to be delivered that will be seen as reception gaps in the report.

Instead of sending the synchronous report immediately upon receiving a checkpoint segment the receiving engine should have some more complex logic:

1. If the report segment bounds are fully claimed (i.e. no gaps) then the report can be sent immediately.
2. Otherwise, the engine should wait a very small window of time for gaps to be filled. The delay time should reset upon any data segments which fill gaps.

In a situation with no loss but lots of out-of-order delivery this will have exactly the same number of reports, they will just be sent when the full checkpointed bounds of data have been received. In a situation with loss this will send reports with the fewest size of claim gaps.

Failure to process any one BTSD will cascade to entire bundle failure

Currently, within the BundleViewV7::Load() function, the calls to Virtual_DeserializeExtensionBlockDataBpv7() are not isolated and a failure to process any single block's block type-specific data (BSTD) will result in the entire bundle considered as failed to load.

There is no reason why the failure should cascade upward like this, it is an expected situation that any particular BPA might receive a block that it cannot process. There are Block Processing Control Flags to explicitly indicate how the BPA should handle a block that it cannot process. Only one specific flag indicates that the bundle is deleted if a block cannot be processed.

The correct behavior is to treat a failure of Virtual_DeserializeExtensionBlockDataBpv7() as a "block cannot be processed" condition per RFC 9171 and behave according to what the associated Block Processing Control Flags indicate.

Cannot change CMake build type

Some earlier change hard-coded the CMAKE_BUILD_TYPE so now it cannot be selected from the command line.

HDTN/CMakeLists.txt

Line 7 in 9f543f2

set(CMAKE_BUILD_TYPE Release)

This should be removed or at least guarded somehow so that setting from the cmake command still works.

Use modern CMake for library dependencies

Especially for boost, which is split into many components, using dependencies as targets makes it much easier and cleaner to manage a multi-platform library like HDTN.

Instead of having to manually use variables like Boost_LIBRARIES and Boost_INCLUDE_DIRS in the right places, use the pre-existing targets like Boost::program_options as link requirements for specific HDTN libraries. This doesn't affect build content or speed, only for maintainability. I can post in comments some examples of using this. It also removes conditionals when complex dependencies are used.

Using this technique also then makes the include and link properties transitive from dependencies to built libraries, simplifying the users of the HDTN libraries.

Mark entire class as exported for convenience

Currently there are many classes that have all of their public functions marked with the LTP_LIB_EXPORT macro (or similar for other libraries). It is possible to mark the entire class as exported to save editing whenever functions are added, similar to

class LTP_LIB_EXPORT LtpUdpEngineManager { ... };

This is just a suggestion to ease maintenance, it wouldn't affect the result of function symbols being exported in the libraries.

Discrepancy in different LtpEngine::TransmissionRequest postconditions

The different overloads/forms of LtpEngine::TransmissionRequest seem to behave differently about sending segments after starting a new outgoing transfer session. The function TrySendPacketIfAvailable() is called for the forms that take a parameter of transmission_request_t but is not called for the other (bare session parameters) forms.

A workaround is to always use the transmission_request_t forms, but it seems like this is a bug if the other overloads don't work consistently.

Recommend to place large numbers of function parameters into a POD struct

For some functions like LtpUdpEngineManager::AddLtpUdpEngine() and the related constructor for LtpUdpEngine there are very many parameters and it makes identifying which position corresponds with what parameter very difficult. In cases like this with many (more than ~4) parameters, it is helpful to follow a pattern to create a plain old data (POD) C++ struct (with its language-provided default constructor/assignment/copy operators) to use as a parameter container and then use a single (or significantly fewer) parameters to the actual functions.

For example, the above function (and its parameter) would look like:

struct LtpUdpEngineConfig {
  uint64_t thisEngineId = 0;
  uint64_t remoteEngineId = 0;
  ...
};

LtpUdpEngineManager::AddLtpUdpEngine(const LtpUdpEngineConfig &config) {
  ...
  auto newEngPtr = boost::make_unique<LtpUdpEngine>(config);
  ...
}

and using it would then be a more obvious matter of:

LtpUdpEngineConfig config;
// set parameters in any order and the name is obvious from this source
config.thisEngineId = ...
config.remotePort = ...
...
auto newEngPtr = engineManager->AddLtpUdpEngine(config);
...

It's important that the struct either have a default constructor or have default values for all fundamental-type values so that it's deterministic.

HDTN no license

The HDTN project currently has no license property set on it, no LICENSE file in the repository, and no license tagging within source file (a few that I inspected). It would be good to at least have a concrete project license set and LICENSE file present to know what the real terms of use are.

Allow modifying the rate limit to a peer engine

Can an API be added to the LtpEngine to change the rate limit after construction?

My use case involves dynamic rate limits (per peer ID) over time. The current API can set the initial rate but cannot (AFAICT) change without removing/adding the peer engine (which will interrupt all in-progress sessions).

Use list-relative paths for includes to allow sub-project use of HDTN

There are a few cases where CMAKE_SOURCE_DIR (the project root) is used as a path anchor which breaks if HDTN is used as a sub-project from some other project source tree. If these were changed to use CMAKE_CURRENT_LIST_DIR instead it would fix the issue and otherwise have the same behavior (independent of whether HDTN is the top project or a sub-project).

Attribute `volatile` does not provide thread safety

There are several places in HDTN where the volatile attribute keyword is used for variables which appear to be shared across multiple threads. The volatile attribute affects compiler optimization but does not guarantee thread safety of the values. These should be made to either use std::atomic<> types or be guarded by a mutex (or equivalent barrier).

I noticed this specifically in the LtpUdpEngine but I suspect it is used elsewhere. In these cases the replacement of volatile uint64_t ... with std::atomic<std::uint64_t> would fix this issue.

Allow same-engine transfers

There are currently three places in LtpEngine.cpp that guard against sessionId.sessionOriginatorEngineId == M_THIS_ENGINE_ID and early-return from processing segments. There is no prohibition against sending segments to your own engine, and for our purposes some transfers are made to the local BPA.

If there is no technical reason to prohibit these, it would be helpful to remove these guards. I did a quick experiment of just removing the three checks and things seem to go through properly.

LTP session state leak with non-checkpoint data timeout

The current LTP engine and session keeping logic does not provide a mechanism for a session to timeout except for red data checkpoint (and EORP, EOB) reports. Unfortunately, the LTP spec (both IETF RFC 5326 and CCSDS 734.1-B-1) are silent about correct engine behavior in the case of a non-checkpoint timeout situation.

I have two simple ways to produce this problem:

1. Send an LTP segment type 0 (red non-checkpoint) with a new Session ID and arbitrary data.
2. Send an LTP segment type 4 (green non-EOB) with a new Session ID and arbitrary data.

In both cases the LTP engine will establish new session state and wait for all red data (with associated reports and their timeouts) or all green data. But if these never arrive, for whatever reason, the LTP engine will persist this session state indefinitely and also never give an indication to the application that this has occurred.

It would be very helpful to the application to have some kind of "no further data received for the session" cancelation from the receiver, with its associated API callbacks for RX session cancellation. The LTP specs don't reserve a cancel reason for this, but the existing code 0 "Client service canceled session" could be used. I could implement this as a client-service-level timeout for green segments not for red segments, as the API doesn't indicate individual red segments to the client. This timeout would be something like "maximum time between receiving any segments associated with a session before cancelling" with a timer reset any time any segment is received for a given RX Session ID.

No CMake runtime inspection to enable optimized function calls

The single CMake option USE_X86_HARDWARE_ACCELERATION assumes that all of the related optimized C functions enabled by that option are both available and working on the target platform. In my case the target is a VM with an non-configurable CPU type, where some are available and some are not. This means I cannot enable the option and I lose some of the benefit of the functions that are available and working.

I recommend to use CMake to check the availability of individual optimized functions and then use separate preprocessor defines to enable/disable the use at a more fine grained level. I will add some comments with examples that I've briefly prototyped. I'm more focused on my own needs now, so won't dig to much into all of the possible combinations.

Performance vs bundle size

Hi, I saw your paper "New Horizons for a Practical and
Performance-Optimized Solar System Internet" and saw that you were using a bundle size of 0.5MB for the performance results. I am interested in what the maximum bundles per second would be, so having results of bundles per second vs bundle size would be very cool.

In the meantime, any clue what the bundles per second for 1 byte payload bundles would be? Obviously it depends on the system you are running but I'm curious what HDTN is capable of.

Guarantee reception report when full red part data is received

In the case where data segments arrive out-of-order and with enough delay that even a deferred reception report (#22) has some gap in it, there may be a point where the full set of data is received (with no retransmission, only out-of-order in the first flight). When this happens the receiving engine must guarantee that an "asynchronous reception report" (one not in response to a checkpoint) is sent so that the sender knows to stop any data retransmission that hasn't yet gone out.

I don't know if this is currently implemented, but I can prepare a small test fixture to exercise this situation.

Problem when building with shared libraries

When building HDTN with BUILD_SHARED_LIBS enabled there is a set of linker errors near the end of the build sequence. It looks like some internal libraries might not have their linker dependencies set properly, or there is some problem with public transitive link dependencies. All of the errors appear to come from the same two libraries and all seem to be related to *Telemetry_t classes not being exported properly.

The specific errors are:

/usr/bin/ld: deps/HDTN/common/outduct_manager/liboutduct_manager_lib.so: undefined reference to `vtable for StcpOutductTelemetry_t'
/usr/bin/ld: deps/HDTN/common/outduct_manager/liboutduct_manager_lib.so: undefined reference to `vtable for LtpOutductTelemetry_t'
/usr/bin/ld: deps/HDTN/common/induct_manager/libinduct_manager_lib.so: undefined reference to `vtable for StcpInductConnectionTelemetry_t'
/usr/bin/ld: deps/HDTN/common/induct_manager/libinduct_manager_lib.so: undefined reference to `vtable for TcpclV3InductConnectionTelemetry_t'
/usr/bin/ld: deps/HDTN/common/outduct_manager/liboutduct_manager_lib.so: undefined reference to `vtable for TcpclV4OutductTelemetry_t'
/usr/bin/ld: deps/HDTN/common/induct_manager/libinduct_manager_lib.so: undefined reference to `vtable for UdpInductConnectionTelemetry_t'
/usr/bin/ld: deps/HDTN/common/outduct_manager/liboutduct_manager_lib.so: undefined reference to `vtable for TcpclV3OutductTelemetry_t'
/usr/bin/ld: deps/HDTN/common/outduct_manager/liboutduct_manager_lib.so: undefined reference to `vtable for OutductTelemetry_t'
/usr/bin/ld: deps/HDTN/common/outduct_manager/liboutduct_manager_lib.so: undefined reference to `vtable for UdpOutductTelemetry_t'
/usr/bin/ld: deps/HDTN/common/induct_manager/libinduct_manager_lib.so: undefined reference to `vtable for InductConnectionTelemetry_t'
/usr/bin/ld: deps/HDTN/common/induct_manager/libinduct_manager_lib.so: undefined reference to `vtable for TcpclV4InductConnectionTelemetry_t'
/usr/bin/ld: deps/HDTN/common/induct_manager/libinduct_manager_lib.so: undefined reference to `vtable for LtpInductConnectionTelemetry_t'

Cannot install HDTN libraries with CMake

The current CMake configuration is missing the necessary install() directives to get the runtime and development files from HDTN installed. This makes it difficult to use HDTN as a library dependency of an external project.

I have some prototyped patches that I have made to both:

1. Use generator expressions in the include paths following best practices of modern CMake
2. Use install directives to install the built libraries and development files to the standard (system) paths. This also allows using standard techniques such as DESTDIR to control the install prefix.

Confusion about multiple outgoing LTP peers

Currently, the LtpUdpEngineManager::AddLtpUdpEngine() interface seems to have some logical issues when using one LTP engine to communicate with multiple peer engines. The inbound configuration separates "own engine ID" from "peer engine ID" but the outbound does not identify the target peer engine ID.

This seems to mean that a single LtpUdpEngineManager can only transmit blocks to a single peer engine, which means that each peer must be associated with a unique local UDP port. There is no logical reason why this restriction should be present, and I think that changing the signature from

AddLtpUdpEngine(const uint64_t thisEngineId, const uint64_t expectedSessionOriginatorEngineId, ...)

to

AddLtpUdpEngine(const uint64_t thisEngineId, const uint64_t remoteEngineId, ...)

would avoid this problem and allow each peer a separate LtpEngine outduct. This would then affect the internal logic and the logic for GetLtpUdpEnginePtr() and some other public functions, but I think these would all be improvements in usability.

Documentation PRs?

How do you feel about documentation for the internals of the project's main components? Starting with LTP, while the RFCs cover the high-level logic, the implementation could perhaps benefit from some structured documentation to get up to speed quicker/make the impact of changes across shared state of sub-components more obvious.

Is the implementation/interface stable enough at this point for documentation to make sense?

If so, what's your input on structure/content?

Multiple TX canceled callbacks are called if multiple cancel segments recevied

The LtpEngine calls the callback set with SetTransmissionSessionCancelledCallback() multiple times per session if multiple "Cancel segment from the receiver" segments are received by the transmitting engine. This is causing bad behavior in my client which doesn't guard against these behaviors.

The API doesn't document either way, but it would be ideal if the actual LtpEngine interface would guarantee that exactly one of the completed or canceled callbacks was called per session (i.e. not both and and not more than one of either). This makes the logic of the client using the LTP engine simpler.

If this isn't possible the documentation should reflect the guarantees either way.

BPSec blocks cannot have multiple results per target

The current definition of Bpv7AbstractSecurityBlock::security_results_t is defined to be a single vector of type-value pairs, which then gets encoded so that each target corresponds with a length-1 array of results per target. This restriction to allow only a single result type-value pair per target is artificial and not part of RFC 9172.

HDTN/common/bpcodec/include/codec/bpv7.h

Line 506 in 5159736

typedef id_value_pairs_vec_t security_results_t;

A more correct definition would be that the results are more like

typedef std::vector<id_value_pairs_vec_t>> security_results_t;

where each outer vector element corresponds with a single target, and the inner vector contains all results for that target block.

This will require implementation changes for serializing, deserializing, and comparing result sets so I'm not attempting a patch here.

Utilization of same next hop when not expected

Using figure 8 (HDTN Four Nodes STCP Routing test) from the user guide as reference.

The desired setup is to have node 4 act as a final destination sink. Node 1 will send data to its final destination using node 3 as a router. In parallel we would also like to have another source node (node 5) use node 2 as a router to send data to node 4. Two independent data pipelines sending data to the same final destination, node 4.

The above can be done without issue. However, once we add an additional next hop to one of the source nodes both source nodes use the same next hop. This is not expected behavior and the hope was to continue to replicate the two parallel independent pipelines as previously described. For example holding all else true, if node 1's HDTN config file has node 2 simply added to it then the end result will have both node 1 and 5 use node 2 as the router. We would like this behavior only after manipulating node 3 to become unavailable so as to allow the router to update optimal path choice based on link status.

Can what is being described be done? We are using TCPCL and the default routing algorithm which I believe is Dijkstra's algorithm according to the README. When we add the next hop the corresponding contact is created in the contact plan with similar settings so as to not skew one contact being favored over another.

Installing HDTN on Raspberry Pi (Model B 8 GB RAM)

./runscript.sh: 12: ./build/common/bpcodec/apps/bpsink-async: not found
./runscript.sh: 19: ./build/module/hdtn_one_process/hdtn-one-process: not found
./runscript.sh: 24: ./build/common/bpcodec/apps/bpgen-async: not found

killing bpgen1...
./runscript.sh: 30: kill: No such process


killing egress...
./runscript.sh: 32: kill: No such process


killing bpsink2...
./runscript.sh: 34: kill: No such process

I ran ./runscript.sh and these are the errors I am receiving. I followed the HDTN video on NASA's website and did make install and looked at the READme for ARM platforms.

Before running runscript, I did the sudo make install command, bc when I did 'make install' it gave me this error:

CMake Error at common/bpcodec/cmake_install.cmake:46 (file):
  file INSTALL cannot set permissions on "/usr/local/lib/libbpcodec.a":
  Operation not permitted.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)


make: *** [Makefile:100: install] Error 1

No configuration control over static/shared libraries

All of the HDTN libraries are currently unconditionally built as static libraries. It would be more convenient as a user if HDTN followed the standard CMake pattern of using a BUILD_SHARED_LIBS project option to control this. The default option can be OFF to retain the current behavior.