nordix/meridio

Facilitator of attraction and distribution of external traffic within Kubernetes via secondary networks

Meridio

Meridio is an Open Source project providing capabilities to facilitate attraction and distribution of external traffic within Kubernetes. It operates on layer 3/4 to provide traffic distribution via so-called secondary networking upholding separation from the traffic distributed on the default "primary" network within the cluster.

In order to attract traffic towards different services exposed by user applications, service addresses (VIPs) are announced to gateways via different kinds of routing protocols monitored by link-supervision mechanisms available in Meridio.

In addition, Meridio enables development and usage of highly configurable network services thanks to traffic classifiers which allows users to separate the traffic into multiple groups. For now, only a TCP/UDP/SCTP stateless (Maglev) load-balancer is supported as network service.

Through an gRPC API hosted in a sidecar container, user applications can control at runtime external networks and network services attached to the pod, and thus start or stop traffic towards the pod.

Getting Started

• High Level Overview
• Architecture and Concepts
• Quick Installation / Demo
• Deployment
• Contributing
• Frequently Asked Questions

Features

Secondary Networking

Isolation of the traffic and the network is a key aspect for Meridio, it improves the resiliency, the security, the decongestion and the minimization of the traffic. In addition, each network can have its own specificities and capabilities provided via different dataplane methods (VPP, OVS, Kernel, accelerated network...) which can carry exotic protocols. Meridio is for now, providing dataplane only for TCP, UDP and SCTP traffic on IPv4 and IPv6.

External Traffic Attraction

Announcing service-addresses (VIP-addresses) to the external gateways permits the traffic to be attracted towards the different services exposed by the target applications. Frontend services provide different connectivity mechanisms such as VLANs or host network interfaces to connect the network services to the gateways. In addition, Multus can offer decoupling of connectivity mechanisms while supplying a whole variety of network interfaces through CNI plugins. To announce the service and ensure the link-supervision, routing protocols are used. For now, BGP and Static BFD are supported.

Network Services

Development of new network services with more or different capabilities (network acceleration, SCTP load-balancing...) is possible within Meridio. As the current default network service, a no-NAT stateless Load-Balancer is offered by Meridio. It provides traffic classification (based on 5-tuple) to steer traffic into multiple different instances of the network service applications can subscribe to.

Runtime Configuration

Meridio users have the flexibility to adjust the network services on the fly as they desire. Traffic attractors, streams gathering traffic into logical groups and traffic classifiers (flows) can be added, removed and updated without any redeployment of the resources, and with no traffic disturbance. Individually, each user pods have the ability to select the traffic to consume at runtime which will produce secondary network reorganization to cover the user pods needs and requests.

Community

Slack

The team is reachable on slack for any question, feedback or help: Slack

Events

• Cloud Native Telco Day EU 2022 - Network Service Mesh at Scale for Telco Networking - Youtube
• KubeDay India 2023 - Use Meridio to Create Network Services for Telco Use Cases Using Secondary Networking for Kubernetes - Youtube

Prerequisites

To run Meridio, the following are required:

• Kubernetes 1.21+
• Spire
• Network Service Mesh 1.5+
• Linux Kernel 5.15+

meridio's Issues

Fix intermittent interface connectivity issue

Sometimes traffic can't be sent over created links, potential upstream bug

Update LB program to nfqueue-loadbalancer

Replace the LB program with nfqueue-loadbalancer

Recovery/resiliency testing (FEs/proxies/targets)

Verify that the overall system still functions and the affected pod eventually recovers in case of pod failure (for FEs, proxies,, targets)

Replace the -ttl 72000 spire setting with renewals

Replace the -ttl 72000 spire setting with renewals to prevent timeouts (after 72000 seconds)

Add FrontEnd/BIRD support (BGP)

Add an LB integrated FrontEnd that includes bird with BGP support

Add build parameter for OS base image in operator

Add build parameter for OS base image in operator

Adding support for multiple trenches

Adding support for multiple trenches/vLANs

New API

Here is the new api specifications

• Configuration Manager
• Target Registry
• Ambassador

Verify fragmentation handling in Meridio

nfqlb is integrated into Meridio but end-to-end verification has not yet done.
The scope of this ticket is to verify fragmentation handling in Meridio
To do once FE alpha is finalized.

Deployment FW documentation (alpha level)

Basic documentation for the deployment FW
Limited alpha scope, basic info needed for use and deployment

Traffic load-balancing issue on a target removal

What happened:

On a target removal (scale-in or disconnection of a target from a trench), the traffic is not received by all remaining targets. For instance, if there are 4 targets connected to a trench and receiving the traffic correctly, if one is removed, the traffic might reach only 2 targets instead of 3.

This bug is happening rarely and randomly.

This could also happen when the e2e tests are running which makes some test cases failing.

STEP: Disconnecting the target from a trench
• Failure [10.618 seconds]
Target
/Meridio/test/e2e/target_test.go:30
  With one trench (trench-a) deployed in namespace red containing 2 VIP addresses (20.0.0.1:5000, [2000::1]:5000) and 4 target pods running ctraffic
  /Meridio/test/e2e/target_test.go:52
    should be possible for a target to disconnect from a trench and connect to it again [It]
    /Meridio/test/e2e/target_test.go:71

    Expected
        <int>: 2
    to equal
        <int>: 3

    /Meridio/test/e2e/target_test.go:78

What you expected to happen:

The traffic should reach and be load-balancer among all remaining targets.

How to reproduce it (as minimally and precisely as possible):

Deploy Meridio with 2 trenches, and 4 targets in each trench.

To reproduce it, we can or run the e2e tests:

make e2e

Or disconnect a target manually:
On a target:

./target-client disconnect -ns load-balancer -t trench-a

On the traffic generator host:

ctraffic -address 20.0.0.1:5000 -nconn 400 -rate 100 -monitor -stats all > v4traffic.json
ctraffic -analyze hosts -stat_file v4traffic.json

Environment:

• Kind (1 Master and 2 Workers)
• Kubernetes 1.20.1
• NSM 1.0
• Meridio - e570a21

Switch spire refreshs to use the NSM api

Add version handling for the NSP API

Add version handling for the NSP API

Generic NSM 1.0 install script for testing

Generic NSM 1.0 install script for testing

Meridio operator helm chart

Add a helm chart for the meridio operator

nTargets to connect to the proxy on the same node

In order to minimize the number of hops needed for connections the nTargets should connect to a proxy hosted on the same node as itself.

Egress traffic failover.

If an FE detects that the link towards the external gateway is broken: BGP down, BFD down or the interface is down (operational or administratively) Meridio should stop using this link for outgoing traffic. The traffic should be diverted to other still active links.
In the case where the LB/FE is combined in one pod (composite setup) the LB might also need to be withdrawn from the egress traffic path.

Minimize the privileges needed for running Meridio

In order to minimize the requirements for running Meridio and strengthen security, evaluate and minimize the need for elevated privileges in Meridio

Handling of hashing collisions

Mitigate or manage target identifier hashing collisions

Unix socket for Meridio ambassador pod communication

Convert the meridio ambassador to use unix sockets for communication

Frontend documentation (alpha level)

Basic documentation for the FrontEnd
Limited alpha scope, basic info needed for use and deployment

Deployment FW Implementation, phase 3

One CR per trench, tested with 3 trenches in the system.

All mandatory trench specific settings handled
by the operator. Such as VIPs, gateway, FW addresses, vlan-id, vlan interface

Operator support for VIP ranges

Add support for setting VIP ranges with the operator

Verify ping responses for VIP addresses

The functionality to ping meridio VIPs should already be in place in the latest LBs.

Verify cluster external pinging of the VIPs

Add liveness/readiness probes

Add liveness/readiness probes for Meridio images

Target integration API

Add an API based on the ambassador pattern to facilitate integration between targets and Meridio

Meridio Operator: add ASN and hold-time in Gateway CRD

ASN and hold-time should be runtime configurable
This needs an update in gateway CRD and controller

Deployment FW Implementation, phase 5

Upgrade support

Meridio traffic streams support

Operator to take repo setting from yaml files

Operator to take repo info from the external yaml files instead of the go code

Add option to replace base image in meridio-operator dockerfile

Redundancy handling for the ipam/nsp pods

Investigate and add high availability for the ipam/nsp pods

Functionality to specify on what nodes to run FE/LBs

Add functionality in order to set on what worker nodes FEs will run (since not all nodes might have external interfaces available)

Note:
Likely not critical for the PRA as cloud based nodes generally are identical / can all provide external access

Operator support for traffic streams

Add configuration support for traffic streams to the operator

Add more e2e tests to meridio operator

Add FrontEnd support for static + BFD

Add/verify support for static routing + BFD in the Meridio FE

Deployment FW Implementation, phase 1

"Plumbing" config FW Implementation in Meridio

Deployment function based on Operator/CRDs

Phase 1:
Basic deployment with the "old"/pre-existing helmcharts, but now deployed through the operator

Meridio traffic flows support

Add dualstack support

Add dualstack support for the Meridio elements

Add FrontEnd support for BGP + BFD

Add/verify support for BGP + BFD in the Meridio FE

Full NSM mesh between FEs and LBs

A full NSM connection mesh is needed between the elements in the FE function block and all elements in all LB function block(s).
Remember that a trench can include several conduits, where each conduit can/will include a LB function block. But for the first implementation it will be enough to consider one conduit as the support for several conduits are to come at a later stage.

The full mesh solution should be generic supporting:

1. One FE being collocated with one LB in the same pod, where the FE must prioritize the forwarding of traffic to this local LB.
   Notice that although the FE and LB is collocated some traffic still may need to be forwarded to another LB. For example
   fragmentation handling will need all fragments from the same IP-message to be handled by the same LB instance although
   arriving though different FEs.
2. The FE and LB being separated in two pods.

Change operator from cluster-scoped to namespace-scoped

Operator should watch for one namespace. This makes it possible to deploy different versions of the operator in one cluster

Test the multi-ip / dualstack functionality recently provided by NSM

The NSM community recently provided new functionality to provide support for multiple IPs on the interfaces (both IPv4 and IPv6). Verify the functionality and evaluate if it provides what we need for dualstack support.

Runtime configuration framework

Adding the framework for runtime configuration

nfqlb 1.0 code review

nfqlb 1.0 code review

Target identifier collision between streams

Currently it is possible 2 targets open a stream (the streams have to be different for this to happen) with the same identifier which will create a conflict on the LB side since the IP rules/routes will collide.

3 solutions are possible:

1. Have common identifier range with a unique identifier for each target

• The number of identifier in use in equal to the number of target in the trench
• Stream and target needs to be detached in the LB. The conduit configures the target rules/routes.

2. Have common identifier range with a unique identifier for each stream opened

• The number of identifier in use in equal to the number of stream opened in the trench

3. Separated identifier ranges

• Each stream should have a max size + offset
• Needs extra parameters in the configmap and some work on the Operator to find available offset

related issue: #32

FE integration with the Meridio operator

Alpha scope: Basic/fundamental config settings

Deployment FW Implementation, phase 2

"Plumbing" config FW Implementation in Meridio

Deployment function based on Operator/CRDs

Phase 2:
Set container specific options with the Operator/CRDs

Scale-in/out of LBs

Add support for scaling in/out the LBs and verify traffic

Deployment FW Implementation, phase 4

Split config into multiple CRDs per trench

7 crds per trench, remaining ones conduit, stream, flow