kinvolk/egress-filtering-benchmark

Egress Filtering Benchmark

This repository contains a set of tools to measure the egress filtering performance using BPF, iptables, ipsets and calico.

How to Use

1. Setup two computers to run the test. You need to have Docker, iptables and ipset installed and you should be able to connect to those computers with SSH without requiring a password.

2. Download the latest Lokomotive release

   Unpack and move to a desired locatin:

   tar xvf lokoctl_$VERSION_linux_amd64.tar.gz
   mv lokoctl_$VERSION_linux_amd64/lokoctl ~/.local/bin/lokoctl

3. Create a Kubernetes cluster using Lokomotive with at least one worker node.

   A minimal working configuration that can be deployed on Packet (acquired by Equinix Metal)

   Update the variables in lokocfg.vars and execute:

   git clone [email protected]:kinvolk/egress-filtering-benchmark.git
   cd lokomotive
   lokoctl cluster apply

   Set location of kubeconfig using the environment variable KUBECONFIG:

   cd egress-filtering-benchmark/lokomotive
   export KUBECONFIG=$PWD/assets/cluster-assets/auth/kubeconfig
   

   Label the worker node as follows:

   kubectl label node calico-benchmark-pool-1-worker-0 nodetype=worker-benchmark
   

4. Configure the parameters of the test in the parameters.py file.

5. Install the required libraries in the client to run the Python script

pip install -r requirements.txt

6. Execute the tests:

$ cd benchmark
$ make
$ python run_tests.py --mode udp --username USERNAME --client CLIENTADDR --server SERVERADDR

This will create some csv files with the information about the test. You can plot them by your self or follow the next step.

7. Plot the data by running

$ python plot_data.py

This will create some svg files with the graphs.

Credits

The BPF filter is inspired by the tc-bpf man page and the Cilium documentation.

egress-filtering-benchmark's Issues

Feature request: Measure latency with netperf TCP_RR instead of ping.

Measuring the latency with ping might be less reliable than netperf and it produces different results.
Also, applications more commonly use TCP and not ICMP packets.

We can use the netperf option TCP_RR (RR=request-response).

The following article says "Generally, we recommend using netperf over ping for latency tests":

https://cloud.google.com/blog/products/networking/using-netperf-and-ping-to-measure-network-latency

Broken link in the article (Performance Benchmark Analysis of Egress Filtering on Linux)

Description

Thanks for the article (Performance Benchmark Analysis of Egress Filtering on Linux), it's a very interesting read!

The own link in the following sentence points to https://github.com/kinvolk/egress-filtering-benchmark/blob/master/pkg/filters/bpf/datapath/bpf.c which returns 404.

Given that we want to keep the tests as simple as possible we decided to implement our own filtering program in eBPF that will be attached to the traffic control layer with a clsact qdisc.

I assume the correct one is: https://github.com/kinvolk/egress-filtering-benchmark/blob/master/pkg/filters/tc-bpf/datapath/bpf.c

Install Cilium on Lokomotive using kubeproxy free mode.

In the benchmark , it was noticed that Cilum has a higher CPU usage compared to other filters. Upon inspection with the profile tool, we noticed a fairly good amount of time in iptables processing as evident by the presence of ipt_do_table and nf_hook_slow function calls.

Profile trace output
https://gist.github.com/ipochi/ae0fad1e8af8be7667f37645b7b0a758

Installing with kube-proxy free mode would probably help improve CPU usage

Use Packet per Second as a metric

Current situation

Currently one of the metric for the Performance benchmark is Throughput in Gbps for TCP/UDP.

Impact
Measuring the throughput with a fixed packet size doesn't give the direct insights compared to measuring packets per second.

Ideal future situation
UDP test doesn't tell the size of the packets and measures only the throughput and it's easy to get a high throughput by using large packets.

The challenge is to maintain line rate throughput with the smallest packets because filtering is a per-packet action. So instead of comparing the throughput with ~64 bytes packets it is an established practice to directly measure the Mpps (million packets per second) and see if the maximum of ~14 Mpps for a 10 G line rate is reached.

Additional information

https://discuss.aerospike.com/t/benchmarking-throughput-and-packet-count-with-iperf3/2791