nordix/kahttp

Keep-alive http test program

kahttp - Keep-alive http test program

This test program is intended for testing many simultaneous (keep-alive) http connections. It is not intended for throughput or latency measurements but is designed to test the limit for how many simultaneous client connectons a http server or proxy can handle.

Build

./build.sh image
ls ./image/kahttp

Usage

Local usage;

# Start server;
kahttp -server -address [::]:5080
# In another shell;
wget -q -O - http://127.0.0.1:5080/
kahttp -address http://127.0.0.1:5080/ -monitor -rate 400 -nclients 40 -timeout 10s | jq .
# In yet another shell;
tcpdump -ni lo 'tcp[tcpflags] == tcp-syn'
# Or for ipv6;
tcpdump -ni lo tcp and 'ip6[13+40]&0x2!=0'

One way to verify that keep-alive is used done is to trace the SYN packets with tcpdump. If there are an initial burst of and then silence keep-alive is used. A steady stream os SYNs indicates that keep-alive is not used.

Deploy a kahttp server on Kubernetes;

kubectl apply -f https://raw.githubusercontent.com/Nordix/kahttp/master/kahttp.yaml

NOTE: The server is very primitive and is intended for simple tests only.

The server will use an internal self-signed certificate for https by default. See below for an instruction howto use a k8s tls secret.

Measure keep-alive connections

Kahttp starts a number of http clients according to the -nclients parameter. Each of these clients will make periodic http GET requests towards the URL specified by the -address parameter. The -rate parameter specifies the total number of http requests per second. This is divided among the clients so a rate of 400 and 40 clients will result in 10 requests per second per client. The keep-alive time is limited (but different) on servers so to make sure connections are kept a certain rate per client must be maintained.

The key metric for measuring keep-alive connections is Dials. This is the total number of actual TCP connects that the clients do. Below is an example where all connections are kept;

> kahttp -address http://127.0.0.1:5080/ -monitor -rate 400 -nclients 40 -timeout 10s | jq .
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 399/399/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 799/799/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 1199/1199/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 1600/1600/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 2000/2000/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 2400/2400/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 2800/2800/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 3200/3200/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 3600/3600/0
{
  "Started": "2019-04-02T13:02:06.815903366+02:00",
  "Duration": 10000112870,
  "Rate": 400,
  "Clients": 40,
  "Dials": 40,
  "FailedConnections": 0,
  "Sent": 3999,
  "Received": 3999,
  "Dropped": 0,
  "FailedConnects": 0
}

In this case we have 40 clients making 3999 requests in total but the number of Dials is 40, so each client make only one connection which is kept alive for the entire run.

This shows the case where keep-alive is disabled using the -disable_ka flag;

# kahttp -nclients 10 -monitor -address https://10.0.0.2/ -host_stats -disable_ka | jq .
Clients act/fail/Dials: 10/0/10, Packets send/rec/dropped: 10/10/0
Clients act/fail/Dials: 10/0/20, Packets send/rec/dropped: 20/20/0
Clients act/fail/Dials: 10/0/30, Packets send/rec/dropped: 30/30/0
Clients act/fail/Dials: 10/0/40, Packets send/rec/dropped: 40/40/0
Clients act/fail/Dials: 10/0/50, Packets send/rec/dropped: 50/50/0
Clients act/fail/Dials: 10/0/60, Packets send/rec/dropped: 60/60/0
Clients act/fail/Dials: 10/0/70, Packets send/rec/dropped: 70/70/0
Clients act/fail/Dials: 10/0/80, Packets send/rec/dropped: 80/80/0
Clients act/fail/Dials: 10/0/90, Packets send/rec/dropped: 90/90/0
{
  "Started": "2019-04-06T08:18:09.44414217Z",
  "Duration": 9897737610,
  "Rate": 10,
  "Clients": 10,
  "Dials": 100,
  "FailedConnections": 0,
  "Sent": 100,
  "Received": 100,
  "Dropped": 0,
  "FailedConnects": 0,
  "Hosts": {
    "kahttp-deployment-ff8b6966-2cjw6": 26,
    "kahttp-deployment-ff8b6966-j45fw": 24,
    "kahttp-deployment-ff8b6966-plvqn": 26,
    "kahttp-deployment-ff8b6966-t867d": 24
  }
}

The number of Dials is equal to the number of Sent requests. This means that no connection is kept alive.

Server host statistics

The example above uses the -host_stats to get server host statistics. Server host statistics only works if the server is kahttp. The kahttp server returns the hostname of the server in the body and in the X-Kahttp-Server-Host: http header. The kahttp client collect the statistics and presents the number of requests to each host.

This functions is not usable for keep-alive testing (the main purpose of kahttp) but may be used for instance to test a canary setup. When -host_stats is used you should most likely also disable keep-alive with -disable_ka.

Server access logging

Access to the server can be logged (to stdout). This may be useful for debugging and tests of low intensity traffic such as health probing.

NOTE: Extensive logging will affect performance.

Server access logging is initiated with the -log-access parameter or the $LOG_ACCESS environment variable. The parameter specifies a sub-path to be logged or "/" to log everything.

$ ./image/kahttp -server -address :7000 -log-access /metrics
2020-07-11 13:02:39; localhost:7000/metrics from 127.0.0.1:43660
2020-07-11 13:02:39; localhost:7000/metrics from 127.0.0.1:43662
2020-07-11 13:02:39; localhost:7000/metrics from 127.0.0.1:43664
...

The log format is "date; <host/path> from <remote_addr>".

Multiple source addresses

For testing of very many connections it is essential that the connections originates from many sources. If just one (prominent) source is used there will likely be a shortage of connection resources since the src/dest addresses are always the same, only the ports can be varied. The resource problem does not have to occur on the endpoints but can be in network elements (NE) in between such as load-balancers or NAT boxes.

To use many source addresses you must first assign a CIDR to the loopback interface as described for mconnect. Then use;

kahttp -address http://127.0.0.1:5080/ -monitor -rate 400 -nclients 40 \
  -timeout 10s -scrcidr 222.222.222.0/24

Set source address

Some times the source address must be set for instance to use a specific ip-family as described in #6.

The -scrcidr can be used for this purpose but with masks /32 and /128;

$ kahttp kahttp -address http://vm-001/cgi-bin/info -monitor -timeout 5s -srccidr 192.168.1.3/32
... (ipv4 used)
$ kahttp kahttp -address http://vm-001/cgi-bin/info -monitor -timeout 5s -srccidr 1000::1:c0a8:103/128
... (ipv6 used)

Https

Kahttp disables certificate verification because the server is most likely using a self-signed certificate. There is currently no way of enabling certificate verification in the kahttp client.

The kahttp server will expose a https server if both -https_key and -https_cert are specified (or the corresponding environment variables $KAHTTP_KEY and $KAHTTP_CERT are set). The address (including the port) can be specified with the -https_addr option (default ":5443").

Implementation is based on this description.

Example;

export KAHTTP_KEY=/tmp/server.key
export KAHTTP_CERT=/tmp/server.crt
openssl genrsa -out $KAHTTP_KEY 2048
openssl req -new -x509 -sha256 -key $KAHTTP_KEY -out $KAHTTP_CERT -days 3650
kahttp -server -address [::]:5080
# In another shell;
wget --no-check-certificate -qO- https://[::1]:5443/
# Or;
kahttp -address https://[::1]:5443/ -monitor -rate 400 -nclients 40 -timeout 10s | jq .

If you use another client, e.g. curl you can verify the self-signed certificate. First the "Common name" used must match the request url. Use the --resolv option, for example;

curl -v --cacert /tmp/server.crt --resolv kahttp.localdomain:5443:[::1] \
  https://kahttp.localdomain:5443/

Client certificate, mTLS

Kahttp accepts and verifies client certificates if given (tls.VerifyClientCertIfGiven). Http requests without client certificate or with a correct certificate are accepted but requests with an invalid certificate are rejected.

curl -v --cacert /tmp/server.crt --resolv kahttp.localdomain:5443:[::1] \
  --cert /tmp/server.crt --key /tmp/server.key \
  https://kahttp.localdomain:5443/
...
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):

The same certificate as for the server is presented as a client certificate which is valid and the (faked) header CERT verify indicates that a client certificate is used.

If an invalid client certificate is used the conneciton fails;

curl --cacert /tmp/server.crt --resolv kahttp.localdomain:5443:[::1] \
  --cert /tmp/kahttp.crt --key /tmp/kahttp.key \
  https://kahttp.localdomain:5443/
...
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Http2

Curl uses http/2 by default, use the --http1.1 option to enforce http1. Use the -http2 flag to make the kahttp client to use http2. Http2 used on a clear-text connection (http) is converted to h2c.

curl --insecure --http1.1 --resolv kahttp.localdomain:5443:[::1] https://[::1]:5443/

kahttp -address https://[::1]:5443/ -monitor -http2 -rate 400 -nclients 40 -timeout 10s | jq .

# This will use h2c;
curl -D - --http2 http://[::1]:5443/

# This will not work;
kahttp -address http://[::1]:5443/ -http2

Use a k8s tls secret

First create a k8s tls secret from the crt and key files;

kubectl create secret tls kahttp-secret --key /cert/kahttp.key --cert /cert/kahttp.crt

In the kahttp manifest mount the secret and set the KAHTTP_KEY and KAHTTP_CERT variables appropriately (see kahttp-secret.yaml);

...
        env:
          - name: KAHTTP_KEY
            value: "/cert/tls.key"
          - name: KAHTTP_CERT
            value: "/cert/tls.crt"
        volumeMounts:
          - name: cert
            mountPath: "/cert"
            readOnly: true
      volumes:
        - name: cert
          secret:
            secretName: kahttp-secret

kahttp's Issues

Garbage in the X-Kahttp-Server-Host header

X-Kahttp-Server-Host: {Kahttp/latest@vm-003  {%!s(int32=0) %!s(uint32=0)}}

The "int32" things should not be there.

The host_stats feature does not work with a proxy

If a proxy, e.g. an "ingress controller" i k8s, is used the "Server:" header is over-written by the proxy.

Feature Request: --resolve as parameter

Hi

It would be great to have a feature in kahttp similar to Curl's --resolve. This could eliminate the use of external name resolution.

Thank you in advance.

Feature Request: --debug parameter

It would be great to have some kind of parameter which could switch on access logging for kahttp.
At a first glance it could log:
time, source address, destination port, path (e.g / or /health)

Thank you in advance

Support for H2C

It would be great if kahttp server could support traffic over http2 without TLS.
This could be governed by the client: If the client initiates http2 connection to the cleartext port the traffic could be over http2.

It would also be great if this could be easily looked up from the response the server gives:
The proto field in the response could also contain if tls is used or not:
Proto: HTTP/1.1 would mean for me: regular http1.1
Proto: HTTP/1.1 + TLS -> tls
Proto: HTTP/2 -> h2c
Proto: HTTP/2 + TLS -> h2
Of course TLS can be a new field also, that could also be easily follow-able for me.

Thank you in advance.

Support for client certificate

This is a feature request:

It would be great if an addition to TLS if a CA is present (in the form of a base64 coded certificate) the kahttp server could check the validity of the Client certificate against that CA. And only send traffic if the client cert is validated.

In Parallel the kahttp client could send client certificate for TLS communication.

Thank you in advance.

http-v1 requests with fault codes counts as succesful

Hi
Could you make it possible to see the distribution between the pods when I am using an IP address instead of the hostname?

kahttp -address http://10.0.0.0 -monitor -host_stats | jq .
{
"Started": "2019-05-16T11:47:51.002196261+02:00",
"Duration": 9927841331,
"Rate": 10,
"Clients": 1,
"Dials": 1,
"FailedConnections": 0,
"Sent": 100,
"Received": 100,
"Dropped": 0,
"FailedConnects": 0
}

kahttp -address http://hostname -monitor -host_stats | jq .
{
"Started": "2019-05-16T11:48:18.434265597+02:00",
"Duration": 9937776212,
"Rate": 10,
"Clients": 1,
"Dials": 1,
"FailedConnections": 0,
"Sent": 100,
"Received": 100,
"Dropped": 0,
"FailedConnects": 0,
"Hosts": {
"kahttp-deployment-5c94487945-8gt7n": 25,
"kahttp-deployment-5c94487945-9njn6": 25,
"kahttp-deployment-5c94487945-gh5bn": 25,
"kahttp-deployment-5c94487945-lzk2r": 25
}
}

Thanks in advance.

Feature Request: Force IPv4 or IPv6 flag

What is missing?

For testing purposes when using the kahttp client it would be helpful with a flag that can force the client to connect over specific IPfamily only. This so that a host that should be reachable over both IPfamilies can be tested properly.

Suggestion

Similar flags exist in curl and wget

man wget
[...]
       -4
       --inet4-only
       -6
       --inet6-only
           Force connecting to IPv4 or IPv6 addresses.  With --inet4-only or
           -4, Wget will only connect to IPv4 hosts, ignoring AAAA records in
           DNS, and refusing to connect to IPv6 addresses specified in URLs.
           Conversely, with --inet6-only or -6, Wget will only connect to IPv6
           hosts and ignore A records and IPv4 addresses....

kahttp httpv2.0 support from client side

Hi

Could you please make it possible to force httpv2.0 from the client side?

Thanks in advance

distribution between the backend pods

Hi

Can you please add a function which is similar to the output of the mconnect application?
mconnect -address -nconn 200 -timeout 10s
Failed connects; 0
Failed reads; 0
mconnect-deployment-5d75b7c48c-jvpf4 50
mconnect-deployment-5d75b7c48c-lbrfn 50
mconnect-deployment-5d75b7c48c-rws6d 50
mconnect-deployment-5d75b7c48c-bh5f8 50

Thanks in advance.

Add a gRPC server

For some tests, especially involving a reverse proxy, just HTTP/2 is not sufficient. There should a possibility to use gRPC calls with kahttp.