cirocosta/ingress_ipvs_exporter

Exports Docker's ingress IPVS metrics

ingress_ipvs_exporter 📡

Prometheus exporter for Docker Swarm Ingress IPVS metrics

Overview

ingress_ipvs_exporter is a Prometheus exporter focused on delivering statistics gathered via netlink regarding IPVS services that live inside the docker swarm's ingress network namespace.

Usage: ingress_ipvs_exporter 
	[--listen-address LISTEN-ADDRESS] 
	[--telemetry-path TELEMETRY-PATH] 
	[--namespace-path NAMESPACE-PATH]

Options:
  --listen-address LISTEN-ADDRESS
                         address to set the http server to listen to 
                         [default: :9100]

  --telemetry-path TELEMETRY-PATH
                         endpoint to receive scrape requests from prometheus 
                         [default: /metrics]

  --namespace-path NAMESPACE-PATH
                         absolute path to the network namespace where ipv is configured
                         [default: /var/run/docker/netns/ingress_sbox]

  --help, -h             display this help and exit

It exports the following metrics:

ipvs_bytes_in_total                             The total number of incoming bytes a virtual server
ipvs_bytes_out_total                            The total number of outgoing bytes from a virtual server
ipvs_connections_total                          The total number of connections made to a virtual server
ipvs_destination_active_connections_total       The total number of connections established to a destination server
ipvs_destination_bytes_in_total                 The total number of incoming bytes to a real server
ipvs_destination_bytes_out_total                The total number of outgoing bytes to a real server
ipvs_destination_connections_total              The total number connections ever established to a destination
ipvs_destination_inactive_connections_total     The total number of connections inactive but established to a destination server
ipvs_destination_total                          The total number of real servers that are destinations to the service
ipvs_services_total                             The total number of services registered in ipvs

Example:

# Create three services that publish ports in ingress
for i in $(seq 1 3); do 
	docker service create \
		--no-resolve-image \
		--detach=true \
		--name service_$i \
		--publish 80 \
		nginx:alpine
done

# Check the ports that these service have been bound to
docker service ls \
	--format '{{ json .Ports }}'
"*:30000->80/tcp"
"*:30001->80/tcp"
"*:30002->80/tcp"

# Make 10 requests to the first service
for i in $(seq 1 10); do
        curl \
                --silent \
                localhost:30000 \
                > /dev/null
done

# Start the exporter
sudo ingress_ipvs_exporter

# Check the metrics captured (stripped the other 
# services for better readability).
curl \
	--silent \
	localhost:9100/metrics | \
		ag ipvs

ipvs_bytes_in_total{fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 4510
ipvs_bytes_out_total{fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 11190
ipvs_connections_total{fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 10
ipvs_destination_active_connections_total{address="10.255.0.12",fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 0
ipvs_destination_bytes_in_total{address="10.255.0.12",fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 4510
ipvs_destination_bytes_out_total{address="10.255.0.12",fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 11190
ipvs_destination_connections_total{address="10.255.0.12",fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 10
ipvs_destination_inactive_connections_total{address="10.255.0.12",fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 10
ipvs_destination_total{fwmark="260",namespace="/var/run/docker/netns/ingress_sbox",port="30000"} 1
ipvs_services_total{namespace="/var/run/docker/netns/ingress_sbox"} 3

Developing

Make sure you have the necessary permissions to run modprobe, ip netns and ipvsadm.

Usually, that means that you need to execute make test as a superuser.

Using sudo, make sure that $PATH is properly set - an easy way of doing so is modifying /etc/sudoers and adding the Go paths to the secure path.

ingress_ipvs_exporter's Issues

Retrieve destination ports used in firewall marks

Hey,

Right now there's no way of correlating the FWMarks across multiple machines.

One way of making that better is by either directly using iptables and doing some parsing or, instead, take a look at iptables-save (https://github.com/cernekee/iptables/blob/b90e798596465f0ea2daff75d95f4f978f0a8377/iptables/iptables-save.c) and use libiptc to retrieve that info with cgo.

libiptc can be retrieved as a dependency of iptables-dev.

Replace `libnetwork/ipvs` by `mqliang/libipvs`

Hey,

Currently, there's no simple way of retrieving the destination statistics as well as updating a gauge that'd show us the current number of active connections of each service/destination.

It looks like libipvs has such functionality implemented, which might be very handy for us.

thx!

Make `--namespace-path` optional instead of required