ALB on BSD Systems

[fititnt]

ALB on BSD Systems

[fititnt]

v0.8.4-alpha, I'm having errors similar to what had before with ansible facts related for permissions of directories. This one is very specific to FreeBSD

TASK [Gathering Facts] ****************************************************************************************************************************************************************************************************************
ok: [ap_foxtrot_centos8]
ok: [ap_delta_ubuntu18]
ok: [ap_echo_debian10]
ok: [ap_golf_archilinux]
fatal: [rocha_anortosito_freebsd12]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"setup": {"ansible_facts": {"discovered_interpreter_python": "/usr/local/bin/python3.6"}, "cmd": "/etc/ansible/facts.d/alb_openresty.fact", "failed": true, "invocation": {"module_args": {"fact_path": "/etc/ansible/facts.d", "filter": "*", "gather_subset": ["all"], "gather_timeout": 10}}, "msg": "[Errno 2] No such file or directory: b'/etc/ansible/facts.d/alb_openresty.fact': b'/etc/ansible/facts.d/alb_openresty.fact'", "rc": 2, "warnings": ["Platform freebsd on host rocha_anortosito_freebsd12 is using the discovered Python interpreter at /usr/local/bin/python3.6, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information."]}}, "msg": "The following modules failed to execute: setup\n"}

dirty fix is deleting the file (like ansible rocha_anortosito_freebsd12 -i hosts.yml -m shell -u root -a "rm -f /etc/ansible/facts.d/alb_openresty.fact") but this must be solved later

[fititnt]

This

-#!/bin/bash
+#!/bin/sh

solves #37 (comment).

[fititnt]

The HAProxy path is also different from the other *nix.

(...)
TASK [ap-application-load-balancer : reconfigure.yml] 
included: /alligo/code/fititnt/ap-application-load-balancer/tasks/haproxy/reconfigure.yml for ap_delta_ubuntu18, ap_echo_debian10, ap_foxtrot_centos8, ap_golf_archilinux, rocha_anortosito_freebsd12, rocha_basalto_opensuse15

TASK [ap-application-load-balancer : haproxy | reconfigure: /etc/haproxy/haproxy.cfg]
changed: [ap_echo_debian10]
changed: [ap_delta_ubuntu18]
fatal: [rocha_anortosito_freebsd12]: FAILED! => {"changed": false, "checksum": "af6b6093e5663f933e09e50c4ce1ae273316ab43", "msg": "Destination directory /etc/haproxy does not exist"}
changed: [ap_golf_archilinux]
[rocha_anortosito_freebsd12] TASK: ap-application-load-balancer : haproxy | reconfigure: /etc/haproxy/haproxy.cfg (debug)> 

On the node

[root@rocha-anortosito-freebsd12 /]# find / -iname haproxy
/usr/ports/net/haproxy
/usr/local/sbin/haproxy
/usr/local/etc/rc.d/haproxy
/usr/local/share/examples/haproxy
/usr/local/share/doc/haproxy
/opt/alb/haproxy

This is our actual yaml task

- name: "haproxy | reconfigure: /etc/haproxy/haproxy.cfg"
  template:
    # src: "{{ role_path }}/templates/openresty/nginx/conf/nginx.conf.j2"
    src: "{{ alb_haproxy_haproxy_template }}"
    dest: /etc/haproxy/haproxy.cfg
    owner: "{{ alb_internal_root_user }}"
    group: "{{ alb_internal_root_group }}"
    mode: '0644'
    backup: yes
  notify:
    - reload haproxy

[fititnt]

HUMMM... FreeBSD does not use systemd. It's the old school service. Not that this is bad, just different. Will need more specific strategy. Also not so easy to find tutorials on internet, but maybe will not take that long.

Also, something I really, really liked on all other OSs tested that are not Centos or Ubuntu/Debian: all they already ship with HAProxy 2.0 or 2.1. So actually they do not need a repo for what we need.

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/etc/rc.d/haproxy

Here I' trying to discover where to make things work and where to put everything.

 
#!/bin/sh
#
# $FreeBSD: branches/2019Q4/net/haproxy/files/haproxy.in 477980 2018-08-24 19:19:26Z demon $
#

# PROVIDE: haproxy
# REQUIRE: DAEMON LOGIN
# KEYWORD: shutdown

#
# Add the following lines to /etc/rc.conf to enable haproxy:
#
# haproxy_enable (bool):	default: "NO"
#				Set to "YES" to enable haproxy
# haproxy_pidfile (str):	default: /var/run/haproxy.pid
#				Set to the full path of the pid file
# haproxy_config (str):		default: /usr/local/etc/haproxy.conf
#				Set to the full path of the config file
# haproxy_flags (str):		default: Autogenerated using pidfile and config options
#				Set to override with your own options
# haproxy_profiles (str):	default: empty
# Set to space-separated list of profiles: for each profile separate haproxy
# process will be spawned, with haproxy-${profile}.conf config file.
# You can override default pidfile and config file for each profile with
# haproxy_${profile}_config and haproxy_${profile}_pidfile.

. /etc/rc.subr

name="haproxy"
rcvar=haproxy_enable
command="/usr/local/sbin/haproxy"
extra_commands="reload configtest hardstop hardreload softreload"
reload_cmd="haproxy_reload"
hardreload_cmd="haproxy_reload"
hardreload_precmd="def_hardreload_option"
softreload_cmd="haproxy_reload"
softreload_precmd="def_softreload_option"
stop_cmd="haproxy_stop"
hardstop_cmd="haproxy_stop"
hardstop_precmd="def_hardstop_signal"

: ${haproxy_enable:="NO"}
: ${haproxy_config:="/usr/local/etc/${name}.conf"}
: ${haproxy_socket:="/var/run/${name}/socket"}
pidfile=${haproxy_pidfile:-"/var/run/haproxy.pid"}

def_hardreload_option()
{
    reload_opt="-st"
}

def_softreload_option()
{
    reload_opt="-x ${haproxy_socket} -sf"
}

def_hardstop_signal()
{
    sig_stop="TERM"
}

load_rc_config $name

is_valid_profile() {
    local profile
    for profile in $haproxy_profiles; do
        if [ "$profile" = "$1" ]; then
            return 0
        fi
    done
    return 1
}

if [ -n "$2" ]; then
    profile=$2
    if ! is_valid_profile $profile; then
        echo "$0: no such profile ($profile) defined in ${name}_profiles."
        exit 1
    fi
    eval haproxy_config="\${haproxy_${profile}_config:-/usr/local/etc/haproxy-${profile}.conf}"
    eval pidfile="\${haproxy_${profile}_pidfile:-/var/run/haproxy-${profile}.pid}"
else
    if [ "x${haproxy_profiles}" != "x" -a "x$1" != "x" ]; then
        for profile in ${haproxy_profiles}; do
            echo "===> ${name} profile: ${profile}"
            /usr/local/etc/rc.d/haproxy $1 ${profile}
            retcode="$?"
            if [ ${retcode} -ne 0 ]; then
                failed="${profile} (${retcode}) ${failed:-}"
            else
                success="${profile} ${success:-}"
            fi
        done
        exit 0
    fi
fi

: ${haproxy_flags:="-q -f ${haproxy_config} -p ${pidfile}"}
configtest_cmd="$command -c -f $haproxy_config"
start_precmd="$command -q -c -f $haproxy_config"
required_files=$haproxy_config
sig_stop=SIGUSR1
reload_opt="-sf"

haproxy_reload()
{
	${command} -q -c -f ${haproxy_config}
	if [ $? -ne 0 ]; then
	    err 1 "Error found in ${haproxy_config} - not reloading current process!"
	fi
	rc_pid=$(check_pidfile ${pidfile} ${command})
	if [ $rc_pid ]; then
		${command} ${haproxy_flags} $reload_opt $(cat ${pidfile})
	else
		_run_rc_notrunning
		return 1
	fi
}

haproxy_stop()
{
	rc_pid=$(check_pidfile ${pidfile} ${command})
	if [ $rc_pid ]; then
		rc_pid=$(cat ${pidfile})
		kill -$sig_stop $rc_pid
		wait_for_pids $rc_pid
	else
		_run_rc_notrunning
		return 1
	fi
}

run_rc_command "$1"

Edit: added another file

[root@rocha-anortosito-freebsd12 /]# cat /etc/rc.conf

zfs_enable="YES"
hostname="vmi321960.contaboserver.net"
ifconfig_vtnet0="inet 144.91.107.138 netmask 255.255.192.0"
gateway_if="vtnet0"
gateway_ip="144.91.64.1"
static_routes="gateway default"
route_gateway="-host $gateway_ip -interface $gateway_if"
route_default="default $gateway_ip"
#ipv6_default_interface="vtnet0"
#ifconfig_vtnet0_ipv6="2a02:c207:2032:1960:0000:0000:0000:0001/64"
#ipv6_defaultrouter="fe80::1%vtnet0"
static_ndp_pairs="gw"
static_ndp_gw="fe80::1%vtnet0 28:99:3a:4d:30:af"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"

[fititnt]

Lastest commit we're already have HAProxy installed based on OS Family (the Ansible Generic package fails even for Debian/Ubuntu when I try to enforce one specific version.

Because BSD, this also means that at least the other step, that separate how to reload/restart/start the HAProxy, must be also dependend of the system.

This is our version using systemd (this Ansible module https://docs.ansible.com/ansible/latest/modules/systemd_module.html).

# Enabled/Started must be AFTER moving new /etc/haproxy/haproxy.cfg
# configurations or Ansible will not be able to update one new valid
# configuration if old one was already with error. By moving this step
# after we avoid user being forced to solve manually on the server 
- name: "haproxy | reconfigure: sudo systemctl enable haproxy.service"
  systemd:
    name: haproxy
    state: started
    enabled: yes

The BSD I guess will use the service, https://docs.ansible.com/ansible/latest/modules/service_module.html. Loggin on rocha_anortosito_freebsd12 I see that service command works like the old school method. But seems that the service alone will not make it aware of some default location of configutation, so have to add rules on /etc/rc.conf file. That's a bit scary if not well done.

[fititnt]

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/share/examples/haproxy/transparent_proxy.cfg

#
# This is an example of how to configure HAProxy to be used as a 'full transparent proxy' for a single backend server.
#
# Note that to actually make this work extra firewall/nat rules are required.
# Also HAProxy needs to be compiled with support for this, in HAProxy1.5-dev19 you can check if this is the case with "haproxy -vv".
#

global
defaults
	timeout client		30s
	timeout server		30s
	timeout connect		30s

frontend MyFrontend
	bind	192.168.1.22:80
	default_backend		TransparentBack_http

backend TransparentBack_http
	mode			http
	source 0.0.0.0 usesrc client
	server			MyWebServer 192.168.0.40:80

#
# To create the the nat rules perform the following:
#
# ### (FreeBSD 8) ###
# --- Step 1 ---
# ipfw is needed to get 'reply traffic' back to the HAProxy process, this can be achieved by configuring a rule like this:
#	fwd localhost tcp from 192.168.0.40 80 to any in recv em0
#
# The following would be even better but this did not seam to work on the pfSense2.1 distribution of FreeBSD 8.3:
#   fwd 127.0.0.1:80 tcp from any 80 to any in recv ${outside_iface} uid ${proxy_uid}
#
# If only 'pf' is currently used some aditional steps are needed to load and configure ipfw:
# You need to configure this to always run on startup:
#
# /sbin/kldload ipfw
# /sbin/sysctl net.inet.ip.pfil.inbound="pf" net.inet6.ip6.pfil.inbound="pf" net.inet.ip.pfil.outbound="pf" net.inet6.ip6.pfil.outbound="pf"
# /sbin/sysctl net.link.ether.ipfw=1
# ipfw add 10 fwd localhost tcp from 192.168.0.40 80 to any in recv em0
#
# the above does the following:
# - load the ipfw kernal module
# - set pf as the outer firewall to keep control of routing packets for example to route them to a non-default gateway
# - enable ipfw
# - set a rule to catches reply traffic on em0 coming from the webserver
#
# --- Step 2 ---
# To also make the client connection transparent its possible to redirect incomming requests to HAProxy with a pf rule:
#   rdr on em1 proto tcp from any to 192.168.0.40 port 80 -> 192.168.1.22
# here em1 is the interface that faces the clients, and traffic that is originally send straight to the webserver is redirected to HAProxy
#
# ### (FreeBSD 9) (OpenBSD 4.4) ###
#   pf supports "divert-reply" which is probably better suited for the job above then ipfw..
#

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/share/examples/haproxy/haproxy.init

#!/bin/sh
#
# chkconfig: - 85 15
# description: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited \
#              for high availability environments.
# processname: haproxy
# config: /etc/haproxy/haproxy.cfg
# pidfile: /var/run/haproxy.pid

# Script Author: Simon Matter <[email protected]>
# Version: 2004060600

# Source function library.
if [ -f /etc/init.d/functions ]; then
  . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
  . /etc/rc.d/init.d/functions
else
  exit 0
fi

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

# This is our service name
BASENAME=`basename $0`
if [ -L $0 ]; then
  BASENAME=`find $0 -name $BASENAME -printf %l`
  BASENAME=`basename $BASENAME`
fi

BIN=/usr/sbin/$BASENAME

CFG=/etc/$BASENAME/$BASENAME.cfg
[ -f $CFG ] || exit 1

PIDFILE=/var/run/$BASENAME.pid
LOCKFILE=/var/lock/subsys/$BASENAME

RETVAL=0

start() {
  quiet_check
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi

  echo -n "Starting $BASENAME: "
  daemon $BIN -D -f $CFG -p $PIDFILE
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch $LOCKFILE
  return $RETVAL
}

stop() {
  echo -n "Shutting down $BASENAME: "
  killproc $BASENAME -USR1
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f $LOCKFILE
  [ $RETVAL -eq 0 ] && rm -f $PIDFILE
  return $RETVAL
}

restart() {
  quiet_check
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi
  stop
  start
}

reload() {
  if ! [ -s $PIDFILE ]; then
    return 0
  fi

  quiet_check
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with '$BASENAME check'."
    return 1
  fi
  $BIN -D -f $CFG -p $PIDFILE -sf $(cat $PIDFILE)
}

check() {
  $BIN -c -q -V -f $CFG
}

quiet_check() {
  $BIN -c -q -f $CFG
}

rhstatus() {
  status $BASENAME
}

condrestart() {
  [ -e $LOCKFILE ] && restart || :
}

# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    restart
    ;;
  reload)
    reload
    ;;
  condrestart)
    condrestart
    ;;
  status)
    rhstatus
    ;;
  check)
    check
    ;;
  *)
    echo $"Usage: $BASENAME {start|stop|restart|reload|condrestart|status|check}"
    exit 1
esac
 
exit $?

---

Edit: and more this one as reference

[root@rocha-anortosito-freebsd12 /]# cat /usr/local/share/examples/haproxy/content-sw-sample.cfg

#
# This is a sample configuration. It illustrates how to separate static objects
# traffic from dynamic traffic, and how to dynamically regulate the server load.
#
# It listens on 192.168.1.10:80, and directs all requests for Host 'img' or
# URIs starting with /img or /css to a dedicated group of servers. URIs
# starting with /admin/stats deliver the stats page.
#

global
        maxconn         10000
        stats socket    /var/run/haproxy.stat mode 600 level admin
        log             127.0.0.1 local0
        uid             200
        gid             200
        chroot          /var/empty
        daemon

# The public 'www' address in the DMZ
frontend public
        bind            192.168.1.10:80 name clear
        #bind            192.168.1.10:443 ssl crt /etc/haproxy/haproxy.pem
        mode            http
        log             global
        option          httplog
        option          dontlognull
        monitor-uri     /monitoruri
        maxconn         8000
        timeout client  30s

        stats uri       /admin/stats
        use_backend     static if { hdr_beg(host) -i img }
        use_backend     static if { path_beg /img /css   }
        default_backend dynamic

# The static backend backend for 'Host: img', /img and /css.
backend static
        mode            http
        balance         roundrobin
        option prefer-last-server
        retries         2
        option redispatch
        timeout connect 5s
        timeout server  5s
        option httpchk  HEAD /favicon.ico
        server          statsrv1 192.168.1.8:80 check inter 1000
        server          statsrv2 192.168.1.9:80 check inter 1000

# the application servers go here
backend dynamic
        mode            http
        balance         roundrobin
        retries         2
        option redispatch
        timeout connect 5s
        timeout server  30s
        timeout queue   30s
        option httpchk  HEAD /login.php
        cookie          DYNSRV insert indirect nocache
        fullconn        4000 # the servers will be used at full load above this number of connections
        server          dynsrv1 192.168.1.1:80 minconn 50 maxconn 500 cookie s1 check inter 1000
        server          dynsrv2 192.168.1.2:80 minconn 50 maxconn 500 cookie s2 check inter 1000
        server          dynsrv3 192.168.1.3:80 minconn 50 maxconn 500 cookie s3 check inter 1000
        server          dynsrv4 192.168.1.4:80 minconn 50 maxconn 500 cookie s4 check inter 1000

[fititnt]

Ok, FreeBSD does not come with an haproxy user/group at all. Some discussion here https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199314 and proposed patch here https://bugs.freebsd.org/bugzilla/attachment.cgi?id=155368&action=diff.

[fititnt]

See Implementation of Ansible Molecule on AP-ALB & travis-ci integration #2

---

Reliable CI testing (like with travis-CI) may be more complicated than other linuxes. Here have some discussion on travis-ci about not supporting travis-ci/travis-ci#1818.

In short, while is not as complicated than running a Windows conteiner inside linux host, run a BSD container requires a BSD host. BSDs are not a major concern on Docker to give full support as a host.

I found some places like these ones

• https://github.com/ustclug/ustcmirror-images/tree/master/freebsd-ports
• https://github.com/weitjong/dockerized/blob/master/urho3d/freebsd/Dockerfile

that seems to use some linux OS (in the case, Alpine and Ubuntu) to then download a BSD image and somewhat "work" as if was BSD.

Not sure if this would be able to emulate a full BSD (at at least the bare minimum that could already be sufficient for our tests). But in this case, I guess that means that

• AP-ALB should have (if already not possible just with playbook vars for custom OSs) to support "act as one OS at install, but as another when running"
• AP-ALB, idealy, but not strong requeriment, be able to run on Alpine Linux.
  • One image that seems to already ship with near everyting that we need is openresty/openresty:alpine-fat. It even comes with luarocks and opm, so maybe its less complicated (or even would run faster than other tests) to implement