Giter Site home page Giter Site logo

haproxyadmin's Introduction

haproxyadmin

A Python library to manage HAProxy via stats socket.

haproxyadmin is a Python library for interacting with HAProxy load balancer to perform operations such as enabling/disabling servers. It does that by issuing the appropriate commands over the stats socket provided by HAProxy. It also uses that stats socket for retrieving statistics and changing settings.

HAProxy is a multi-process daemon and each process can only be accessed by a distinct stats socket. There isn't any shared memory for all these processes. That means that if a frontend or backend is managed by more than one processes, you have to find which stats socket you need to send the query/command. This makes the life of a sysadmin a bit difficult as he has to keep track of which stats socket to use for a given object(frontend/backend/server).

haproxyadmin resolves this problem by presenting objects as single entities even when they are managed by multiple processes. It also supports aggregation for various statistics provided by HAProxy. For instance, to report the requests processed by a frontend it queries all processes which manage that frontend and return the sum.

The library works with Python 2.7 and Python 3.6, but for development and testing Python 3.6 is used. The Six Python 2 and 3 Compatibility Library is being used to provide the necessary wrapping over the differences between these 2 major versions of Python.

>>> from haproxyadmin import haproxy
>>> hap = haproxy.HAProxy(socket_dir='/run/haproxy')
>>> frontends = hap.frontends()
>>> for frontend in frontends:
...     print(frontend.name, frontend.requests, frontend.process_nb)
...
frontend_proc2 0 [2]
haproxy 0 [4, 3, 2, 1]
frontend_proc1 0 [1]
frontend1_proc34 0 [4, 3]
frontend2_proc34 0 [4, 3]
>>>
>>>
>>> backends = hap.backends()
>>> for backend in backends:
...    print(backend.name, backend.requests, backend.process_nb)
...    servers = backend.servers()
...    for server in servers:
...       print(" ", server.name, server.requests)
...
backend_proc2 100 [2]
  bck_proc2_srv4_proc2 25
  bck_proc2_srv3_proc2 25
  bck_proc2_srv1_proc2 25
  bck_proc2_srv2_proc2 25
haproxy 0 [4, 3, 2, 1]
backend1_proc34 16 [4, 3]
  bck1_proc34_srv1 6
  bck_all_srv1 5
  bck1_proc34_srv2 5
backend_proc1 29 [1]
  member2_proc1 14
  member1_proc1 15
  bck_all_srv1 0
backend2_proc34 100 [4, 3]
  bck2_proc34_srv2 97
  bck2_proc34_srv1 2
  bck_all_srv1 1
>>>

The documentation of the library is available at http://haproxyadmin.readthedocs.org

  • HAProxy in multi-process mode (nbproc >1)
  • UNIX stats socket, no support for querying HTTP statistics page
  • Frontend operations
  • Backend operations
  • Server operations
  • ACL operations
  • MAP operations
  • Aggregation on various statistics
  • Change global options for HAProxy

Use pip:

pip install haproxyadmin

From Source:

sudo python setup.py install

Build (source) RPMs:

python setup.py clean --all; python setup.py bdist_rpm

Build a source archive for manual installation:

python setup.py sdist
  1. Bump versions in docs/source/conf.py and haproxyadmin/__init__.py

  2. Commit above change with:

    git commit -av -m'RELEASE 0.1.3 version'
    
  3. Create a signed tag, pbr will use this for the version number:

    git tag -s 0.1.3 -m 'bump release'
    
  4. Create the source distribution archive (the archive will be placed in the dist directory):

    python setup.py sdist
    
  5. pbr will update ChangeLog file and we want to squeeze them to the previous commit thus we run:

    git commit -av --amend
    
  6. Move current tag to the last commit:

    git tag -fs 0.1.3 -m 'bump release'
    
  7. Push changes:

    git push;git push --tags
    

I would love to hear what other people think about haproxyadmin and provide feedback. Please post your comments, bug reports, wishes on my issues page.

Apache 2.0

This program was originally developed for Booking.com. With approval from Booking.com, the code was generalised and published as Open Source on github, for which the author would like to express his gratitude.

Project website: https://github.com/unixsurfer/haproxyadmin

Author: Pavlos Parissis <[email protected]>

haproxyadmin's People

Contributors

crovner avatar unixsurfer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

haproxyadmin's Issues

'ConnectionRefusedError' is not defined

Throws an exception on Python 2.7.9:

  File "/usr/local/lib/python2.7/dist-packages/haproxyadmin/haproxy.py", line 110, in __init__
    if is_unix_socket(_file) and connected_socket(_file):
  File "/usr/local/lib/python2.7/dist-packages/haproxyadmin/utils.py", line 179, in connected_socket
    except (ConnectionRefusedError, PermissionError, socket.timeout, OSError):
NameError: global name 'ConnectionRefusedError' is not defined

Packages version:

haproxyadmin (0.2.1)
six (1.11.0)

Bug: Setting the IPs of newer HAProxy versions fails with nbproc > 1

I ran into a problem setting the IP of servers, running with nbproc > 1. It seems HAProxy syncs the changes by itself over the processes, so that the outputs of the socket writes differ. I confirmed this with HAProxy v 1.7.11 and 1.8.6 on Centos 7:

Traceback (most recent call last):
  File "namerd-haproxy.py", line 270, in <module>
    main()
  File "namerd-haproxy.py", line 262, in main
    schedule["check"]()
  File "namerd-haproxy.py", line 194, in check_namerd
    haproxy_servers[target_haproxy_backend].setaddress(new_address) #, new_port)
  File "/vagrant/haproxyadmin/server.py", line 146, in setaddress
    return compare_values(values)
  File "/vagrant/haproxyadmin/utils.py", line 271, in compare_values
    raise IncosistentData(values)
haproxyadmin.exceptions.IncosistentData: Received different data per HAProxy process

The 'values' variable in 'utils/compare_values' had the following content:

[(1, "IP changed from '127.0.0.1' to '10.10.26.114' by 'stats socket command'"), (1, "no need to change the addr by 'stats socket command'"), (1, "no need to change the addr by 'stats socket command'"), (1, "no need to change the addr by 'stats socket command'")]

Used HAProxy config:

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    nbproc      4
    stats socket /run/haproxy/1.sock mode 0744 level admin process 1
    stats socket /run/haproxy/2.sock mode 0744 level admin process 2
    stats socket /run/haproxy/3.sock mode 0744 level admin process 3
    stats socket /run/haproxy/4.sock mode 0744 level admin process 4

defaults
    mode http

frontend  main
    bind *:5000
    default_backend             app

backend app
    balance     roundrobin
    server  app1 127.0.0.1:80 check

I tested the following on the command line to see if the IP propagates by itself through all HAProxy processes:

 [root@namerd-haproxy00 vagrant]# echo "show servers state" | socat /run/haproxy/1.sock -
 1
 # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
 7 app 1 app1 10.10.26.114 0 0 1 1 1574 8 2 0 6 0 0 0 - 80
 
 [root@namerd-haproxy00 vagrant]# echo "show servers state" | socat /run/haproxy/2.sock -
 1
 # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
 7 app 1 app1 10.10.26.114 0 0 1 1 1576 8 2 0 6 0 0 0 - 80
 
 [root@namerd-haproxy00 vagrant]# echo "set server app/app1 addr 10.10.10.10 port 3333" | socat /run/haproxy/1.sock -
 IP changed from '10.10.26.114' to '10.10.10.10', port changed from '80' to '3333' by 'stats socket command'
 
 [root@namerd-haproxy00 vagrant]# echo "show servers state" | socat /run/haproxy/2.sock -
 1
 # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
 7 app 1 app1 10.10.10.10 0 0 1 1 1594 7 2 0 6 0 0 0 - 3333
 
 [root@namerd-haproxy00 vagrant]# echo "show servers state" | socat /run/haproxy/3.sock -
 1
 # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port
 7 app 1 app1 10.10.10.10 0 0 1 1 1602 7 0 0 7 0 0 0 - 3333

connected_socket() sometimes times out

Hey Pavlos,

How are you doing? I'm investigating a problem with haproxyadmin, which sometimes fails to send commands through the socket. It turns out to be the function connected_socket() in utils.py, which has a hardcoded timeout of 0.1 seconds. It wouldn't be a problem if the function only opened a connection. But it does more than that; it also sends a show info command and expects a response. Sometimes we don't get a response in time, causing this function to fail.

I'm considering submitting a patch to make this timeout configurable. I would add a parameter to the constructor of the HAProxy class, similar to retry and retry_interval. This param could also be passed along to HAProxyProcess so it can be used when sending other commands (see internal.py file, line 82), for consistency.

Does it make sense?

Support management/connection over TCP

Currently, the connection is established via unix socket files. It would be nice if this library would also support connecting to HAProxy over TCP as well. :)

[EDIT]:
URLs might be a good way on how to define how it connects.

hap = haproxy.HAProxy('unix:///var/run/haproxy.sock')  # via unix socket
hap = haproxy.HAProxy('tcp://127.0.0.1:1937')  # via TCP

New Release

Are the 2 years of changes for this project going to be released on to PyPi. Or this project as is and needs to be cloned from master?

Installation Error

[root@host haproxyadmin]# python2.7 setup.py install
ERROR:root:Error parsing
Traceback (most recent call last):
File "/app/interpreters/python/2.7.11/lib/python2.7/site-packages/pbr/core.py", line 111, in pbr
attrs = util.cfg_to_args(path, dist.script_args)
File "/app/interpreters/python/2.7.11/lib/python2.7/site-packages/pbr/util.py", line 264, in cfg_to_args
wrap_commands(kwargs)
File "/app/interpreters/python/2.7.11/lib/python2.7/site-packages/pbr/util.py", line 566, in wrap_commands
cmdclass = ep.resolve()
File "/app/interpreters/python/2.7.11/lib/python2.7/site-packages/pkg_resources/init.py", line 2297, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
ImportError: No module named scripts.setupcommands
error in setup command: Error parsing /root/haproxyadmin/setup.cfg: ImportError: No module named scripts.setupcommands
[root@cblb-e1a-01 haproxyadmin]# pip2.7 list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
appdynamics (4.2.3.0)
appdynamics-bindeps-linux-x64 (4.2.3.0)
appdynamics-proxysupport-linux-x64 (1.8.0.51)
awscli (1.10.52)
botocore (1.4.42)
colorama (0.3.7)
docutils (0.12)
futures (3.0.5)
jmespath (0.9.0)
meld3 (1.0.2)
pbr (1.10.0)
pip (9.0.1)
psutil (5.0.0)
pyasn1 (0.1.9)
python-dateutil (2.5.3)
rsa (3.4.2)
s3transfer (0.1.1)
setuptools (29.0.1)
six (1.10.0)
supervisor (3.1.3)
uWSGI (2.0.12)
wheel (0.29.0)

`show_map` on a cleared map returns `['']`

import haproxyadmin
hap = haproxyadmin.haproxy.HAProxy(socket_file="/var/lib/haproxy/stats")
hap.clear_map(1) 
hap.show_map("/etc/haproxy/backend.map") # => ['']

Expected it to return []

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.