tsuru / healthcheck-as-a-service Goto Github PK

tsuru service API for on demand web application monitoring.

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.50% Python 99.33% HTML 0.09% Procfile 0.07%

healthcheck-as-a-service's Introduction

healthcheck-as-a-service

This project is a http API to abstract healthcheck operations, like verify if a web application is working fine and notify to some users if some of this applications are in trouble.

supported backends:

zabbix

configuring

API_URL - the api base url
API_DEBUG - enables the debug mode

zabbix backend

ZABBIX_URL - the zabbix api endpoint
ZABBIX_USER - zabbix user
ZABBIX_PASSWORD - zabbix password
ZABBIX_HOST_GROUP - host group used to create the web monitoring
ZABBIX_HOST - host used to create the web monitoring

mongodb storage

MONGODB_DATABASE - default is hcapi
MONGODB_URI - mongodb full address

deploying

zabbix (optional)

$ git clone [email protected]:zabbix/zabbix-docker.git
$ cd zabbix-docker
$ docker-compose -f docker-compose_v3_alpine_mysql_latest.yaml up -d

hcaas

$ git clone [email protected]:tsuru/healthcheck-as-a-service.git
$ cd healthcheck-as-a-service
$ tsuru app-create hcaas python
$ tsuru env-set -a hcaas API_DEBUG=true ZABBIX_URL=$ZABBIX_URL MONGODB_URI=$MONGODB_URI
$ tsuru app-deploy -a hcaas .
$ export API_URL=$(tsuru app-info -a hcaas | grep Address: |awk '{print $2}')

installing healthcheck tsuru plugin

$ tsuru plugin-install hc <API-URL>/plugin

using healthcheck tsuru plugin

creating new service

edit service.yaml endpoint:production with hcaas address

$ tsuru service create service.yaml

creating a new healtcheck

$ tsuru service-instance-add <healthcheck-service> <healthcheck-name>

removing a healthcheck

$ tsuru service-instance-remove <healthcheck-service> <healthcheck-name>

adding a new url to be monitored

$ tsuru hc add-url <healthcheck-service> <healthcheck-name> <url> [expected string]

removing a url

$ tsuru hc remove-url <healthcheck-name> <url>

adding a new watcher

$ tsuru hc add-watcher <healthcheck-service> <healthcheck-name> <email>

removing a watcher

$ tsuru hc remove-watcher <healthcheck-name> <email>

listing service hostgroups

$ tsuru hc list-service-groups <healthcheck-service> [search-keyword]

adding instance to a hostgroup

$ tsuru hc add-group <healthcheck-service> <healthcheck-name> <hostgroup-name>

removing instance from a hostgroup

$ tsuru hc-remove-group <healthcheck-service> <healthcheck-name> <hostgroup-name>

listing instance hostgroups

$ tsuru hc list-groups <healthcheck-service> <healthcheck-name>

development

Pull requests are very welcomed! Make sure your patches are well tested.

Running tests

If you are using a virtualenv, all you need is:

$ make test

healthcheck-as-a-service's People

Contributors

Stargazers

Watchers

Forkers

pombredanne msabramo nedmax cloudxtreme marcusaram scorphus bhyvex hadokencode cezarsa kjmusic50 iamsingularity guilhermebr gabrielgene rafilkmp3 raphaelvrossi

healthcheck-as-a-service's Issues

create zabbix backend

return a better error message when an url does not exists

If we try to remove an url that does not exists the plugin will rase a http 500 error.

Brackets in subject are grouping different hosts in the same email thread

I will submit a PR.

backends/zabbix: cannot add watchers that already exists in Zabbix

add-watcher fails to add a user that already exists in Zabbix. Currently, we check if the user exists in our database, but not in Zabbix, so it works for users that the API creates, but do not work with pre-registered users.

plugin: handle duplicate url error

plugin send a http 500 error and show a traceback to user. We should handle it in a better way.

plugin: list-urls doesn't return comment related with url

When add a new healthcheck, comment related with url is added on Zabbix but healthcheck plugin doesn't return any data from Zabbix trigger.

admin 403 forbidden

when I try to list a hc that was not created by my team

$ tsuru hc list-urls whc
Traceback (most recent call last):
  File "/Users/wagnersza/.tsuru/plugins/hc", line 243, in <module>
    main(sys.argv[1], *sys.argv[2:])
  File "/Users/wagnersza/.tsuru/plugins/hc", line 235, in main
    command(cmd)(*args)
  File "/Users/wagnersza/.tsuru/plugins/hc", line 111, in list_urls
    response = proxy_request(name, "GET", url, "", headers)
  File "/Users/wagnersza/.tsuru/plugins/hc", line 51, in proxy_request
    return urllib2.urlopen(request, timeout=30)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

add oauth support

the next tsuru version will support oauth. tsuru will send an authorization token for each request to the api. the api should verify if the user is valid and has authorization needed to do the operation requested.

return a better error message when an url already exists

Currently, the api and the plugins returns the HTTP 500 error.

plugin: add command to url status

hc status-url

remove fails after adding watchers

After wadding watchers to the instance, we can no longer remove it without removing the watcher first. The API tries to remove the user group before removing the user itself. Here's the error we get from logs:

2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]: Error handling request
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]: Traceback (most recent call last):
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/sync.py", line 131, in handle_request
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     respiter = self.wsgi(environ, resp.start_response)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     return self.wsgi_app(environ, start_response)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     response = self.make_response(self.handle_exception(e))
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     reraise(exc_type, exc_value, tb)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     response = self.full_dispatch_request()
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     rv = self.handle_user_exception(e)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     reraise(exc_type, exc_value, tb)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     rv = self.dispatch_request()
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     return self.view_functions[rule.endpoint](**req.view_args)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/home/application/current/healthcheck/auth.py", line 25, in decorated
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     return fn(*args, **kwargs)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/home/application/current/healthcheck/api.py", line 97, in remove
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     get_manager().remove(name)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/home/application/current/healthcheck/backends/__init__.py", line 91, in remove
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     self._remove_group(healthcheck.group_id)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/home/application/current/healthcheck/backends/__init__.py", line 183, in _remove_group
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     self.zapi.usergroup.delete(id)
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/pyzabbix/__init__.py", line 140, in fn
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     args or kwargs
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/pyzabbix/__init__.py", line 117, in do_request
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]:     raise ZabbixAPIException(msg, response_json['error']['code'])
2014-09-03 17:41:18 -0300 [api][49185a6aa9aa]: ZabbixAPIException: ('Error -32602: Invalid params., User group "hc-chico" is the only group that user "[email protected]" belongs to. while sending {\'params\': (u\'45\',), \'jsonrpc\': \'2.0\', \'method\': \'usergroup.delete\', \'auth\': u\'redacted\', \'id\': 1}', -32602)

plugin: add command to list watchers

hc list-watcher <hcname>

plugin: add command to list urls

hc list-url <hcname>

plugin: should handle expired tsuru token in the proxy

Here's what happened when I used a expired token:

Traceback (most recent call last):
  File "/Users/f/.tsuru/plugins/hc", line 201, in <module>
    main(sys.argv[1], *sys.argv[2:])
  File "/Users/f/.tsuru/plugins/hc", line 193, in main
    command(cmd)(*args)
  File "/Users/f/.tsuru/plugins/hc", line 93, in remove_url
    proxy_request(name, "DELETE", "/url", body=body, headers=headers)
  File "/Users/f/.tsuru/plugins/hc", line 51, in proxy_request
    return urllib2.urlopen(request, timeout=30)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized

all operations should be atomic

We deal with multiple stuff: groups, users, items, web monitoring, mongodb, etc. We need to make sure that we're always in consistent state.

cannot add watcher to multiple instances

Every watcher creation is a new user, so it's not possible to add a watcher to multiple instances. For instance:

% tsuru hc add-watcher instance1 [email protected]
ok
% tsuru hc add-watcher instance2 [email protected]
BOOM

Here is the log from the API:

2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]: Error handling request
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]: Traceback (most recent call last):
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/sync.py", line 131, in handle_request
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     respiter = self.wsgi(environ, resp.start_response)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     return self.wsgi_app(environ, start_response)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     response = self.make_response(self.handle_exception(e))
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     reraise(exc_type, exc_value, tb)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     response = self.full_dispatch_request()
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     rv = self.handle_user_exception(e)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     reraise(exc_type, exc_value, tb)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     rv = self.dispatch_request()
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     return self.view_functions[rule.endpoint](**req.view_args)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/home/application/current/healthcheck/auth.py", line 25, in decorated
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     return fn(*args, **kwargs)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/home/application/current/healthcheck/api.py", line 74, in add_watcher
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     get_manager().add_watcher(name, watcher)
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/home/application/current/healthcheck/backends/__init__.py", line 77, in add_watcher
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     "period": "1-7,00:00-24:00",
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/pyzabbix/__init__.py", line 140, in fn
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     args or kwargs
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:   File "/usr/local/lib/python2.7/dist-packages/pyzabbix/__init__.py", line 117, in do_request
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]:     raise ZabbixAPIException(msg, response_json['error']['code'])
2014-09-03 17:29:13 -0300 [api][49185a6aa9aa]: ZabbixAPIException: ('Error -32602: Invalid params., User with alias "[email protected]" already exists. while sending {\'params\': {\'passwd\': \'\', \'alias\': u\'[email protected]\', \'user_medias\': [{\'mediatypeid\': \'1\', \'active\': 0, \'period\': \'1-7,00:00-24:00\', \'severity\': 63, \'sendto\': u\'[email protected]\'}], \'usrgrps\': [u\'46\']}, \'jsonrpc\': \'2.0\', \'method\': \'user.create\', \'auth\': u\'redacted\', \'id\': 1}', -32602)