Python client for Consul (http://www.consul.io/)
License: MIT License
This repo was an early Python client for Consul.io.
It was maintained roughly from 2014 to 2018 until the original maintainers stopped using Consul in their day-to-day stack.
As of April 2024, criteo/py-consul appears to be the most well maintained fork.
🙏💚
As per the docs here: https://www.consul.io/docs/agent/http/agent.html#agent_service_register
Shouldn't agent.service.register take an IP address?
Perhaps I'm being a consul noob, but it seems hard to get a service with an IP address using the agent interface instead of the catalog interface...
@cablehead aad9ee4 is missing on master.
When a 'register' call (either catalog.register or agent.service.register) results in a 400 'bad request' response from consul, the body of the response usually contains a fairly clear error.
Is there a way to make that error visible?
I am getting errors and by looking at exceptions, it seems python2.7 code is being executed.
Here is the traceback on my ipython console.
Any comments and help is appreciated.
(pyenv3.4)[root@ip-172-31-18-118 consul]$ ipython
Python 3.4.3 (default, Dec 14 2015, 11:41:31)
Type "copyright", "credits" or "license" for more information.
IPython 4.0.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import consul
In [2]: c = consul.Consul(port=8600, host="localhost")
In [3]: c.agent.services()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, **httplib_request_kw)
375 try: # Python 2.7, use buffering of HTTP responses
--> 376 httplib_response = conn.getresponse(buffering=True)
377 except TypeError: # Python 2.6 and older
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
BadStatusLine Traceback (most recent call last)
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeou
t, release_conn, **response_kw)
558 timeout=timeout_obj,
--> 559 body=body, headers=headers)
560
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, **httplib_request_kw)
377 except TypeError: # Python 2.6 and older
--> 378 httplib_response = conn.getresponse()
379 except (SocketTimeout, BaseSSLError, SocketError) as e:
/usr/local/lib/python3.4/http/client.py in getresponse(self)
1170 try:
-> 1171 response.begin()
1172 assert response.will_close != _UNKNOWN
/usr/local/lib/python3.4/http/client.py in begin(self)
350 while True:
--> 351 version, status, reason = self._read_status()
352 if status != CONTINUE:
/usr/local/lib/python3.4/http/client.py in _read_status(self)
320 # sending a valid response.
--> 321 raise BadStatusLine(line)
322 try:
BadStatusLine: ''
During handling of the above exception, another exception occurred:
ProtocolError Traceback (most recent call last)
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
369 retries=self.max_retries,
--> 370 timeout=timeout
371 )
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeou
t, release_conn, **response_kw)
608 retries = retries.increment(method, url, error=e, _pool=self,
--> 609 _stacktrace=sys.exc_info()[2])
610 retries.sleep()
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
244 if read is False:
--> 245 raise six.reraise(type(error), error, _stacktrace)
246 elif read is not None:
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/packages/six.py in reraise(tp, value, tb)
308 if value.__traceback__ is not tb:
--> 309 raise value.with_traceback(tb)
310 raise value
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeou
t, release_conn, **response_kw)
558 timeout=timeout_obj,
--> 559 body=body, headers=headers)
560
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, **httplib_request_kw)
377 except TypeError: # Python 2.6 and older
--> 378 httplib_response = conn.getresponse()
379 except (SocketTimeout, BaseSSLError, SocketError) as e:
/usr/local/lib/python3.4/http/client.py in getresponse(self)
1170 try:
-> 1171 response.begin()
1172 assert response.will_close != _UNKNOWN
/usr/local/lib/python3.4/http/client.py in begin(self)
350 while True:
--> 351 version, status, reason = self._read_status()
352 if status != CONTINUE:
/usr/local/lib/python3.4/http/client.py in _read_status(self)
320 # sending a valid response.
--> 321 raise BadStatusLine(line)
322 try:
ProtocolError: ('Connection aborted.', BadStatusLine("''",))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
<ipython-input-3-c8036864a656> in <module>()
----> 1 c.agent.services()
/usr/local/pyenv3.4/lib/python3.4/site-packages/consul/base.py in services(self)
533 """
534 return self.agent.http.get(
--> 535 lambda x: json.loads(x.body), '/v1/agent/services')
536
536
537 def checks(self):
/usr/local/pyenv3.4/lib/python3.4/site-packages/consul/std.py in get(self, callback, path, params)
32 uri = self.uri(path, params)
33 return callback(self.response(
---> 34 self.session.get(uri, verify=self.verify)))
35
36 def put(self, callback, path, params=None, data=''):
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/sessions.py in get(self, url, **kwargs)
478
479 kwargs.setdefault('allow_redirects', True)
--> 480 return self.request('GET', url, **kwargs)
481
482 def options(self, url, **kwargs):
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream,
verify, cert, json)
466 }
467 send_kwargs.update(settings)
--> 468 resp = self.send(prep, **send_kwargs)
469
470 return resp
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/sessions.py in send(self, request, **kwargs)
574
575 # Send the request
--> 576 r = adapter.send(request, **kwargs)
577
578 # Total elapsed time of the request (approximately)
/usr/local/pyenv3.4/lib/python3.4/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
410
411 except (ProtocolError, socket.error) as err:
--> 412 raise ConnectionError(err, request=request)
413
414 except MaxRetryError as e:
ConnectionError: ('Connection aborted.', BadStatusLine("''",))
In [4]:
I burned a few hours on this, so hopefully this will save the next person a future headache.
I used python-consul to register a new service, with a corresponding health check. Next, I tried setting the service to healthy with: `consul.agent.check.ttl_pass('service:MY_SERVICE_ID').
This was completely ignored by Consul, which would return a 200 OK but leave the health check in a critical state.
The problem ended up being that I initialized the check as Check.ttl(60). This should be Check.ttl('60s'). I'm not sure what Consul does with an integer TTL, but it's nothing good. It expects a string TTL, so python-consul should probably cast it before sending it on.
Hi there, after install consul
import consul
then...
.../python2.7/site-packages/consul/init.py in ()
version = '0.3.10'
from consul.std import Consul
.../python2.7/site-packages/consul/std.py in ()
from six.moves import urllib
ImportError: cannot import name urllib
Hey, we're likely to add a twisted compatible version of the client in the next few weeks. Is it ok to depend on any extra dependencies? I'd prefer to use treq than the raw twisted Agent class.
Cheers!
Sorry I don't have time to dig into this and help resolve it right now. Will try to circle around and get back to it, but I wanted to log a potential issue. Getting potentially unwaranted error message when I try to query Consul (using python-consul, of course) for a service with a tag specifier.
This code:
def simple_passthru_consul_catalog_service(self, service_name, tag):
consul_ = consul.Consul('consul')
print consul_.catalog.services()
return consul_.catalog.service(service="redis-holt-overflow", tag="master")
Produces this output and error message:
('44', {u'consul-8301': [u'udp'], u'consul-8300': [], u'consul-8302': [u'udp'], u'consul': [], u'redis-holt-overflow': [u'master', u'slave'], u'consul-8400': [], u'consul-53': [u'udp'], u'consul-8500': []})
Attempt #9 of 60 failed.
ERROR:root:Consul connection problem - Exception:No JSON object could be decoded
Traceback (most recent call last):
File "/bin/python/consul_def.py", line 72, in get_consul_service_with_name_withfunc
[index, json_objs] = consul_catalog_service_func("redis-holt-overflow", "master")
File "/bin/python/consul_def.py", line 59, in simple_passthru_consul_catalog_service
return consul_.catalog.service("redis-holt-overflow", "master")
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 1079, in service
'/v1/catalog/service/%s' % service, params=params)
File "/usr/local/lib/python2.7/dist-packages/consul/std.py", line 30, in get
return callback(self.response(requests.get(uri)))
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 119, in cb
data = json.loads(response.body)
File "/usr/lib/python2.7/json/init.py", line 326, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
For reference here is the curl response directly from Consul:
curl -s localhost:8500/v1/catalog/service/redis-holt-overflow
[{"Node":"consul","Address":"192.168.59.103","ServiceID":"fearbattle-lm:redis.m:6380","ServiceName":"redis-holt-overflow","ServiceTags":["master"],"ServiceAddress":"","ServicePort":6380},{"Node":"consul","Address":"192.168.59.103","ServiceID":"fearbattle-lm:redis.s:6381","ServiceName":"redis-holt-overflow","ServiceTags":["slave"],"ServiceAddress":"","ServicePort":6381}]
Again, my apologies for not working out a solution or digging deeper. Under pressure right now.
Thee is simple script with consul.aio:
import asyncio
import consul.aio
loop = asyncio.get_event_loop()
c = consul.aio.Consul()
@asyncio.coroutine
def go():
session = yield from c.session.create(ttl=20)
assert session
response = yield from c.session.destroy(session)
assert session
if __name__ == '__main__':
loop.run_until_complete(go())
If I run it I'll get an exception:
Unclosed connector
connector: <aiohttp.connector.TCPConnector object at 0x7f9dc0121cc0>
Exception ignored in:
Python 3.4.3
aiohttp 0.16.3
python-consul 0.3.19
A ServiceID and ServiceName can be specified when registering a check in consul. They are less obvious in the API docs (I plan on making a PR to update the docs) but here is an example of it working.
bash-4.1# cat service.json
{
"ID": "redis1",
"Name": "redis"
}
bash-4.1# cat check.json
{
"ID": "mem",
"Name": "Memory utilization",
"Notes": "Ensure we don't oversubscribe memory",
"Script": "/usr/local/bin/check_mem.py",
"Interval": "10s",
"ServiceName": "redis",
"ServiceID": "redis1"
}
bash-4.1# curl -X PUT -H 'Content-Type: application/json' -d @service.json 172.17.0.21:8500/v1/agent/service/register
bash-4.1# curl -X PUT -H 'Content-Type: application/json' -d @check.json 172.17.0.21:8500/v1/agent/check/register
bash-4.1# curl 172.17.0.21:8500/v1/agent/checks
{"mem":{"Node":"node4","CheckID":"mem","Name":"Memory utilization","Status":"critical","Notes":"Ensure we don't oversubscribe memory","Output":"/bin/bash: /usr/local/bin/check_mem.py: No such file or directory\n","ServiceID":"redis1","ServiceName":"redis"}}
If a ServiceID is provided, a ServiceName can optionally be provided. If the ServiceName is provided without the ServiceID, it is not registered in the check.
/cc @sgargan for your interest
Hey there,
love consul, consul-template and the python client.
I would like to supress the INFO logs fired up when a client connects. How can I change that to DEBUG?
Depending on the conncetion method, correct?
kv.get assumes JSON-encoded data has been stored, but kv.put does not encode as JSON.
I would be happy to submit a patch if required.
Is it feasible for this lib to follow http redirects (either by default or optional)? This is required in our environment and doesn't seem hard to add.
Thanks!
Hi there I just did a pip install python-consul and got...
Installing collected packages: python-consul
Running setup.py install for python-consul
File "/home/andre/.venvs/Celery3.1/lib/python2.7/site-packages/consul/aio.py", line 31
resp = yield from aiohttp.request(method, uri,
^
SyntaxError: invalid syntax
But...
Successfully installed python-consul
Hello,
just had this problem today, although this might be a bit of a odd case, but I was in an environment with an old version of request (0.12) and tried to used python-consul from it, but it was failing with the following error:
File "/usr/lib/python2.7/dist-packages/consul/base.py", line 1642, in create
callback, '/v1/acl/create', params=params, data=data)
File "/usr/lib/python2.7/dist-packages/consul/std.py", line 39, in put
self.session.put(uri, data=data, verify=self.verify)))
File "/usr/lib/python2.7/dist-packages/consul/std.py", line 23, in response
response.status_code, response.headers, response.text)
File "/usr/lib/python2.7/dist-packages/requests/models.py", line 809, in text
content = str(self.content, encoding, errors='replace')
TypeError: unicode() argument 2 must be string, not None
In the requirements.txt of python-consul, there is no version specified for requests. I don't know which is the lowest version of requests which works with python-consul, but I know it's greater than 0.12. If this could be updated, this would be great.
Can't see mention of it.
I'd like to be able to set a timeout on the requests made by requests.
http://docs.python-requests.org/en/latest/user/quickstart/#timeouts
I could work on a pull request if one of the following options looks acceptable.
consul = Consul(timeout=12)and/or
consul = Consul()
consul.acl.create(rules=rules, timeout=timeout)timeout would be optional in both cases.
In [11]: session = c.session.renew(s)
---------------------------------------------------------------------------
NotFound Traceback (most recent call last)
<ipython-input-11-979c9ee18dcd> in <module>()
----> 1 session = c.session.renew(s)
/usr/local/python2.7/lib/python2.7/site-packages/consul/base.pyc in renew(self, session_id, dc)
1454 return self.agent.http.put(
1455 callback(is_json=True, one=True, allow_404=False),
-> 1456 '/v1/session/renew/%s' % session_id, params=params)
1457
1458 class ACL(object):
/usr/local/python2.7/lib/python2.7/site-packages/consul/std.pyc in put(self, callback, path, params, data)
37 uri = self.uri(path, params)
38 return callback(self.response(
---> 39 self.session.put(uri, data=data, verify=self.verify)))
40
41 def delete(self, callback, path, params=None):
/usr/local/python2.7/lib/python2.7/site-packages/consul/base.pyc in cb(response)
113 raise ACLPermissionDenied(response.body)
114 if response.code == 404 and not allow_404:
--> 115 raise NotFound(response.body)
116 if is_200:
117 data = response.code == 200
Hi, I just noticed that there is no tag v0.3.18.
There are various places in code for example
https://github.com/cablehead/python-consul/blob/master/consul/base.py#L521
https://github.com/cablehead/python-consul/blob/master/consul/base.py#L535
which does a json.loads as a result of which things can break please check logs below
http://logs.openstack.org/62/245362/33/check/gate-tooz-tox-py27-consul/d0a069a/testr_results.html.gz
Do we really need a json.loads here can we make it optional ?
Change is here https://review.openstack.org/#/c/245362/33/tooz/drivers/consul.py
I am available on #openstack-state-management with username vilobhmm for more details.
Add support for /v1/agent/service/maintenance/ and /v1/agent/maintenance endpoints.
The API documentation were recently partially deleted. any chance of getting those back?
http://python-consul.readthedocs.org/en/latest/#api-documentation
Hello,
Is there a way to update a particular tag of already registered service? I know re-registering the service would work but then that's a pain and unsafe of losing out original service_id etc.. I just want to update a tag or list of tags of registered services ( I would query them by service_name or so).
Thanks.
Time of method consul.Health.state('any') is much more longer than native HTTP query (about 4 times longer, 2 to 7 sec) with 50+ services. We have latest version from pip.
KV.get handles response in callback.
If response has 403 status code (rpc error: rpc error: ACL not found) then we have ValueError: No JSON object could be decoded.
def callback(response):
if response.code == 500:
raise ConsulException(response.body)
elif response.code == 404:
data = None
"""Bug is here"""
else:
data = json.loads(response.body)
if not keys:
for item in data:
if item.get('Value') is not None:
item['Value'] = base64.b64decode(item['Value'])
if not recurse:
data = data[0]
return response.headers['X-Consul-Index'], dataAny thoughts about refactoring the classes in base.py it into separate modules? Is this something you'd like to see @cablehead?
Hello,
I would like to do service lookup ( I have only service id, service name and one tag with me).
As of now , in Python-Consul, following methods support lookup of service(s):
Thanks.
It doesn't appear watches are currently supported, having them would be great! (or letting me know they exist); something akin to https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py could be a pattern that is emulated.
Can't see a mention of it in the docs.
I must have been on crack.
Could you add the wait option to the kv.get() function, so that it is possible to specify a timeout when waiting for a key to change?
Thanks
curl -X PUT -d '{"behavior":"delete"}' localhost:8500/v1/session/create
{"ID":"acf52e93-6b45-6297-6425-bb97a340b144"}
import consul
c=consul.Consul()
c.session.create(name='aaa',behavior='delete')
Traceback (most recent call last):
File "", line 1, in
TypeError: create() got an unexpected keyword argument 'behavior'
python-consul:
class Session(object):
def init(self, agent):
self.agent = agent
self.check = Consul.Health.Check(agent)
def create(
self,
name=None,
node=None,
checks=None,
lock_delay=15,
dc=None):
the 'behavior' is lack in create()
Add the ability to use SSL to encrypt requests traffic if consul is fronted by an SSL offloading load balancer.
Hi,
I noticed that the docs don't clearly state the behaviour of kv.get('key',...) if there is no key. I think it would be great to be explicit on this point. The current behaviour seems to be to return None for the value, but 1 for the index which seems a tad odd...
I think that an unset key should either return an index of None, or riase an exception. (I feel the exception is the right model). If you like, let me know which you prefer, and I will submit a pull request with a proposed fix.
Thanks, Shaheed
For example:
assert not name.startswith('/'), 'keys should not start with a forward slash'Otherwise it's pretty hard to know whats going on with an AssertionError if you happen to want to catch it.
I'm not sure if there's a plan for fully supporting Python 3.x but when I try to query catalog for a service that has a particular tag I get the following error:
----> 1 c.catalog.service('database', tag='developmentd')
/usr/local/lib/python3.4/dist-packages/consul/base.py in service(self, service, dc, tag, index, consistency)
563 return self.agent.http.get(
564 callback(is_indexed=True),
--> 565 '/v1/catalog/service/%s' % service, params=params)
566
567 class Health(object):
/usr/local/lib/python3.4/dist-packages/consul/std.py in get(self, callback, path, params)
26
27 def get(self, callback, path, params=None):
---> 28 uri = self.uri(path, params)
29 return callback(self.response(requests.get(uri)))
30
/usr/local/lib/python3.4/dist-packages/consul/std.py in uri(self, path, params)
23 if not params:
24 return uri
---> 25 return '%s?%s' % (uri, urllib.urlencode(params))
26
27 def get(self, callback, path, params=None):
AttributeError: 'module' object has no attribute 'urlencode'
In Python 3 urlencode is in urllib.parse.urlencode.
I might be doing something stupid, but I’m starting out with Consul and I bind a token using c = Consul(token="my-token"), can verify it’s on c.agent.token, but it’s not sent when I try to register a service. Which leads to a local success but entries like
2015/10/21 20:43:12 [WARN] agent: Service 'metrics.test' registration blocked by ACLs
in my consul logs. So the data never reaches my cluster.
Glancing over https://github.com/cablehead/python-consul/blob/master/consul/base.py#L593-L655 looks like it completely ignores the token? Or is it me?
Can you add session endpoint and then add an acquire kwarg to KV.put()?
Hi,
We'd like to use Consul over HTTPS using a self-signed certificate that we trust.
The only way we found that works is using the environment variable REQUESTS_CA_BUNDLE. But this is global for the whole process and we use requests for other HTTP queries where the custom CA is not necessarily wanted.
Requests provides a verify argument for its session/requests where we can give a custom CA bundle, but python-consul does not.
We may write the pull request, but first : would you accept the feature ?
Also, should we allow disabling certificate verification completely ?
I didn't find where?
Best match: python-consul 0.4.7
Downloading https://pypi.python.org/packages/source/p/python-consul/python-consul-0.4.7.tar.gz#md5=e695b8897d1e84eb23a8d9d20a6d47da
Processing python-consul-0.4.7.tar.gz
Writing /var/folders/tw/wh1w60j512j7lks_1nhtmq2r002js8/T/easy_install-7x5eun/python-consul-0.4.7/setup.cfg
Running python-consul-0.4.7/setup.py -q bdist_egg --dist-dir /var/folders/tw/wh1w60j512j7lks_1nhtmq2r002js8/T/easy_install-7x5eun/python-consul-0.4.7/egg-dist-tmp-I8VvED
File "build/bdist.macosx-10.10-intel/egg/consul/aio.py", line 41
resp = yield from aiohttp.request(method, uri,
^
SyntaxError: invalid syntax
zip_safe flag not set; analyzing archive contents...
Copying python_consul-0.4.7-py2.7.egg to /Users/vilobhmm/source/mitaka/add-consul-driver/tooz/.venv/lib/python2.7/site-packages
Adding python-consul 0.4.7 to easy-install.pth file
With Consul.Health.service() I can pull the current health of a specified service. I'd like to be able to pull the current health of all services. This is how I do it with the consul CLI:
consul watch -type=checks cat
Like the current Health.service(), I need to use an index so that I can continually pull changes to the health checks. It looks like this is not possible with Consul.Health.service(), and I don't see how to do it with any of the other classes.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.