hashicorp/consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

• Website: https://www.consul.io
• Tutorials: HashiCorp Learn
• Forum: Discuss

Consul provides several key features:

• Multi-Datacenter - Consul is built to be datacenter aware, and can support any number of regions without complex configuration.

• Service Mesh - Consul Service Mesh enables secure service-to-service communication with automatic TLS encryption and identity-based authorization. Applications can use sidecar proxies in a service mesh configuration to establish TLS connections for inbound and outbound connections with Transparent Proxy.

• API Gateway - Consul API Gateway manages access to services within Consul Service Mesh, allow users to define traffic and authorization policies to services deployed within the mesh.

• Service Discovery - Consul makes it simple for services to register themselves and to discover other services via a DNS or HTTP interface. External services such as SaaS providers can be registered as well.

• Health Checking - Health Checking enables Consul to quickly alert operators about any issues in a cluster. The integration with service discovery prevents routing traffic to unhealthy hosts and enables service level circuit breakers.

• Dynamic App Configuration - An HTTP API that allows users to store indexed objects within Consul, for storing configuration parameters and application metadata.

Consul runs on Linux, macOS, FreeBSD, Solaris, and Windows and includes an optional browser based UI. A commercial version called Consul Enterprise is also available.

Please note: We take Consul's security and our users' trust very seriously. If you believe you have found a security issue in Consul, please responsibly disclose by contacting us at [email protected].

Quick Start

A few quick start guides are available on the Consul website:

• Standalone binary install: https://learn.hashicorp.com/collections/consul/get-started-vms
• Minikube install: https://learn.hashicorp.com/tutorials/consul/kubernetes-minikube
• Kind install: https://learn.hashicorp.com/tutorials/consul/kubernetes-kind
• Kubernetes install: https://learn.hashicorp.com/tutorials/consul/kubernetes-deployment-guide
• Deploy HCP Consul: https://learn.hashicorp.com/tutorials/consul/hcp-gs-deploy

Documentation

Full, comprehensive documentation is available on the Consul website: https://consul.io/docs

Contributing

Thank you for your interest in contributing! Please refer to CONTRIBUTING.md for guidance. For contributions specifically to the browser based UI, please refer to the UI's README.md for guidance.

consul's Issues

Increase Raft size limit to >500 MB on 64-bit systems

Changes to 'tags' etc..

How open are you guys to thinking about alternate tagging for service nodes?

For my purposes, I'd really like to be able to tag a node as running both a macro version of a service (Authentication v2.0 for example) and also possibly versions of specific messages (whether it's REST HTTP JSON or protocol buffers or amqp envelope bodies...) and then be able to resolve something like this by having some things have default values...

client_grant.v1.authentication.v2.us-west-1a.consul 
client_grant.authentication.us-west-1a.consul
authentication.consul

and being able to publish serf tags as appropriate to create these records.

what do you think?

Bootstrap while not a server?

Does it make sense that -bootstrap works when the agent isn't in -server mode? It lets me do it, but I'm not sure if this makes sense and the documentation doesn't make this clear. Actually, it makes it sounds like it requires server:

The initial server must be in bootstrap mode. 

Enforce minimum check interval

People can shoot themselves in the foot with a very low check interval if they do not use a suffix like "s" or "m". This causes a health check to run nearly continuously. Prevent this by enforcing a hard minimum of 1 second. Also try to be somewhat intelligent and check if an interval is set to be nanoseconds and correct to seconds.

CLI to register/degister/list services etc.

Integrating this into prod now, running into an issue where registering services kind of sucks or is really jank. I understand that for some services they can use a Consul lib at some point to register. But, for example, things like Statsite, what I want to do is add the registration as part of the init script.

Right now, I have to use curl which works but is jank. I'd love to see something like this:

$ consul service-register service.json
...
$ cat service.json | consul service-register -

And so on.

Can't register service with HTTP API by curl in command line ,"Missing Service Name"

Thanks for develop this amazing system. My team are testing consul on 5 nodes small cluster(3 server+2 client)

Consul v0.1.1.dev (b8d185a+CHANGES)
Consul Protocol: 1 (Understands back to: 1)

when running the following:

curl -d '{"Service":{"ID":"1234","Name":"mysql-1","Tags":["slave"],"Port":3306,"check":{"Name":"MySQL Service Check","Script":"/root/mysqlcheck.sh orz1bsg","Interval":"3s"}}}' http://127.0.0.1:8500/v1/agent/service/register

It failed to register a service and reports "missing service name". Tried on each node and got the same result.

It works Ok when write the above json to a file like /etc/consul.d/mysql_chk.json and loaded when consul join the cluster or using kill -s SIGHUP.

Appreciate if there is clue.

Thanks,

Joe

Improve reliability of anti-entropy when agent is bootstrap server

The agent does not always run anti-entropy immediately after acquiring leadership when it is a server+bootstrap. This causes a 30 second delay at times.

Add code coverage + testing badges

Nil pointer starting agent w/ -server option

Trying to start consul in server mode using:

bin/consul agent -server -data-dir /tmp/consul

crashes with this stacktrace:

==> WARNING: It is highly recommended to set GOMAXPROCS higher than 1
==> Starting Consul agent...
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x81dfb0b]

goroutine 1 [running]:
runtime.panic(0x83e8c20, 0x8952748)
/usr/local/go/src/pkg/runtime/panic.c:266 +0x9a
sync/atomic.StoreUint64(0x18c8c5b4, 0x0, 0x0, 0x825f295)
/usr/local/go/src/pkg/sync/atomic/asm_386.s:196 +0xb
github.com/hashicorp/raft.(_raftState).setCurrentTerm(0x18c8c5b0, 0x0, 0x0)
/home/jwilder/go/src/github.com/hashicorp/raft/state.go:74 +0x42
github.com/hashicorp/raft.(_Raft).setCurrentTerm(0x18c8c5b0, 0x0, 0x0)
/home/jwilder/go/src/github.com/hashicorp/raft/raft.go:1271 +0x135
github.com/hashicorp/raft.NewRaft(0x18c36500, 0xb74ccf98, 0x18c42f80, 0xb74ccfb8, 0x18c9c820, ...)
/home/jwilder/go/src/github.com/hashicorp/raft/raft.go:177 +0x74a
github.com/hashicorp/consul/consul.(_Server).setupRaft(0x18c01e60, 0x0, 0x0)
/home/jwilder/go/src/github.com/hashicorp/consul/consul/server.go:280 +0x55f
github.com/hashicorp/consul/consul.NewServer(0x18c45d20, 0x18c45d20, 0x8359c40, 0x18c90f40)
/home/jwilder/go/src/github.com/hashicorp/consul/consul/server.go:169 +0x49f
github.com/hashicorp/consul/command/agent.(_Agent).setupServer(0x18c4df50, 0x18c8c410, 0x18c0ae10)
/home/jwilder/go/src/github.com/hashicorp/consul/command/agent/agent.go:189 +0x34
github.com/hashicorp/consul/command/agent.Create(0x18c8c410, 0xb74cbe38, 0x18c42bd0, 0x0, 0x806e4f3, ...)
/home/jwilder/go/src/github.com/hashicorp/consul/command/agent/agent.go:103 +0x40b
github.com/hashicorp/consul/command/agent.(_Command).setupAgent(0x18c0ac30, 0x18c8c410, 0xb74cbe38, 0x18c42bd0, 0x18c904c0, ...)
/home/jwilder/go/src/github.com/hashicorp/consul/command/agent/command.go:147 +0x69
github.com/hashicorp/consul/command/agent.(_Command).Run(0x18c0ac30, 0x18c0a010, 0x3, 0x3, 0x0)
/home/jwilder/go/src/github.com/hashicorp/consul/command/agent/command.go:241 +0x4e1
github.com/mitchellh/cli.(*CLI).Run(0x18c31b80, 0x18c31b80, 0x844cae8, 0x9)
/home/jwilder/go/src/github.com/mitchellh/cli/cli.go:69 +0x1d7
main.realMain(0x8058913)
/home/jwilder/go/src/github.com/hashicorp/consul/main.go:36 +0x1e0
main.main()
/home/jwilder/go/src/github.com/hashicorp/consul/main.go:12 +0x1e

Consul should default DNS port to 53 for recursor

I specified -recursor 8.8.8.8 thinking that should just work. It launched successfully then only showed this error much later:

    2014/02/22 09:09:06 [ERR] dns: recurse failed: dial udp: missing port in address 8.8.8.8

We should probably just default to port 53 if no port is specified since any recursing DNS server will likely be on this port anyways.

Filtered services by health check state

It would be really helpful if there was an endpoint to get an array of healthy services, in other words filtered by state:

Multiple options:
http://localhost:8500/v1/health/state/{state}/service/{service}
http://localhost:8500/v1/health/state/{state}?service={service}
http://localhost:8500/v1/catalog/service/{service}?state={state}

For example for service foo if I want just the passing nodes:
http://localhost:8500/v1/catalog/service/foo?state=passing

[
    {
        "Address": "10.199.246.104",
        "Node": "dev-use1b-pr-16-thal-cnsl-0004",
        "ServiceID": "foo",
        "ServiceName": "foo",
        "ServicePort": 8000,
        "ServiceTags": [
            "version:1.0.0"
        ]
    }
]

Note, in this case I don't care to receive the check details (payload with Checkid, etc.) but rather the services themselves.

Happy to do the work and pull request if we can agree on the API.

CLI options for bind address

Seeing how -serf-bind indicates where the serf agent binds to, shouldn't server/http/dns/rpc have matching names, like -server-bind, -http-bind, etc.? Either that, or -serf-bind should be -serf-addr. I'll bet we have the same inconsistency in Serf's config options. It would be easy to straighten these up before a public release. Simple cosmetics though.

"Must provide a DataDir" is unclear

The output if -data isn't specified is "Must specify a DataDir". This should probably be "Must specify data directory using -data"

Also, might want to just make that flag -data-dir to be explicit

TLS support for Consul RPCs

Support use of TLS to secure the internal RPC communication. Gossip uses symmetric keys from Serf already.

Bind and drop privileges

Since Consul is meant to be able to act like a DNS server, it is preferable for it to be THE DNS server, running on port 53. Binding to this port requires root, but it is not ideal for consul itself to actually run as root.

Consul should take a -user flag so it can setuid to that user after binding ports.

make test fails

Not sure if tests should be green on master, but as of 5e129c3 there are intermittent failures (every other run fails). Log here

ultra-tiny mistake in the docs :)

http://www.consul.io/docs/agent/basics.html

Paragraph: Running an Agent

Example shows:

$ consul agent -data=/tmp/consul

When it should be

$ consul agent -data-dir=/tmp/consul

Add synced .gopath

When I go get inside VM, all the dependencies are in /opt/gopath of the guest VM however my text editor/shell is in the host. Therefore, it is kind of hard to take a look at the dependencies' src when necessary.

Suggested solution:

1. Create an empty folder .gopath in the repo
2. Synced that folder to the VM config.vm.synced_folder ".gopath/", "/opt/gopath"

Whenever we build in the VM we will have all the dependencies' src available in the host as well. Let me know if you have any other solution, I can make a PR for this.

Help text is too wide (> 80 chars)

On my console (fixed at 80 chars), the help text looks really bad. You need to make sure that -h help text doesn't go past that.

`consul services`

I really think there should be a command line consul services just like there is a consul members command. As I'm working through ops over here it would just be very handy...

Add a leader election module

Leader election ( and Lock ) modules are pretty useful.
It would be great to have this in consul.
https://github.com/coreos/etcd/blob/master/Documentation/modules.md#leader-election

make failed on os x

$ make
go get -d -v ./...
github.com/hashicorp/serf (download)
github.com/inconshreveable/muxado (download)
github.com/hashicorp/logutils (download)
github.com/mitchellh/cli (download)
github.com/ryanuber/columnize (download)
echo  | xargs -n1 go get -d
--> Installing dependencies to speed up builds...
# github.com/armon/gomdb
../../armon/gomdb/mdb.c:8507:46: warning: data argument not used by format string [-Wformat-extra-args]
/usr/include/secure/_stdio.h:49:56: note: expanded from macro 'sprintf'
# github.com/hashicorp/consul
github.com/armon/gomdb(__TEXT/__text): unexpected GOT reloc for non-dynamic symbol mdb_mid2l_insert
github.com/armon/gomdb(__TEXT/__text): unexpected GOT reloc for non-dynamic symbol mdb_mid2l_insert
make: *** [all] Error 2

Ideas?

Reconcile "reap" when no cluster leader

If a member fails and is subsequently "reaped" from the Serf pool, this will trigger the node to be deregistered on the cluster leader. However, if there is no cluster leader, this event is lost. All other member events deal with this using a periodic reconciliation. However, after a reap, the node is no longer in the Serf pool, but exists in the catalog. The presence in the catalog must be reconciled with the absence in Serf and treated as a "reap" event. However, this is an extreme edge case, and also complex as there is no simple way to do this currently.

Add bind address and port map

The agent has a large number of address based configurations, which is necessary due to the number of sub-systems to configure. However, its rather painful as an operator to set them all. Much nicer would be a single -bind address, and a port map {"http": 8500, "dns": 8600, ...} that would specify all the ports. This also cleans up the flags a lot, since there is only a single -bind and then potentially many -port-http -port-dns etc flags. Those would all be optional if the default ports are wanted.

consul doesn't run on Centos 6.5

consul

consul: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by consul)

Centos 6.5 ships with glibc 2.12.

Document potential stale read situation

/cc: @aphyr @jpfuentes2

We demand that all reads and writes are serialized through a leader. All writes naturally go through the Raft log, but we do reads directly from the leader state. The leader "leases" it's position for a configurable time, and checks that it is still leader each LeaderLeaseTimeout interval. However, if the leader is partitioned, a new leader could be elected and new writes processed. A read from the old leader could return a stale value in this situation.

This situation is hard to trigger, since by default a leader leases for 300msec, and a node only starts an election if it hasn't heard from the leader in 300msec - 600msec. This means generally the new election only happens after the old leader has already stepped down.

Document compiler prerequisites

Consul attempts to talk to itself over ipv6

For various reasons many of our boxes have ipv6 hard disabled. Consul appears to be trying to talk to itself over ipv6 which causes failure.

curl -v 'http://127.0.0.1:8500/v1/catalog/nodes'
* About to connect() to 127.0.0.1 port 8500 (#0)
*   Trying 127.0.0.1... connected
> GET /v1/catalog/nodes HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 127.0.0.1:8500
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< X-Consul-Index: 0
< Date: Fri, 18 Apr 2014 03:17:33 GMT
< Content-Length: 62
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0
failed to get conn: dial tcp [::]:8300: network is unreachable

PagerDuty integration

Cannot do DNS query for nodes containing periods in the name

The DNS query parser gets confused and refuses to handle it.

When using the JSON from the agent/check/register HTTP API, a JSON error is thrown

"cannot unmarshal string into Go value of type time.Duration"

This is fixed by removing the 's' in the Interval and/or TTL fields.

Ability to specify a config_dir in the -config-file

It would be useful from an operations point of view if when running you could just specify the -config-file. Then in the -config-file you could define a config_dir.

This way you can have your consul agent configuration in a file, for example /etc/consul.json, and your service/check configuration in a directory, e.g. /etc/consul.d/.

Reduce heap allocations

We seem to be doing a very high number of heap allocations. Not clear why.

Clear Raft peer set on graceful leave

Server node should clear the peer set on a graceful leave, otherwise we can cause issues on restart.

Service consumer registration

Maybe there is already a way to do this, but having a way of registering / deregistering a node's use of a service upon discovery might be helpful in building a service relationship graph and verifying at a glance that services are being discovered properly.

Support CNAME output if address to service isn't an IP

Raft got into state where it repeatedly messaged that last index is greater

After frequent restart (while testing an init script) I somehow managed to get consul/raft into a state where it rejected all messages. I attempted to set -log-level {debug,trace} but the only thing it would print is:

consul:     2014/04/22 21:58:11 [WARN] raft: Rejecting vote since our last index is greater

Deleting the -data-dir fixed the local issue.

Support for linux on ARM

Just interested to hear if anyone has tried this on an ARM device?

Was there any reasons why it wasn't cross compiled for ARM?

Thanks

[Feature proposal] Reverse proxy

Why?

DNS is susceptible to caching and disabling DNSSEC is slightly out of the comfort zone of ordinary deployment engineers. An interesting approach by airbnb/synapse is to use a reverse proxy.

Benefits?

• Common load-balancing strategy e.g. round-robin
• Automatic redispatch when error

Telemetry Requirements

I'd like to see some way to monitor Consul from an operations perspective.

It would be really great to be able to see

1. what services are most frequently requested
2. what services are being requested that aren't present at all (something failed somewhere!)
3. what services are up and running and never being requested (should be spun down)

Bootstrapping unclear in docs

This is a docs issue. This might be something that would be solved by a getting started section but I think it should be a top-level section in the main docs as well (under "Consul Agent"): "Bootstrapping"

Some open questions that aren't clear from the docs:

• I know the bootstrap flag exists, but do I run it ONLY on the first node? Can I run it on additional nodes as well?
• How do I bootstrap a cluster if they're all coming up at one time? i.e. a CloudFormation script.

Feature request: Debian package + repository

I want to start by saying I really appreciate the excellent work hashicorp are doing.

It would be great if you could consider hosting a Debian package repo for Consul, this makes it much easier to manage updates and automated deployment.

Personally I find reprepo a very quick-and-easy tool for managing repositories - quick howto here: http://anonscm.debian.org/gitweb/?p=mirrorer/reprepro.git;a=blob_plain;hb=HEAD;f=docs/short-howto

Docs: Clarify what the "index" is on the HTTP API

Right now it is just described like this: The "?index=" parameter is what was returned in a "X-Consul-Index" header. But why is it there? (I think I know why, but for the docs, we should specify)

Docs: How do I register services?

You mention service definitions are this JSON on the "service definition" page, but make no mention about how to register them. If I read the docs exhaustively, of course I'd learn, but for the newcomer (me), it would be nice to include a sentence or two and a link that explains how to register.

Need a `-join` flag

Uses un-allocated .consul top-level domain

The tool uses the “.consul” top-level domain, which will likely be problematic when that is created as top-level domain in the Domain Name System. This should be replaced with either a reserved or allocated name space for the purpose.

New Guide for putting consul behind bind

I'm going to write up a Guide for how to set up Bind and Consul to cooperate so that consul doesn't have to be run as root and bind to port 53

-encrypt flag: gone from the code but listed in the docs

The -encrypt flag was removed from command/agent/command.go in 9220145 but it is still mentioned in the docs, e.g. website/source/docs/agent/encryption.html.markdown .

Warning if GOMAXPROCS == 1

You said it yourself that Consul can block the scheduler thread so it really needs GOMAXPROCS > 1. If GOMAXPROCS is 1 at runtime, Consul should probably issue a warning.

Add a separate `Output` field for Checks

Currently we abuse the "Notes" field for human readable descriptions of TTL checks, but also use it to capture the output of script runs. We should have Notes which is always a description and Output which captures stdout/stderr.