haproxytech/haproxy-consul-connect

HaProxy Connector for Consul Connect. Enables Service Mesh with Consul and HaProxy using TLS and Consul Discovery

HAProxy Connect

Consul Connect provides a simple way to setup service mesh between your services by offloading the load balancing logic to a sidecar process running alongside your application. It exposes a local port per service and takes care of forwarding the traffic to alives instances of the services your application wants to target. Additionnaly, the traffic is automatically encrypted using TLS, and can be restricted by using intentions by selecting what services can or cannot call your application. HAProxy is a proven load balancer widely used in the industry for its high performance and reliability. HAProxy Connect allows to use HAProxy as a load balancer for Consul Connect.

Architecture

Three components are used :

• HAProxy, the load balancer
• Dataplane API, which provides a high level configuration interface for HAProxy
• HAProxy Connect, that configures HAProxy through the Dataplane API with information pulled from Consul.

To handle intentions, HAProxy Connect, sets up a SPOE filter on the application public frontend. On each connection HAProxy checks with HAProxy Connect that the incomming connection is authorized. HAProxy Connect parses the request certificates and in turn calls the Consul agent to know wether it should tell HAProxy to allow or deny the connection.

Requirements

• HAProxy >= v1.9 (http://www.haproxy.org/)
• DataplaneAPI >= v1.2 (https://www.haproxy.com/documentation/hapee/1-9r1/configuration/dataplaneapi/)

How to use

./haproxy-consul-connect --help
Usage of ./haproxy-consul-connect:
  -dataplane string
    	Dataplane binary path (default "dataplane-api")
  -enable-intentions
    	Enable Connect intentions
  -haproxy string
    	Haproxy binary path (default "haproxy")
  -haproxy-cfg-base-path string
    	Haproxy binary path (default "/tmp")
  -http-addr string
    	Consul agent address (default "127.0.0.1:8500")
  -log-level string
    	Log level (default "INFO")
  -sidecar-for string
    	The consul service id to proxy
  -sidecar-for-tag string
    	The consul service id to proxy
  -stats-addr string
    	Listen addr for stats server
  -stats-service-register
    	Register a consul service for connect stats
  -token string
    	Consul ACL token./haproxy-consul-connect --help

Minimal working example

You will need 2 SEPARATE servers within the same network, one for the server and another for the client. On both you need all 3 binaries - consul, dataplaneapi and haproxy-consul-connect.

The services

Server

Create this config file for consul:

{
  "service": {
    "name": "server",
    "port": 8181,
    "connect": { "sidecar_service": {} }
  }
}

Run consul:

consul agent -dev -config-file client.cfg

Run the test server:

python -m SimpleHTTPServer 8181

Run haproxy-connect (assuming that haproxy and dataplaneapi are $PATH):

haproxy-consul-connect -sidecar-for server

Client

Create this config file for consul:

{
  "service": {
    "name": "client",
    "port": 8080,
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [
            {
              "destination_name": "server",
              "local_bind_port": 9191
            }
          ]
        }
      }
    }
  }
}

Run consul:

consul agent -dev -config-file server.cfg

Run haproxy-connect (assuming that haproxy and dataplaneapi in $PATH) :

haproxy-consul-connect -sidecar-for client -log-level debug

Testing

On the server:

curl -v 127.0.0.1:9191/

Contributing

For commit messages and general style please follow the haproxy project's CONTRIBUTING guide and use that where applicable.

haproxy-consul-connect's Issues

Feature: Support Consul Service Router

Introduced in Consul 1.6.

When Connect is enabled, the Consul UI has a tab that allows the user to create dynamic routes.

Could this be implemented in HAProxy-Consul-Connect?

https://www.consul.io/docs/connect/config-entries/service-router

Consul's Layer 7 routing feature: https://www.consul.io/docs/connect/l7-traffic

Better packaging: removen the need for 3 binaries

The program is hard to deploy because it requires 2 additional binaries, ideally, it should:

• included dataplaneapi (either binary, either use the libraries from dataplaneapi)
• include haproxy binary bundled in binary (extract it at startup and run it)

Having to deploy 3 binaries is too boring

Modify GNUmakefile / travis configuration to run all integration tests

For now, the tests run at build time are limited to those not using dataplaneapi binary nor haproxy.

We need to test fully, so we need to modify Makefile to fetch HAProxy/dataplaneapi versions and to add it to the path to fully run tests.

@dcorbett-haproxy @aiharos How do you want to proceed?

Intentions SPOA gives expected capabilities error

Currently, the SPOA that checks Intentions is giving me an error when a request is received:

echo_1           | time="2019-09-18T20:02:40Z" level=info msg="spoe: hello from @: map[capabilities:pipelining engine-id:54DB81E3-F4EA-4899-BE8D-5F6F1EC62A1A max-frame-size:16380 supported-versions:2.0]"
echo_1           | time="2019-09-18T20:02:40Z" level=error msg="spoe: error handling connection: hello: expected capabilities [async pipelining]"
echo_1           | 127.0.0.1:52874 | GET /
echo_1           | time="2019-09-18T20:02:40Z" level=info msg="spoe: hello from @: map[capabilities:pipelining engine-id:54DB81E3-F4EA-4899-BE8D-5F6F1EC62A1A max-frame-size:16380 supported-versions:2.0]"

Build steps

Can the build steps be documented pls?

I downloaded the zip from maser and i got this:
go test -timeout 30s ./...
go: github.com/go-openapi/[email protected] requires
github.com/globalsign/[email protected]: invalid version: git fetch --unshallow -f https://github.com/globalsign/mgo in /root/go/pkg/mod/cache/vcs/0eb69f2163bb2fb915dbc4b9c83a68ab02240cc5c1aaa829e77cb6f9fd51e98d: exit status 128:
fatal: git fetch-pack: expected shallow list
make: *** [test] Error 1

Am sure i'm doing something wrong. I'm just running "make"

go version go1.13.6 linux/amd64

Consider sharing structs from Consul's API package

In https://github.com/haproxytech/haproxy-consul-connect/blob/556c75e4dede0754103b0b39b4d3f523be90cc13/consul/config.go

I see some struct duplication from the ones in our api package. Is there a good reason to need that vs just embedding our api package structs (say for upstreams or TLS configs) in your own wrapper that can add any additional context you want to pass through?

It might be reasonable as it is, I'm just thinking that as we add features especially with discovery chain stuff and some of our L7 config models it seems like wasted maintenance effort to keep up with copying more and more fields over to duplicate structs rather than just embedding ours.

Concretely, the biggest concerns there are the Config maps in Service.Proxy and Upstream and also the other fields in Upstream like MeshGateway. We'll be adding lots more config flags for things like timeouts, retries, circuit breaking etc. in to these places and as it stands now you'll have to explicitly add code to plumb them every time in the /consul package rather than just passing through and letting the /haproxy package just map them to what it needs.

Intentions not working

Build from latest master.
When i run with -enable-intentions, this is what i see in the logs and connections are not going through. Works without using that flag but intentions are not honored.

ERRO[0018] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=8eae2a31-e9c3-4d14-97c0-6a255c51c798: response was 422: "{"code":602,"message":"index in body is required"}"
INFO[0021] handling new configuration
ERRO[0021] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=95c05b05-1380-47fb-9ca4-5ea7e7707e24: response was 422: "{"code":602,"message":"index in body is required"}"
INFO[0024] handling new configuration
ERRO[0024] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=2d8d70f8-568c-409c-a555-f0c422bb5e5b: response was 422: "{"code":602,"message":"index in body is required"}"
INFO[0027] handling new configuration
ERRO[0027] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=339c6bb3-14c2-487a-b092-75e234741fa6: response was 422: "{"code":602,"message":"index in body is required"}"

Certificate verification doesn't support intermediate CA chains

haproxy-consul-connect/haproxy/spoe.go

Lines 45 to 47 in 556c75e

	_, err = cert.Verify(x509.VerifyOptions{
	Roots: cfg.CAsPool,
	})

Verifying certificates is not providing intermediates that may be present. That means it will only work in the Primary DC and even then it will only work for CA providers that choose to sign directly with their root key. (In the future this will likely be zero providers!).

The easiest thing to do is to follow the code in our SDK here: https://github.com/hashicorp/consul/blob/fd3c56ff68829821da4be139185c0a96938e1929/connect/tls.go#L265

At a high level it:

• Parses the certificate given by the client as a bundle - the first certificate present is always the leaf, any other certificates presented are assumed to be intermediates, either from the current CA or cross-signed by previous CAs during a rotation.
• Add the intermediates, if any to an intermediates x509.Pool
• Call verify with the roots and intermediates etc.

SPOE filter for Intentions uses 'tcp-request content accept'

With Intentions enabled, the HAProxy configuration that's auto-generated contains this in the frontend for the local service:

frontend front_downstream 
  mode http
  bind 0.0.0.0:21000 name front_downstream_bind crt /tmp/haproxy-connect-002132058/e727ca4e64d85da74a3846094069b9bd80f41a92f36c5a16fd2417a1b2443f42 ca-file /tmp/haproxy-connect-002132058/6552a0a59b20ca0d1997c4e5e9afb8a5c3e31dc00aada13727733f742d79d4d5 ssl verify required
  timeout client 30000
  filter spoe engine intentions config /tmp/haproxy-connect-002132058/spoe.conf
  tcp-request content accept if { var(sess.connect.auth) -m int eq 1 }
  default_backend back_downstream

The tcp-request content accept line will accept the connection if there is no "Deny" intention in Consul forbidding it. However, there is no rule that rejects the connection otherwise. So, all requests are accepted, no matter whether the are denied in Consul.

Workaround:

Editing the configuration so that tcp-request content accept if is replaced with tcp-request content reject unless fixes it:

tcp-request content reject unless { var(sess.connect.auth) -m int eq 1 }

Migrate to new Consul API

While trying to add tests for Consul, I discovered that the client's binary API has been modified.
Apply the fixes done in hashicorp/consul#7637 to ensure it works with recent versions of Consul APIs

Sync Unit Tests between Consul & HAProxy-Consul-Connect

Added initial tests on Consul side: hashicorp/consul#7637

We need to find a way too sync both

Features that we would like to have

Hi!

As you have noticed I have started to contribute to this project. I am doing that because I am POCing a service mesh solution for Egnyte that is using this project.

To make it work, apart from #51 and #59, I would also need a few more features, so I would like to ask how do they align with your plans - perhaps you are already working on some of them or on the contrary: you will not even accept PRs with them. Let's have a chat about it. :)

So here it goes, in no specific order:

• Customizable HAproxy logs format
• Add custom config fragment to the upstream services frontends - mostly to capture some request headers for enriching the logs
• Add custom config fragment to the upstream services backends - for various things, like rspirep directives for example,
• Add a completely custom additional frontend(s) - where we would some custom logic above, f.e. splitting the traffic based on the paths between the upstream backends
• Listening on the UNIX sockets - as an alternative to above approach we could put the custom loginc into a separate HAproxy instance in front of the sidecar. To minimize the performance penalty we would like it to connect to the sidecar over these sockets.
• Custom load balancing algorithms? - maybe it doesn't make sense but we are looking for a way to make the load balancing as fair (even) as possible with the distributed load balancers..

So what do you think?

Layer 7 traffic management?

Hi!

Is HAProxy as a sidecar compatible with L7 traffic management features relatively recently added to Consul?

If not but it is possible, then do you plan to implement it?

Thanks!

Consider removing `-enable-intentions` flag

We very purposefully don't have a mode in Connect where you can choose to not enforce intentions. I'd strongly recommend not including one in this integration either. It's understandable if you needed a flag for initial testing or seeing what the overhead may be etc. but I would really prefer to keep away from adding more knobs that affect the security of any integration.

It's always possible to use permissive intentions that allow everything after all (that's even the default if you haven't setup restrictive ACLs). If you feel strongly that this is something you need to keep long term for testing the overhead of intention enforcement, could we at least make it on-by-default and Opt out (i.e. DisableIntentionChecking) rather than opt -in?

Also, from what I can see the Certificate validation is being done in your SPOE handler and so presumably not in HAProxy at all which means that disabling SPOE is more than just disabling intention checks - it's disabling cert validation too. In other words it's a free for all, you don't even need a valid client cert, any old self-signed thing will do. That implies it should be named something more general like -disable-security since it's not only intention enforcement but all of the mutual TLS benefits that go away.

If an option is kept for this I suggest that it needs very clear docs and sufficiently strong warnings about the precise way it weakens the threat model.

Avoid relying on non-guaranteed proxy naming conventions

haproxy-consul-connect/main.go

Lines 57 to 59 in 55bff42

	if strings.HasSuffix(s.Service, "sidecar-proxy") {
	continue
	}

This is a naming convention only and there is nothing in Consul stopping users from naming proxies differently if they don't use the sidecar_service helper.

The canonical way to tell if something is a Proxy in Connect's data model is if service.Kind == "connect-proxy". If it's important that it's specifically a sidecar proxy (not say an ingress loadbalancer) which is true in this case you need to also check for service.Proxy.DestinationServiceID != "" - The thing that canonically makes a Proxy service a sidecar is that field indicating that it is has exactly one app (destination) instance behind it.

So to correctly skip sidecar proxies in this loop the following should work:

  // Skip all sidecar proxy registrations
  if s.Service.Kind == api.ServiceKindConnectProxy && s.Service.Proxy.DestinationServiceID != "" {
      continue
    }

NOTE: In fact this whole loop might need to change making this issue moot. I'll explain more in the issue that I'll link below, but this was important to note to make sure the guarantees are understood here.

Logs

How do i get haproxy stats and logs? Do we need to use consul-template for those?

Documenting the threat model/improving inter-process security.

This is partly a suggestion about clarifying security model docs and partly a suggestion to improve the overall security of this integration if I'm understanding the current model correctly.

The diagram in the README is fantastic for showing how this integration works. It's not currently clearly stating the threat model.

As far as I can see, the haproxy-connect process is the one that is authenticated by an ACL to fetch certificates. Which way does the dataplane API connection work though? And from what I can see it talks to HAProxy dataplane as a client which means it doesn't expose an open interface that another process on the box can connect to and get access to certificates/keys? If I'm right that sounds good.

One potential issue I see though is that it uses a hard-coded dataplane user/pass for HAProxy:

haproxy-consul-connect/haproxy/config.go

Lines 16 to 17 in 556c75e

	dataplaneUser = "haproxy"
	dataplanePass = "pass"

This means that any other process on the box that can talk on that same unix socket and happens to know these credentials (it is open source after all) can completely reconfigure the proxy, for example to disable SPOE intention checking.

One possible simple solution would be to generate a strong crypto-random password for HAProxy on every startup so it's only in-memory in the consul-haproxy process and passed through to HAProxy. I guess the Config is written to disk so I'd suggest using the secure (hashed) password config to pass it through and hashing the random password in consul-haproxy. That way you can have high confidence that the only process able to configure the proxy is same consul-haproxy process that stated it.

Add golangci-lint to CI pipeline

We should add some linting to the pipeline to help us catch issues with the code.

golangci-lint seems like a good choice, but I'm interested to hear if there are any other preferences before implementing this.

Watcher data race

https://github.com/haproxytech/haproxy-consul-connect/blob/master/consul/watcher.go

I see a couple of issues in the watcher:

• removeUpstream is setting a done bool under lock but the startUpstream loop reads that bool while not holding the lock which is a data race.
• there is no way to close/stop watcher and free goroutines. This is not a big deal in normal usage but our lesson from Consul's TestAgent is that this pattern leads to really inefficient test runs with tons of orphan goroutines by the end etc. I'd recommend plumbing a Stop mechanism or Context through every goroutine spawned right from the start so all resources can be cleaned up in tests and in case you ever need to support managing multiple proxies or decoupling process lifetime and proxy lifetime etc. It's way easier if it's just there already!

Working example

Is there a working example or a link that I can check? Need an example with sidecar proxy and HAProxy-consul-connect working together. Thanks for the help!

changes to dataplane-api causing issues

I setup a test service in consul and tried starting this with the following command:
./haproxy-consul-connect -sidecar-for test_v1

HAProxy binary and dataplaneapi are in the path so it finds them. However, haproxy-consul-connect exits with this:

INFO[0000] consul: watching service test_v1
INFO[0000] consul: watching service test_v1-sidecar-proxy
INFO[0000] consul: leaf cert for service test changed, serial: 98:04, valid before: 2020-02-17 04:12:05 +0000 UTC, valid after: 2020-02-14 04:12:05 +0000 UTC
INFO[0000] consul: leaf cert for test ready
INFO[0000] consul: CA certs changed, active root id: 98:c3:ff:ca:a7:1a:f2:a5:ed:0b:04:3a:80:2c:62:a8:ec:35:3c:64
INFO[0000] consul: CA certs ready
INFO[0000] received consul config update
INFO[0000] handling new configuration
INFO[0000] haproxy: 045/195724 (15416) : New worker #1 (15417) forked
ERRO[0005] timeout waiting for dataplaneapi: error calling GET /v1/specification: response was 404: "{"code":404,"message":"path /v1/specification was not found"}"
INFO[0005] Shutting down because timeout waiting for dataplaneapi: error calling GET /v1/specification: response was 404: "{"code":404,"message":"path /v1/specification was not found"}"...
INFO[0005] cleaning config...
INFO[0005] killing dataplane-api with sig 15
INFO[0005] killing haproxy with sig 15
WARN[0005] haproxy: 045/195729 (15416) : Exiting Master process...
ERRO[0005] haproxy exited
ERRO[0005] dataplane-api exited

Noticed dataplaneapi now has /v2 instead of /v1. Also, it's now called dataplaneapi, not dataplane-api. Ofcourse, renamed dataplaneapi to dataplane-api go get this working.

Can anyone pls look into this?

Thanks!

Tag Lookup

Note if this is changed it might make #7 invalid.

haproxy-consul-connect/main.go

Lines 61 to 68 in 55bff42

	if t == *serviceTag {
	serviceID = s.ID
	break OUTER
	}
	}
	}
	if serviceID == "" {
	log.Fatalf("No sidecar proxy found for service with tag %s", *serviceTag)

Having skipped sidecars (assuming that works, see #7), this code attempts to find the first tag match on an actual service instance. But if it fails it returns an error about not finding the sidecar proxy which is confusing since it might have been there you just skipped it.

I think I understand the intent and it's rely on the existing sidecarFor lookup code copied from Consul (great - was going to recommend that part), but as it is there are a lot of confusing ways this can break: 

• if there is a proxy registered but no service instance (which is technically valid although not that useful, but in practice can happen transiently due to timing of processes registering). 
• if there are multiple services registered locally with that tag, it will arbitrarily pick the first one which isn't great UX. Better would be to detect the ambiguity and error in the same way LookupProxyIDForSidecar in the Consul command code does.

I suggest it might be easier if all the lookup stuff was moved out of the /consul package and into the main.go so that you can handle all this resolving stuff in one place and just start the main body of the work with a proxy's ID already resolved or all argument errors already handled with messages that are clear to the user where there is ambiguity.