hashicorp/consul-aws

Consul-AWS

consul-aws syncs the services in an AWS CloudMap namespace to a Consul datacenter. Consul services will be created in AWS CloudMap and the other way around. This enables native service discovery across Consul and AWS CloudMap.

This project is versioned separately from Consul. Supported Consul versions for each feature will be noted below. By versioning this project separately, we can iterate on AWS integrations more quickly and release new versions without forcing Consul users to do a full Consul upgrade.

Installation

1. Download a pre-compiled, released version from the Consul-AWS releases page.

2. Extract the binary using unzip or tar.

3. Move the binary into $PATH.

To compile from source, please see the instructions in the contributing section.

Usage

consul-aws can sync from Consul to AWS CloudMap (-to-aws), from AWS CloudMap to Consul (-to-consul) and both at the same time. No matter which direction is being used consul-aws needs to be connected to Consul and AWS CloudMap.

In order to help with connecting to a Consul cluster, consul-aws provides all the flags you might need including the possibility to set an ACL token. consul-aws loads your AWS configuration from .aws, from the instance profile and ENV variables - it supports everything provided by the AWS golang sdk. A default AWS region is not assumed. You can specify this with the standard AWS environment variables or as part of your static credentials.

Apart from that a AWS CloudMap namespace id has to be provided. This is how consul-aws could be invoked to sync both directions:

$ ./consul-aws sync-catalog -aws-namespace-id ns-hjrgt3bapp7phzff -to-aws -to-consul

Contributing

To build and install consul-aws locally, Go version 1.21+ is required. You will also need to install the Docker engine:

• Docker for Mac
• Docker for Windows
• Docker for Linux

Clone the repository:

$ git clone https://github.com/hashicorp/consul-aws.git

To compile the consul-aws binary for your local machine:

$ make dev

This will compile the consul-aws binary into dist/$OS/$ARCH/consul-aws as well as your $GOPATH.

To create a docker image with your local changes:

$ make dev-docker

Testing

If you just want to run the tests:

$ make test

Or to run a specific test in the suite:

go test ./... -run SomeTestFunction_name

Note: To run the sync integration tests, you must specify INTTEST=1 in your environment and AWS credentials. You must also have a Consul server running locally.

Compatibility with Consul

consul-aws is compatible with supported versions of Consul. See long-term support docs for more information.

consul-aws's Issues

Decide what to do with AWS Healthchecks

Right now we only fetch healthy instances and there is no point in getting health check data from aws.

[req] Publish docker images

This is a request to publish the tool to docker hub. I noticed in build-support there is already a Release.dockerfile, I was hoping there would be a corresponding image on docker hub.

I was able to build and run the Release dockerfile locally without trouble.

The only issue I had was because I was running in ECS/Fargate and I was running a sidecar consul container. In that environment one is unable to control task start-up ordering. It'd be helpful if the fetchIndefinitely method would wait longer than a couple of seconds. I had to wrap my docker command into a shell script to make it work reliably:

sh -c "until nc -z localhost:8500; do sleep 1; echo Waiting for consul to come up...; done; exec consul-aws sync-catalog -to-consul -to-aws -aws-namespace-id $namespace_id"

Multi AWS Account guidance

I'm sure we're not the only one running multiple AWS Accounts but a single Consul cluster across all accounts, with service names indicating aws account (dev, test, etc).

Since cloud map seems to be a per-account service, it would be good to have some guidance on how to set up consul-aws against multiple accounts.

Add Uniqueness to Tests for AWS Cloud Map Namespaces

While testing the previous and current Consul release in parallel in CI, I ran into the following issue https://circleci.com/gh/hashicorp/consul-aws/36.

[feature request] sync only services with specified tag (consul > cloudmap)

I want to register all services form cloudmap to consul catalog and some (with specified tag) from consul catalog to cloud map. Currently it is not possible. I can only sync all services between cloud map and consul catlaog.

Add `-region=<region_name>` argument.

We're currently forced to use the AWS_REGION environment variable to set/override AWS region where pretty much everything else is available via CLI argument, so just adding this for the sake of consistency and user experience.

"panic: assignment to entry in nil map" on startup

I'm running consul-aws on the same host that one of my consul masters is running on, and as soon as I execute the following command the application crashes:

$ consul-aws sync-catalog -aws-namespace-id <namespace-id> -to-aws -consul-service-prefix consul_ -http-addr http://<consul-ip-address>:8500
panic: assignment to entry in nil map

goroutine 38 [running]:
github.com/hashicorp/consul-aws/catalog.(*aws).create.func1(0xc0003317a0, 0xc0001f4000, 0xc000355ce0, 0x14, 0xc0003557a0, 0x1c, 0xc00036c6a0, 0xb, 0x7bca, 0xc00036c6a0, ...)
	/go-build/catalog/aws.go:366 +0x133
created by github.com/hashicorp/consul-aws/catalog.(*aws).create
	/go-build/catalog/aws.go:362 +0x3b1

This also happens when I run the command on a consul agent and point it to the local-ip address that the agent is advertised on.
Not sure what extra information I can give...we have no auth on our consul cluster and the EC2 instance has an IAM role with full permissions on Cloud Map.

$ consul-aws version
consul-aws v0.1.1

No documentation available for consul-aws

Hey there!

I am in the process of updating the Learn guide for this tool: https://learn.hashicorp.com/tutorials/consul/sync-aws-services

I wanted to link to this tool's documentation, but looking at consul.io and this repository, I can't seem to locate any documentation. The Learn guide provides sparse configuration information for the available subcommands and parameters, but they aren't contextually relevant for the tutorial, and I'd like to remove them in favor of linking to structured docs.

If I've overlooked something, please let me know! 😺

Thanks!

cc @trujillo-adam for 👀

Fix hash for rootcerts

The hash for the rootcerts is different for the latest golang version 1.11.4 and we want to fix go.sum. see prometheus/prometheus#5007.

A service without a defined port generates errors

v0.1.1

AWS Cloud Map doesn't seem happy about creating services that have no ports.

Given a minimal service definition of:

service {
    name = "foo"
}

The consul-aws output is spammed with:

2019-02-05T20:19:29.085Z [ERROR] aws: cannot create nodes: error="InvalidInput: A valid port number must be supplied.

Setting a port of 0 has no effect.

[req] Purge synced services on exit

Currently, consul-aws publishes the consul services and nodes. But does not clean-up after itself on exit. When we stopped the services and removed the container daemon we found all services were still listed.

I found this out when I decided to remove the discovery zone (via Terraform). AWS Cloud Map will not let one delete the zone until all services are deleted, and one cannot remove a service until all instances are deleted. There was much ... clicking that day.

Thank you

ARM binaries misnamed to `project`

When migrating over the build and release process, the ARM binaries were misnamed as project rather than the variable for the actual consul project that is being built. This behavior is seen when downloading the zip from releases.hashicorp.com and if you unzip the binary with unzip consul-aws_<version>_linux_arm64.zip you will end up with a binary called project. The fix is merged in hashicorp/consul-eco-releases#10 and will be named properly moving forward.

The misnaming affected only ARM binaries (consul-aws_<version>_linux_arm64.zip, consul-aws_<version>_linux_armelv5.zip, consul-aws_<version>_linux_armhfv6.zip) of version:

• 0.1.2

Workaround:
Instead of just running unzip consul-aws_<version>_linux_arm64.zip where you'd end up with a binary called project, you can run unzip consul-aws_<version>_linux_arm64.zip && mv project consul-aws to properly rename the binary to what is expected.

Still maintained ?

Hi !
Just to know if the repo is still maintained and if there is an alternative to sync consul with cloudmap ?

Flaky Sync test

Sometimes, the current sync test fails due to the check of whether the AWS service is deleted. https://circleci.com/gh/hashicorp/consul-aws/59

test issue

test

Only SRV records ?

Hi,
Since the upgrade to 0.1.2, when I sync-catalog, my SRV records are created but I don't have any A record in route 53.

dig consul.sandbox

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> consul.sandbox
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4430
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul.sandbox.                        IN      A

;; AUTHORITY SECTION:
sandbox.                579     IN      SOA     ns-1536.awsdns-00.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 2 msec
;; SERVER: 10.232.0.20#53(10.232.0.20)
;; WHEN: Wed Jul 29 17:50:16 UTC 2020
;; MSG SIZE  rcvd: 130

dig srv consul.sandbox

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> srv consul.sandbox
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21020
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul.sandbox.                        IN      SRV

;; ANSWER SECTION:
consul.sandbox.         1       IN      SRV     1 1 8300 srv-kqxpnfnipvmxzb27_10.232.34.232_8300.consul.sandbox.
consul.sandbox.         1       IN      SRV     1 1 8300 srv-kqxpnfnipvmxzb27_10.232.27.19_8300.consul.sandbox.
consul.sandbox.         1       IN      SRV     1 1 8300 srv-kqxpnfnipvmxzb27_10.232.18.127_8300.consul.sandbox.

;; Query time: 2 msec
;; SERVER: 10.232.0.20#53(10.232.0.20)
;; WHEN: Wed Jul 29 17:51:50 UTC 2020
;; MSG SIZE  rcvd: 264

curl consul.sandbox
curl: (6) Could not resolve host: consul.sandbox; Unknown error

Am I missing something ?

Support syncing a specific Consul Service.

I'd like the ability to sync specific Services between CloudMap<>Consul.

For example, the ability to sync Services under https://consul.somewhere.com:8500/ui/production/services/some_service into a some_service CloudMap Namespace.

What consul versions does this tool support?

The readme states This project is versioned separately from Consul. Supported Consul versions for each feature will be noted below. but I am unable to find a list of supported consul versions.

Is this documented anywhere?