This repo can be used to show Consul service discovery, Consul Connect, intentions and service failover between datacenters (via prepared query)
- Single Region - located in
terraform/single-region-demo
- demonstrates: service discovery, Consul Connect, and intentions
- Multi Region - located in
terraform/multi-region-demo
- demonstrates: service discovery, Consul Connect, intentions and service failover between datacenters using a prepared query
- Simple Three Tier Architecture
- Front tier:
web_client
service - Middle tier: two services
listing
andproduct
- Back tier:
MongoDB
service
- Front tier:
- Resources Deployed in each Region
- Consul Configuration Single DC
- Front Tier to the Middle Tier communicates using Consul Service Mesh
- Middle Tier to Back Tier Communicates via Service Discovery only
- Demo begins with Service Discovery by showing connections between Middle & Back Tier
- Note: Intentions have no effect on the Back Tier, it's not using Consul Service Mesh
- Multi Region Configuration
A PowerPoint version and larger versions of these diagrams is included in the
./diagrams
directory
- AWS account & credentials
- AWS Route53 Hosted Zone ID
- Terraform creates FQDNs for instances & load balancers
- Terraform version support
- Terraform CLI 0.11.x - use
master
branch - TFC/TFE - use
master
ordemo
branches- set Terraform version on workspaces to 0.11.14
- Terraform CLI > 0.12.4 - use
terraform-0.12
branch
- Terraform CLI 0.11.x - use
- AWS AMIs are publicly available in
us-east-1
,us-east-2
,us-west-1
andus-west-2
- To customize AMIs view the Packer README
- switch to the directory for the desired demo variant
- for single region:
cd terraform/single-region-demo
- for multi-region:
cd terraform/multi-region-demo
- for single region:
- Make a copy of the example variables file
cp terraform.auto.tfvars.example terraform.auto.tfvars
- Edit
terraform.auto.tfvars
and set values as described in comments
- Fork this repo
- Make a copy of the script reference/setup-tfe-example.sh and name it
setup-tfe.sh
- ensure TFH is installed URL specified in script (includes fix not yet incorporated in official release)
- populate Variables in script
- configure TFC/TFE organization
- specify your repo and branch
- specify workspaces names for
single-region-demo
andmulti-region-demo
- script sets working directory for each workspace
- configure other variables - see
terraform.auto.tfvars.example
for descriptions
- Run
setup-tfe.sh
to create TFC/TFE workspaces with VCS connections and populated variables
The multi-region terraform code uses a post provisioner which requires specifying the AWS ssh keyname & the ssh private key itself (via file reference or as a string). This process is simplified if you push your system's default ssh key (~/.ssh/id_rsa.pub) to the AWS regions used with this demo.
- (OPTIONAL) use script reference/push-ssh-key-to-aws.sh to push local SSH key to every AWS region using AWS CLI
- edit script and set value of
aws_keypair_name
andpublickeyfile
- edit script and set value of
- Set Terraform variables (in
terraform.auto.tfvars
orsetup-tfe.sh
)ssh_key_name
- must exist in both AWS regions (default: us-west-2 & us-east-1)- Specify either
ssh_pri_key_data
orssh_pri_key_file
that refers to private SSH key referenced byssh_key_name
ssh_pri_key_file
- file path to the private key (does not work with TFC/TFE)ssh_pri_key_data
- contents of private key as data with newlines replaced with\n
(required for TFC/TFE)- remove newlines with command:
awk '{printf "%s\\n", $0}' ~/.ssh/id_rsa
- remove newlines with command:
- Deploy with Terraform (takes 3-4 minutes)
- (OPTIONAL) add aliases from Terraform output
working_aliases
to .bash_profile- eliminates need to remove keys from known_hosts before each demo
- (OPTIONAL) bookmark web URLs specified in Terraform output
working connections
- Edit
~/.ssh/known_hosts
and remove entries from previous demos (unless using ssh aliases) - Follow instruction in Terraform output
working connections
- Open two URL's in Browser
- web-page rendered by
web_client
service - Consul UI
- web-page rendered by
- Open two URL's in Browser
- Verify the all services are running in Consul UI
- Show High Level Architecture diagram
Explain the three tiers and communications between the services. We are going to start by discussing Service Discovery and Service Registration
In order for services to be discoverable via Consul, they first must register themselves.
- open new terminal tab & connect to
MongoDB
instance- use connection string in terraform output
working connections
- use connection string in terraform output
This instance's Consul client publishes all the services running on it to Consul. In this demo, MongoDB is the only service on this host, so let's review it's config.
- run
./1-cat-cons-config.sh
to displayMongoDB
service configuration
Explain service config & health check basics. Consul uses this information to build a Service Catalog. Consul monitors health checks to insure only healthy services listed in catalog.
- (optional) - show service tab on Consul UI, select MondoDB & show service checks
- you can
exit
the ssh connection toMongoDB
instance
Once services are registered, other services can easily discover them. Let's show how the listing service connects to MongoDB.
- open new terminal tab & connect to
listing
instance- use connection string in terraform output
working connections
- use connection string in terraform output
The listing service is written in NodeJS and isn't Consul aware. It reads the environment variables DB_URL to get MongoDB servers address.
- run
./1-cat-system.sh
to show listing service systemd configuration
It's systemd config set DB_URL Environment Var to mongodb.service.consul.
- run
./2-dig.sh
to show a dns query using dig for all mongodb services
Consul automatically resolves service names via DNS to an IP where mongodb is running. Dig queries Consul via DNS and returns healthy mongodb service.
- (optional) ping the mongodb service:
ping mongodb.service.consul
- (optional) list services registered with Consul:
consul catalog services
Service discovery allows service to easily discover and connect to other services. But Network traffic is unencrypted unless services coded with encryption.
- run
./3-nw-traffic.sh
to show network traffic betweenlisting
andmongodb
- Traffic will stream immediately as the connection to mongodb is persistent
- Watch the traffic and press
ctrl-c
after you see the database records displayed
Point out that the data including db creds is traversing network in plaintext.
- Hit Cntl-C to exit network traffic dump
- you can
exit
the ssh connection tolisting
instance
Next we're going to talk about Consul Service Mesh and some of the enhancements it brings.
- Show Consul Configuration Single DC diagram
You'll notice that the Front Tier and Middle tier have sidecar-proxies. They are configured to communicate via Consul Service Mesh. The Back Tier is configured to only use Service Discovery.
- open new terminal tab & connect to
webclient
instance- use connection string in terraform output
working connections
- use connection string in terraform output
In the diagram, webclient communicates with listing & product services. Let's show how the webclient service is configured.
- run
./1-cat-system.sh
to show webclient service systemd configuration
Systemd config sets Environment Vars for listing & product to localhost @ unique ports. So how are these local ports re-directed to the proper services? via the proxy.
- run
./2-cat-cons-config.sh
to displaywebclient
service configuration
Describe the connect-sidecar service block. Upstream connections bind a local port to services in Consul. Consul proxies localhost:10001 to
product
services AND encrypts the traffic. Consul proxies localhost:10002 tolisting
services AND encrypts the traffic. (Note: product uses "prepared query" to enable failover to other datacenters.)
- run
./3-dig.sh
to show a dns query using dig for all product services - (optional) list listing services using dig:
dig +short listing.connect.consul srv
Like earlier, this host resolves service names to an IP where product is running. Notice, here we search on servicename.CONNECT.consul instead of .SERVICE.consul. This limits our results to services using Consul Connect.
- (optional) query mongoDB service using connect
dig +short mongodb.connect.consul srv
Nothing returned as the MongoDB services are not configured to use Consul Connect (refer to diagram if necessary).
The sidecar proxies use TLS to encrypt communications.
- run
./4-nw-traffic.sh
to show network traffic betweenwebclient
andproduct
- wait or refresh the web_client web-page to see network traffic
As we can see, the network traffic is TLS encrypted gibberish.
- Hit Cntl-C to exit network traffic dump
- you can
exit
the ssh connection towebclient
instance
Intention enable defining specific services that each service can communicate.
- Show Web Client web page, and point out its communicating with Listing & Product
Now we'll stop all services (that are configured for Service Mesh) from communicating.
- Open Consul UI and select Intentions tab
- Create an Intention from
*
to*
of typeDeny
and click save
- Create an Intention from
- Show Web Client web page and point out it cannot communicate with Listing or Product services
Lets allow web_client to communicate with the listing service.
- Switch back to Consul Intentions UI
- Create Intention from
web_client
tolisting
of typeAllow
and click save
- Create Intention from
- Show Web Client web page and point out it can now communicate with Listing
(OPTIONAL) Intentions specify which service can initiate communications.
- Switch back to Consul Intentions UI
- Create Intention from
product
toweb_client
of typeAllow
and click save
- Create Intention from
- Show Web Client web page and point out the it still cannot communicate with product
Note: web_client cannot talk to product because the intention added allows product to initiate connections to web_client, and not other way around. So, lets add a connection that allows web_client to initiate communications with product.
- Switch back to Consul Intentions UI
- Create Intention from
web_client
toproduct
of typeAllow
and click save
- Create Intention from
- Show Web Client web page and point out both working again
Describe "Scalability of Intentions": If you have 6
web_client
instances, 17listing
instances and 23product
instances; you'd have6 * 17 + 6 * 23 = 240
endpoint combinations to define. These can be replaced with just 2 intention definitions. Another example: If you double the number of backends, you have to add another 240 endpoint combinations. With Intentions, you do nothing because intentions follow the service.
Steps required to configure
web_client
to talk toproduct
via connect
- Configure services for Service Mesh
- Tell
web_client
thatproduct
service is reachable on localhost ports - Consul Service Mesh handles balancing traffic between 1, 2, 20, 100 healthy instances
- Consul Service Mesh encrypts all network traffic
web_client
knows nothing about TLSweb_client
knows nothing about mutual-TLS authenticationweb_client
doesn't have to manage certificates, keys, CRLs, CA certs...web_client
simply makes the same simple, unencrypted requests it always has
By configuring
product
to listen only onlocalhost
, you've reduced the security boundary to individual server instances --- all network traffic is encrypted. Only steps necessary to enable existing application to configure Consul Service Mesh and configure app to communicate to port on localhost.
product
knows nothing about TLSproduct
knows nothing about mutual-TLS authenticationproduct
doesn't have to manage certificates, keys, CRLs, CA certs...product
simply sees simple, unencrypted traffic coming to it
Describe how the web_client service has been displaying configuration information from Consul's K/V store.
- Show Web Client web page, point out Configuration Section
- the
product
service reads the Consul K/V store (along with the mongodb records) - data returns to
web_client
and displayed
- the
Webclient service is configured to use a "prepared query" to find the
product
service. If every product service in the current DC fails, it looks for the service in other DCs. Additional info on the prepared query can be found in the prepared_query reference
- Show Web Client web page
- point out Configuration Section under Product API
- shows datacenter = dc1
Now we're going to trigger the product service to fail in this datacenter.
- Open Consul UI and select Key/Value tab (make sure it's in dc1)
- Click on
product
folder and valuerun
- Change value from
true
tofalse
- Click on
- Switch to the Consul UI services and watch the
product
services fail their health checks- wait until there are two failures for
product
and its proxy
- wait until there are two failures for
Now that the service has failed, let view the webpage.
- Show Web Client web page & hit refresh
- point out Configuration Section under Product API
- now shows datacenter = dc2, as the instance responding is in DC2
- ssh into any one of the previous servers (in DC1)
- type
dig +short product.connect.consul srv
- no product services are available in this DC
- type
dig +short product.query.consul srv
- but by searching the prepared query (as the web_client is doing) shows services in the other datacenter
- disable the failover by setting the
product/run
key back totrue
- wait for the services to become active again in DC1
- type
dig +short product.query.consul srv
and show how he results have changed back to this DC - type
dig +short product.connect.consul srv
- shows the same results as the query, as all the services in this DC are working