capgemini/apollo

:rocket: An open-source platform for cloud native applications based on Apache Mesos and Docker.

apollo's Issues

Have mesos span multiple availability zones

We should be able to achieve something like the above - having 2 masters in one AZ and another in a second AZ (providing higher availability)

Provision docker registrator so containers spinning up in the cluster publish their information to consul

Needs to be on each slave

we might want to wait on this hashicorp/terraform#1329

Create a docs/ subfolder with a roadmap.md

Create high level roadmap where we can dump desired features into

Update dnsmasq spec test to test the config file provisioned

We need a test for ensuring this file exists with the correct content -

echo "server=/consul/127.0.0.1#8600" > /etc/dnsmasq.d/10-consul

Consul Web UI unavailable through VPN

A bug in the consul config. Cannot connect to the web UI.

Similar to this issue - hashicorp/consul#599
Probably just needs a config edit (around the bind address)

http://www.consul.io/docs/agent/options.html

Upgrade to mesos 0.22

Create a readme doc on how to setup VPN access for yourself through the nat sever

Mainly just a lift / shift from here https://www.airpair.com/aws/posts/ntiered-aws-docker-terraform-guide

with addition of installing tunnelblick

Set the consul datacenter config option by AWS zone we are in

At the moment datacenter is defaulting to "DC1"

It should be whatever AZ that particular node is in (for example "eu-west-1")

Create a packer image for the front-end load balancer to services in the cluster

Should include -

• haproxy
• docker
• weave
• consul
• consul-template

Discuss / brainstorm approach to Monitoring and alerting

We need to be able to monitor the entire stack. Some things we need to monitor -

• Node level monitoring (CPU/Mem/Disk etc...) - Some of this can be achieved through consul / atlas but we still need alerting
• Docker container monitoring
• Mesos framework monitoring (maybe http://www.antonlindstrom.com/2015/02/24/monitoring-mesos-tasks-with-prometheus.html), Mesos also has its stats endpoint.
• Alerts on monitors - alerting roll-up - types of alerts
• Service level monitoring (related to docker monitoring)

Experiment setting mesos up in a coreOS cluster

Could possibly borrow some stuff from here -
https://gist.github.com/andyshinn/92f9175a8cc79185314e

Not sure of the implications or extra hoops required to run mesos on coreOS. Seems some people have done some experimentations -

http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/
https://groups.google.com/forum/#!searchin/coreos-user/mesos/coreos-user/hsjuvrL8eM0/wgHVVEARxnYJ
https://github.com/veverjak/coreos-mesos-marathon

Add an architecture diagram to the docs

Could potentially use https://ide.visualops.io

Upgrade ubuntu to kernel 3.18 to use overlayfs

http://ubuntuhandbook.org/index.php/2014/12/install-linux-kernel-3-18-ubuntu/

Create centos packer box for aws

Setup nat server with two factor for external access

use these to help
http://timlegge.blogspot.co.uk/2009/11/running-multiple-ssd-daemons-in-ubuntu.html
http://www.linux.com/community/blogs/133-general-linux/783135-securing-ssh-with-two-factor-authentication-using-google-authenticator
http://www.tecmint.com/ssh-two-factor-authentication/

Create a mule demonstrator

This is basically mule running in a docker container that we can deploy in the cluster

Configure consul via terraform on master/slaves

Create a packer template for digitalocean

• Should be the same as aws but with digitalocean flavours

Deploy cadvisor to the cluster for monitoring docker containers

Needs to be deployed on each slave

we might want to wait on this hashicorp/terraform#1329

Fix routing to amazon private_dns instances via the VPN

We should be able to get to (in a browser) the mesos admin UI via the private_dns name through the vpn server.

We have this in the provisioning -

 "sudo iptables -t nat -A POSTROUTING -j MASQUERADE",

but it doesnt appear to be working correctly.

See also http://serverfault.com/questions/637612/trouble-hitting-ec2-amazonprovideddns-server-over-vpn

Create a chaos monkey container that kills off other containers

We should be able to demo chaos monkey. The demo would be -

start up a bunch of containers
start up the chaos monkey container
watch some containers die
watch mesos reprovision those containers automagically

Cannot access the internet from the private VPC

We do this on the NAT instance -

sudo iptables -t nat -A POSTROUTING -j MASQUERADE

which is supposed to allow traffic via iptables routing from the private instances to reach the internet via the NAT machine. However this does not work at the moment so we need to fix / find a workaround

Move to ansible for provisioning from terraform

At the moment we have a set of bash scripts (which works) but is slightly clunky / ugly.

We should be able to switch over to ansible fairly easily. This would allow us to -

• Have a playbook that is generic (with box specific stuff in it) - that could be run over any cloud
• Take advantage of ansible dynamic inventory for provisioning into cloud where we dont know the ip / details ahead of time.
• Remove some slightly messy bash code e.g. -

ssh "$node" "echo '{\"ui_dir\": \"/opt/consul-ui\", \"server\": true, \"bootstrap_expect\": ${nodes}, \"service\": {\"name\": \"consul\", \"tags\": [\"consul\", \"bootstrap\"]}}' >/etc/consul.d/bootstrap.json"

For the second one we could use potentially use this https://github.com/adammck/terraform-inventory (AWS only at the moment) which generates a dynamic ansible inventory based on a terraform state file.

Configure weave through terraform on the mesos masters/slaves

• Needs configuring on the fly via the terraform plan

Create a terraform plan for Digitalocean

Depends on #12 . Should maybe hang fire until we test a bit more with AWS to avoid having to refactor in 2 places.

Terraform apply fails first time, applies correctly on 2nd run

On a first run getting this error -

aws_instance.mesos-master.0: Creation complete
Error applying plan:

1 error(s) occurred:

* dial tcp 52.16.231.211:22: i/o timeout

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

The IP address obviously changes run to run with the instances changing.

Create a docker compose.yml to allow us to test the stack

Something like http://go.docker.com/e/44082/mesos-with-docker-compose-html/3rhy4/317817635

but will added consul + weave

Lock package versions in packer install.sh scripts

e.g. we do this

sudo apt-get install -y mesos

which just installs 'latest' from the repo. So at the moment we think we're building a 0.21 box but its actually a 0.22 one (since thats the latest in the repo)

Add a document outlining current limitations / things not yet addressed

e.g.

performance testing
monitoring
logging

Fix any issues against terraform 0.4.0

Due date for 0.4 to land is this week. We need to resolve any issues there may be with that (I had a few issues while trying to go to terraform master branch)

Tidy up README.md

Should include

• proper name of project
• brief description of what the project does
• Notes on how to install / run
• Links to other bits and bobs in the docs

Set collaborators up on quay.io for use as a private docker registry

Add support for SSL authentication

See https://github.com/CiscoCloud/microservices-infrastructure/blob/master/security-setup

https://github.com/CiscoCloud/microservices-infrastructure/blob/master/docs/security/security.rst

Dependency ordering issue with mesos master / slave nodes

We have no dependency set in mesos-slaves.tf, meaning if the slave node comes up first before a master then it will not connect to the mesos cluster properly.

We need to bring up all the masters first, then join the slaves

Create a single script for bootstrapping AWS

At the moment to get up and running on AWS we need to

• run terraform apply
• run a set of commands to setup the VPN and get the VPN key
• import the key into tunnelblick and connect to the VPN
• open up the web interfaces to the UIs

We should be able to create a single wrapper script that chains all of this so we just run a single command, and then end up with some web browser tabs open in a browser after all is completed

Create a drupal demonstrator

If we can use the michaelpage site - that would be nice

Basically mysql / memcache / drupal in containers that we can deploy to the stack. Depends on #53

Create a vagrant based setup

Consisting of -

• Vagrant provider
• Getting started guide for Vagrant

Create a dmz network & document networks

https://github.com/deverton/terraform-aws-consul as an example

Create a packer template for GCE

• Same as aws but google flavoured

Discuss, agree, implement plan for pluggable approach to provisioning infra services via terraform

It would be nice if we had pluggability so that we could mix and match our infrastructure on-demand.
For example we should be able to pick and choose which mesos frameworks to install http://mesosphere.com/docs/frameworks/ at runtime.

This should be possible by extracting stuff out to terraform modules https://www.terraform.io/docs/configuration/modules.html , and then composing them as we wish.

Later down the line we might have some higher order tool (command line / UI) do the composing of the terraform plan, I dont think thats needed imminently, but a way forward for implementing the underlying pluggability is desired.

Create a component overview docs section with decision logs

Components we need to document (for now) -

• Consul
• Weave
• Docker
• Mesos
• Marathon
• Dnsmasq
• Ansible

Create serverspec tests that test the end state of configuration after applied through terraform

These should test things like -

• services all up and running correctly
• can ping the web interfaces
• files provisioned correctly with correct values
  etc..

Lookup mesos-master IP addresses more dynamically in the nat server

At the moment we have this -

"echo ${var.master_ips.master-0} >> /home/ubuntu/masters",
"echo ${var.master_ips.master-1} >> /home/ubuntu/masters",
"echo ${var.master_ips.master-2} >> /home/ubuntu/masters"

It would be better if we could somehow lookup ${var.master_ips.*} using count.index or something similar (if possible)

Might be able to reuse a function from over here

https://www.terraform.io/docs/configuration/interpolation.html

Change wercker build to build non amazon images

• Currently we're building AWS and its using my AWS account credits
• lets switch this to just build the default images (no aws integration) and run the tests against those

I think there will be some issues to sift through around virtualbox (as there are some virtualbox scripts getting added in those)

wercker not catching correct exit code on test failures

Sometimes a serverspec test fails in wercker but the build just carries on and it looks like it passes

Set up slack notifications for wercker

See http://blog.wercker.com/2013/11/27/Slack-Notifications.html

We should push this to an #apollo channel on slack

Internal links and redirects for mesos not redirecting properly

For example if you visit http://10.0.1.11:5050 and .13 is the master. Mesos will try redirecting you, it will redirect to -

http://ip-10-0-1-13.eu-west-1.compute.internal:5050/

Which is not resolvable through the browser / VPN.

A similar thing occurs if you click on a link from the web interface to one of the frameworks. We probably need to set -

/etc/mesos-master/hostname
/etc/marathon/conf/hostname

to addresses that we can resolve properly.

Install and configure dnsmasq

• Download / installation of the package should be via packer
• Turning on / configuration should be done via terraform

This is so we can serve DNS requests for the .consul domain via the consul DNS server.

See www.morethanseven.net/2014/04/25/consul/ for more info

Add mesos-slave to Amazon auto scaling group

We should be able to just provision a single slave instance, then use amazon auto-scaling to handle provisioning new slave instances dynamically on demand.

See http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-scale-based-on-demand.html
and http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/GettingStartedTutorial.html

Add a continuous deployment demonstrator

Similar to our jenkins setup internally and kinda like this https://mesosphere.com/blog/2015/04/02/continuous-deployment-with-mesos-marathon-docker

We should probably create a simple demonstrator github repo - and spin out the other necessary bits in AWS. Maybe we could use a cloud CI (e.g. wercker / travis / circleCI / drone.io / or something similar so we don't have to do the jenkins heavy lifting).
If we hooked this up to quay.io for the docker registry then we could build / push from the cloud CI to quay.io and trigger a deployment to marathon.

Wercker supports deployments - maybe we could write a custom deployment step for that to deploy into AWS (needs more investigation)