cloudsidedev/appside

Multitenant environment automation.

AppSide ======

Read the documentation on appside.rtfd.io

For the playbooks, visit the appside-playbooks repository

More Screencasts: Installation, Basic setup, Atlantis setup, Atlantis provision, Project provision

Get an invite and join the conversations!

AppSide is a multitenant environment automation tool based on Ansible.

AppSide is your transparent platform that incorporates the various digital infrastructures into a continuous workflow. At the same time, it creates a consistent environment for the entire development process, saving time and money. AppSide is an open source developer and DevOps application that maps the entire development process (continuous deployment & provisioning) from development to test & staging to production. This ensures that the systems being managed on the different infrastructures are deployed in the same way. At the same time, AppSide creates a consistent environment (on-premise, in the cloud or on bare metal) throughout the development process. The administrators and developers (DevOps) can therefore independently control the entire toolchain. The automation process is created using Ansible playbooks and contains the admin and developer code, which is stored in a git repository (encrypted). The code (AppSide) runs on any UNIX-like operating system where Python can run, including Linux, BSD, and OSX. Orchestration of the AppSide code requires only SSH access to the respective systems. AppSide has been designed and optimized from DevOps for DevOps.

Features

Provisioning:

• Multitenant architecture (different teams with different environments)
• Supports development, testing, staging and production
• All configuration files are encrypted in git with ansible-vault
• Provision all nodes with one command

Development:

• Provides a Vagrant based development environment called atlantis
• Code locally on any Unix-like system or Windows (cygwin)

Deployment:

• Made for Bedrock projects and bedrock-capistrano deployments
• Deploy and rollback with one command

Infrastructure:

• Builtin Percona XtraDB Cluster 5.6 and GlusterFS support for sharing web uploads on multiple nodes
• Easy development environments with Vagrant
• Easy server provisioning with Ansible (Ubuntu 16.04, PHP 5.6/7)

Technologies

The technology behind AppSide uses countless best-in-class programs and maps them in a toolbox. The software accesses an extensive repository of various freeware packages. Out-of-the-box, many enterprise features already exist, such as:

• Load balancing
• Apache / PHP
• Web Accelerator / Caching / PageSpeed
• Distributed file system
• Master-Master Database
• Backup & Monitoring Integration
• Jailkit - chroot
• ...

Installation

AppSide is hosted on PiP using python3. pip3 install appside will install appside. To start using it you first need to initialize it: appside init follow the onscreen instructions to set it up!

Developers

Contribute a new feature

• Create a new issue, e.g. #XX new superfeature
• Create local branch: git checkout -b XX-new-superfeature
• Code on it.
• Push it to remote as new branch: git push -u origin XX-new-superfeature
• Create new pull request (base: master ... compare: XX-new-superfeature)

Get all vars

ansible all -m setup --tree /tmp/facts -i examples/YOUR_TENANT/local/inventory -a "filter=ansible_distribution*"

Contributing

Contributions are welcome from everyone.

Join us!

appside's Issues

Fix Percona XtraDB Cluster

to test from scratch (Debian Family):

% ssh testing "sudo apt purge percona-xtradb-cluster-client-5.6 percona-xtradb-cluster-common-5.6 percona-xtradb-cluster-galera-3.x percona-xtradb-cluster-server-5.6 percona-xtrabackup ; sudo rm -r /root/.my.cnf /etc/mysql ; sudo apt-get autoremove"

then:

% make provision limit=testing tags=mysql

macOS one-line installer complete run

In order to test AppFlow installations on a fresh macOS environment we need some kind of virtualization for starting tests in a clean vm, this link can help: https://github.com/geerlingguy/macos-virtualbox-vm/blob/master/README.md

Support an optional commit message for checkins

The current commit message is "Auto commit".

I think it would be nice to be able to define a custom message. Example: appflow checkin env=production -m "Added new host"

vagrant-env

Eventual usage of https://github.com/gosuri/vagrant-env to be more flexible.

Implement htpasswd (basic auth) support

Devs want to be able to protect a given vhost with Apache2 Basic Authentication. This will implemented using a new var called htpasswd_password, if set to false no authentication will occur, if set to anything else this string will be the password, example:

Disable basic authentication:

  foo:
    state: enabled
    ...
    htaccess:
    htpasswd_password: false
    ...

Enable basic authentication:

  foo:
    state: enabled
    ...
    htaccess:
    htpasswd_password: my-secret-password
    ...

in this case when opening http://foo basic auth requires username foo and password my-secret-password. The username will always default to the vhost item (as in this example foo).

Include domain or IP in bash prompt

Enable gzip for svg files too

Relocate Vagrant's VM settings to ~/.appflow/config

we want to move config VM settings like:

    atlantis.vm.provider "virtualbox" do |v|
      v.customize ["modifyvm", :id, "--cpus", 2, "--memory", 2048, "--name", "vagrant-atlantis", "--natdnshostresolver1", "on"]
    end

to ~/.appflow/config in order to allow some more personalisation.

vagrant reload atlantis fails

When running vagrant reload atlantis I get this error. I am running reload because the server is unresponsive ref #31

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["controlvm", "b206ffb4-a127-4b0c-bd46-18bbb02354e4", "poweroff"]

Stderr: 0%...10%...20%...30%...40%...50%...
Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to power off machine
VBoxManage: error: The VM session was aborted
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component SessionMachine, interface ISession
VBoxManage: error: Context: "RTEXITCODE handleControlVM(HandlerArg *)" at line 234 of file VBoxManageControlVM.cpp
```

appflow bin

We want % appflow provision=<tenant> ....
Should be a wrapper around Makefile to start.

appflow -> /usr/local/bin/appflow
library -> /usr/local/share/appflow

this will lead to introduction of development and stable releases.

Enable mod_pagespeed

In playbooks/roles/web/tasks/main.yml, mod_pagespeed is currently commented out.

Leveraging some filters of this Apache module would be very beneficial for overall performance of the websites running on Appflow. mod_pagespeed has filters to minimize request overhead, minimize payload size, and even filters to improve rendering time in the browser.

assh for deploy user

the deploy user has assh as default ssh enabled, we need to:

% mkdir -p ~/tmp/.ssh/cm
% sudo su - ; cd /usr/local/bin ; ln -s /usr/local/go/bin/assh .

this must be handled in https://github.com/ttssdev/appflow/blob/master/playbooks/roles/common/tasks/assh.yml.

Turn off PHP sessions

In the global php.ini file, PHP sessions are currently turned on automatically. That means a PHPSESSID cookie is set even though it's not being used or enforced by a project.

make init

we something like make init for creating a initial ~/.appflow skeleton configuration layout and ~/.appflow/config with your tenant name in it.

Improve default Varnish config

Current file: https://github.com/ttssdev/appflow/blob/3326173f651a4e6b5ace960f167d56659a2221ff/playbooks/roles/web/templates/etc/varnish/default.vcl.j2

I'd like to see a few things changed:

• The default cache time should be increased, it's currently 2 minutes (https://github.com/nicolargo/varnish-nginx-wordpress/blob/master/varnish/varnish4-wordpress#L203-L206)
• Add cache hit/miss headers to make debugging easier (https://gist.github.com/ryan-frankel/10271802#file-varnishwithwordpress-vcl-L174-L187)
• Ignore some WordPress cookies (https://github.com/nicolargo/varnish-nginx-wordpress/blob/master/varnish/varnish4-wordpress#L78-L99)
• What else? Put them in the comments.

Some resources:

• http://ryanfrankel.com/using-varnish-with-wordpress/
• https://konstantin.blog/2011/my-website-is-now-super-fast-with-varnish/
• https://konstantin.blog/2011/varnish-and-preview-posts-in-wordpress/
• http://pastebin.com/7KpZrAG0
• http://perrymitchell.net/article/a-load-balanced-website-using-php-fpm-nginx-mysql-and-varnish-part-3/

tmate daemon for easy access on atlantis

We need a small tmate daemon on atlantis to be easly accessible for remote assistance

motd, add info about latest AppFlow

would be nice to have infos about the last AppFlow run (date) in motd.

Provision should update hosts file automatically

It would be handy if I could specify the URLs of a site somewhere so they get automatically added to my hosts file. This requires the Vagrant::Hostsupdater or Vagrant Host Manager plugin. (The latter one also updates the hosts file on the vm.)

Varying Vagrant Vagrants uses a file named vvv-hosts for this.

re-enable RBL check in Icinga

re-enable RBL check in Icinga

Check if all files are encrypted before making a push

As @hubeRsen reported we nedd to check if all files are encrypted before making a push

Support for managing root certificates

As an administrator I want to be able to manage the available root certificates on my servers.
E.g. I want to add self signed root certificates.
To do this I want to be able to do the following manual steps with appflow:

• mkdir /usr/share/ca-certificates/extra
• cp AnyRootCert.crt /usr/share/ca-certificates/extra/
• add the new certificate to /etc/ca-certificates.conf
• make the certificates available system wide: dpkg-reconfigure ca-certificates

atlantis service hangs

it may happen from time to time that some services on atlantis freeze or hang, this issue is here for starting to debug such situations. So actually when atlantis freezes we should check this way:

• ping atlantis, does the VM respond?
• ssh atlantis, possible to login via ssh?
• open https://atlantis/lb, is the LB running? u: admin p: admin
• open http://atlantis, are haproxy, varnish and apache2 running (note: no HTTPS here)?
• open http://atlantis:8080, is apache2 only running (note: no HTTPS here)?

at this point we can debug via ssh to see if some process is not running anymore:

ssh atlantis "ps aux | grep apache2"
root      7025  0.0  1.2 485748 26236 ?        Ss   16:46   0:00 /usr/sbin/apache2 -k start
www-data  7036  0.0  2.1 489536 44856 ?        S    16:46   0:00 /usr/sbin/apache2 -k start
www-data  7037  0.1  2.9 572452 59940 ?        S    16:46   0:03 /usr/sbin/apache2 -k start
www-data  7038  0.1  3.1 502332 64296 ?        S    16:46   0:04 /usr/sbin/apache2 -k start
www-data  7039  0.1  3.9 573760 81392 ?        S    16:46   0:04 /usr/sbin/apache2 -k start
www-data  7040  0.2  5.0 597660 103220 ?       S    16:46   0:06 /usr/sbin/apache2 -k start
www-data  7868  0.0  2.5 501964 51756 ?        S    16:50   0:01 /usr/sbin/apache2 -k start
www-data  7937  0.0  1.6 489208 32884 ?        S    16:50   0:00 /usr/sbin/apache2 -k start
www-data  7938  0.0  1.5 489520 32168 ?        S    16:50   0:01 /usr/sbin/apache2 -k start
www-data  9159  0.2  2.5 502324 52408 ?        S    16:55   0:05 /usr/sbin/apache2 -k start
eim      18602  0.0  0.0  12828   992 ?        Ss   17:29   0:00 grep apache2

ssh atlantis "ps aux | grep varnish"
root      2339  0.0  0.2 126900  5288 ?        Ss   16:45   0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:60821 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
nobody    2340  0.1  0.9 292884 19884 ?        Sl   16:45   0:05 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:60821 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
eim      18612  0.0  0.0  12828   992 ?        Ss   17:29   0:00 grep varnish

ssh atlantis "ps aux | grep haproxy"
haproxy   1362  0.0  0.3  42196  7068 ?        Ss   16:45   0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
eim      18623  0.0  0.0  12832   992 ?        Ss   17:29   0:00 grep haproxy

the basic idea behind this is to understand which component has crashed in order to start investigation.

advanced-ssh-config

Support https://github.com/moul/advanced-ssh-config/blob/master/README.md.

create one-liner installation

would ne nice to have a one-liner installation of AppFlow like brew: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)". We can then reference this in https://github.com/ttssdev/appflow/wiki/Walkthrough.

Define a reference tenant config skeleton

We want to have a tenant config skeleton called mrrobot which acts as reference for latest AppFlow variables, it should populate ~/.appflow/tenant/mrrobot and ~/.appflow/vault/mrrobot.

atlantis monitoring

as suggested by @grappler we should provide atlantis with adequate monitoring, options are:

• monit (bottom-up)
• https://github.com/prometheus/node_exporter
• icinga2 (top-down)
• munin (top-down)
• https://nicolargo.github.io/glances/

Add support for a memory object caching system

... like Memcached.

The memcache module for PHP is already available (maybe we should make Memcached (with d) available too), but no server.

Could https://github.com/geerlingguy/ansible-role-memcached be useful for this?

appflow vagrant

actually when running appflow vagrant reload atlantis instead of appflow vm reload atlantis we get this:

mkdir -p ~/Downloads/Software
mkdir -p ~/Downloads/Software/Vagrant-Boxes
cd ~/Downloads/Software/Vagrant-Boxes &&  wget -c http://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box  && mv trusty-server-cloudimg-amd64-vagrant-disk1.box trusty64.box && cd ~/Documents/webdev/appflow
--2017-01-11 11:13:54--  http://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
Resolving cloud-images.ubuntu.com...

we should avoid this in order to break things, so maybe appflow vagrant should be an alias for appflow vm.

appflow-dev-environment

would be interesting to replace atlantis with a Docker based solution. We need to create different containers for each service like nginx, PHP, etc. A valid approach could be docker-compose or Docker definition inside ansible. This task also covers complete rewrite of AppFlow CLI app using Symfony, branch: https://github.com/ttssdev/appflow/tree/55-appflow-dev-environment, bin: https://github.com/ttssdev/appflow/blob/55-appflow-dev-environment/bin/appflow.

Symfony:

• Defining and Processing Configuration Values

Security: Disable Directory Listing

• Disable Directory Listing everyhwere.
• Disallow executing PHP directly in wp-includes and web/app.

Ignore .DS_Store when decrypting

This would prevent an error about a not encrypted file, see the following example:

➜  appflow git:(master) make decrypt env=staging
[decrypt][example][staging]
ERROR! input is not encrypted for ~/.appflow/tenant/appflow-example/staging/.DS_Store
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful

Remove version info in PHP/Apache2

We don't want to show, per default, Apache version, PHP version and OS Version, this implies:

• expose_php = Off
• ServerTokens Prod
• ServerSignature Off

Fix GlusterFS peer probe and volume creation

Still some issues with GlusterFS peering and volume creation in https://github.com/ttssdev/appflow/blob/master/playbooks/roles/web/tasks/glusterfs.yml

Raise default Vagrant memory value to 4GB

Considering the today most devs have machines with at least 8 or 16GB of physical RAM we could start considering to increase the default Vagrant memory value to 4GB, actually it's set to 2GB as defined in https://github.com/ttssdev/appflow/blob/master/Vagrantfile#L44.

Especially MySQL intensive tasks and queries could take advantage of more memory, actually some atlantis nodes crash during while executing some heavy queries, like reported by @hubeRsen and @grappler.

Improve structure of htaccess.j2

Looking at the htaccess.j2 file and noticed a few issues:

• The iThemes Security block should be wrapped with a <IfModule mod_rewrite.c> condition

• Options -Indexes can be removed since it's now disabled by default, see 68b9e2f

• This block should be merged into one <IfModule mod_rewrite.c> section

• The WordPress rules should be extended to support a Multisite flag.

  Currently you have to add the Multisite rules to the site config which means they get added before all the other security checks. They should be the last one, just like the default WP rules.
  The rules for Multisite have a few variables which we need to define in the config. If one of them is set the provisioner would add the multisite site rules to the .htaccess file. Note that the multisite rules are a replacement of the default rules, so both versions shouldn't be added, which is currently the case.

  Pseudo code (which can be improved once we know if this is possible):

  # BEGIN WordPress
  <IfModule mod_rewrite.c>
  {% if item.value.multisite is defined %}
   RewriteEngine On
   RewriteBase /
   RewriteRule ^index\.php$ - [L]
   # add a trailing slash to /wp-admin
   RewriteRule ^{{ item.value.multisite.subdir_match }}wp-admin$ {{ item.value.multisite.subdir_replacement_01 }}wp-admin/ [R=301,L]
  
   RewriteCond %{REQUEST_FILENAME} -f [OR]
   RewriteCond %{REQUEST_FILENAME} -d
   RewriteRule ^ - [L]
   RewriteRule ^{{ item.value.multisite.subdir_match }}(wp-(content|admin|includes).*) /{{{ item.value.multisite.subdir_replacement_12 }}} [L]
   RewriteRule ^{{ item.value.multisite.subdir_match }}(.*\.php)$ /{{ item.value.multisite.subdir_replacement_12 }} [L]
   RewriteRule . index.php [L]
  {% else %}
   RewriteEngine On
   RewriteBase /
   RewriteRule ^index\.php$ - [L]
   RewriteCond %{REQUEST_FILENAME} !-f
   RewriteCond %{REQUEST_FILENAME} !-d
   RewriteRule . /index.php [L]
  {% endif %}
  </IfModule>
  # END WordPress

Upgrade to PHP 7

Documentation: Add guides to wiki

I know this is just the beginning of the appflow setup, but it would be tremendously helpful to have some sort of documentation. The following parts would be great to have a documentation on:

• Troubleshooting guide (common mistakes, error codes and how to fix)
• Add new client project (in my case specifically for WordPress setups)
• Document the common make commands of appflow
• Common stuff like add PHP module and provision it on all nodes

I probably can help with the documentation or at least testing it out. Let me know if you'd need any help here.

wp-cron should use HTTPS

/etc/cron.d/wp-cron entries should be HTTPS.

No bash completion for appflow bin

The appflow bin from #25 works great but it doesn't support autocomplete for arguments.

➜  appflow git:(master) make [tab]
all              debug            firstrun         jenkins          provision-debug  status           tenant           vagrant
args             decrypt          help             limit            reset            support          uninstall        vagrant-centos
checkin          encrypt          init             local            skip_tags        syntax-check     update           vault
checkout         env              install          provision        ssh              tags             url              verbose

➜  appflow git:(master) appflow [tab]
LICENSE                            Vagrantfile*                       config.example                     playbooks/
Makefile                           Vagrantfile.local.yml              docker-compose.yml                 utils/
README.md                          Vagrantfile.local.yml.sample.yaml  docs/                              \~pwd/
Vagrant-Boxes@                     appflow*                           examples/

Typo3 support

we want to add vhost support for Typo3 CMS, this is similar to vhost-dev-wp_bedrock.conf.j2:

        ...
        DocumentRoot /var/www/vhosts/typo3/web

        <Directory /var/www/vhosts/typo3/>
                Options FollowSymLinks
                AllowOverride All
                # Apache 2.4 only syntax, see: http://httpd.apache.org/docs/2.4/upgrading.html
                # Require all granted
                Order allow,deny
                Allow from all
        </Directory>
        ...

Initial setup issues have been following but should have been fixed through 930744b and aa094f7:

Set Expires header for static resources

Right now, Apache in Appflow does not set any Expires HTTP headers for resources like images, SVGs, etc. In my opinion this is basic 101 of performance optimization and easy to achieve at the same time.

Docs: http://httpd.apache.org/docs/current/mod/mod_expires.html

sshd hardening

ssdh settings must be tuned for higher security standards, will be developed in branch 38-sshd-hardening. Features we want to add:

• Disable root login.
• Limit access to AllowUsers.
• Limit access to specific IP ranges (must be probably set of firewall).
• Disable password auth.
• Add fail2ban.
• Disable OS banner.
• Disable DSA and ECDSA.
• Disable DH 1024 Bit and ECDH.
• Disable Arcfour, 3DES and CBC-Ciphers.
• Disable MD5, SHA1, <128 Bit.
• Disable 3DES, !3DES in OpenSSL.

Test can be done with ssh-audit.
Example settings:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key

KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
MACs          [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256
Ciphers       [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr

DebianBanner no

# Uncomment the two entries below after configuring and testing SSH pubkey authentication
# PasswordAuthentication no

# ChallengeResponseAuthentication no
# AllowUsers user1 user2 user3

PermitRootLogin no

Cloud logging

This is definitely a hot topic, would be great to add some kind of https://www.loggly.com support or similar, this way logs could be shipped directly to a target log server or cloud solution. It's important for devs in order to be able to get their logs in case of troubles or performance analysis.

CI

We want to add continuous integration via Jenkins to the AppFlow development process.
This means when we push to master or some else to-define branch we automatically:

• Create atlantis or atlantis.centos from scratch each time.
• Start a clean AppFlow provisioning for that new node.
• Send results via mail or notify in Slack.
• Destroy the created Vagrant VM.
• Restart the process.

all this via Jenkins which can run remotely or locally.

Performance: Enable HTTP/2

See also https://info.varnish-software.com/blog/now-available-varnish-cache-5-http2-support

HAProxy PXC down

In case WP can't reach the backend DB for whatever reason it returns an HTTP 500 status code.
It currently seems that HAProxy will not see such an error code as an issue and does not mark the node as down. The HAProxy section for backend is defined here: https://github.com/ttssdev/appflow/blob/master/playbooks/roles/lb/templates/etc/haproxy/haproxy-cfg.j2#L103 in the backend HAProxy checks Varnish then, like:

server compute-01 xxx.xxx.xxx.xxx:6081 cookie control01 check port 6081 inter 12000 rise 3 fall 3

Varnish should return the 500 from Apache2, we need to check this.

Better solution:

Another issue is that each node has 127.0.0.1 mysql.local as MySQL backend reference, here we should set the IP of the VIP keepalived address (port 3307), if possible, or the IP of the load balancer, this way each node will use HAProxy for MySQL queries and HAProxy always correctly knows which PXC is working and which not -- This allows us to not exclude a webserver service but just a PXC service.

borg assh

Need to investigate about borg and assh.

Define local provisioning

Enable provisioning of the local node via ansible_connection=local as described in http://docs.ansible.com/ansible/intro_inventory.html#non-ssh-connection-types -- could be a new tenant called local or example tenant mrrobot, open to ideas.

Create a skeleton for example configurations

we need to define a skeleton with example basic configurations, something that can be referenced in the documentation and allows a first easy AppFlow run. Something called mrrobot tenant is already defined here: https://github.com/ttssdev/appflow/tree/master/examples, this has to be cleaned-up and get up running with something like appflow demo or similar. Another idea is to startup AppFlow after the first run e show directly a link to https://atlantis with demo content.

make provision does not decrypt

since ansible 2.2.0.0 the make provision does not automatically decrypt tenant's configs, seems to be related to ansible/ansible#18364.