cloudsidedev / appside Goto Github PK
View Code? Open in Web Editor NEWMultitenant environment automation.
Home Page: http://cloudside.ch
License: GNU Affero General Public License v3.0
Multitenant environment automation.
Home Page: http://cloudside.ch
License: GNU Affero General Public License v3.0
Need to investigate about borg and assh.
... like Memcached.
The memcache module for PHP is already available (maybe we should make Memcached (with d) available too), but no server.
Could https://github.com/geerlingguy/ansible-role-memcached be useful for this?
would be nice to have infos about the last AppFlow run (date) in motd.
I'd like to see a few things changed:
Some resources:
we something like make init
for creating a initial ~/.appflow
skeleton configuration layout and ~/.appflow/config
with your tenant name in it.
In the global php.ini
file, PHP sessions are currently turned on automatically. That means a PHPSESSID
cookie is set even though it's not being used or enforced by a project.
We want % appflow provision=<tenant> ...
.
Should be a wrapper
around Makefile
to start.
appflow -> /usr/local/bin/appflow
library -> /usr/local/share/appflow
this will lead to introduction of development
and stable
releases.
Looking at the htaccess.j2 file and noticed a few issues:
The iThemes Security block should be wrapped with a <IfModule mod_rewrite.c>
condition
Options -Indexes
can be removed since it's now disabled by default, see 68b9e2f
This block should be merged into one <IfModule mod_rewrite.c>
section
The WordPress rules should be extended to support a Multisite flag.
Currently you have to add the Multisite rules to the site config which means they get added before all the other security checks. They should be the last one, just like the default WP rules.
The rules for Multisite have a few variables which we need to define in the config. If one of them is set the provisioner would add the multisite site rules to the .htaccess file. Note that the multisite rules are a replacement of the default rules, so both versions shouldn't be added, which is currently the case.
Pseudo code (which can be improved once we know if this is possible):
# BEGIN WordPress
<IfModule mod_rewrite.c>
{% if item.value.multisite is defined %}
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
# add a trailing slash to /wp-admin
RewriteRule ^{{ item.value.multisite.subdir_match }}wp-admin$ {{ item.value.multisite.subdir_replacement_01 }}wp-admin/ [R=301,L]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule ^{{ item.value.multisite.subdir_match }}(wp-(content|admin|includes).*) /{{{ item.value.multisite.subdir_replacement_12 }}} [L]
RewriteRule ^{{ item.value.multisite.subdir_match }}(.*\.php)$ /{{ item.value.multisite.subdir_replacement_12 }} [L]
RewriteRule . index.php [L]
{% else %}
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
{% endif %}
</IfModule>
# END WordPress
This would prevent an error about a not encrypted file, see the following example:
➜ appflow git:(master) make decrypt env=staging
[decrypt][example][staging]
ERROR! input is not encrypted for ~/.appflow/tenant/appflow-example/staging/.DS_Store
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful
Decryption successful
the deploy
user has assh
as default ssh
enabled, we need to:
% mkdir -p ~/tmp/.ssh/cm
% sudo su - ; cd /usr/local/bin ; ln -s /usr/local/go/bin/assh .
this must be handled in https://github.com/ttssdev/appflow/blob/master/playbooks/roles/common/tasks/assh.yml.
actually when running appflow vagrant reload atlantis
instead of appflow vm reload atlantis
we get this:
mkdir -p ~/Downloads/Software
mkdir -p ~/Downloads/Software/Vagrant-Boxes
cd ~/Downloads/Software/Vagrant-Boxes && wget -c http://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box && mv trusty-server-cloudimg-amd64-vagrant-disk1.box trusty64.box && cd ~/Documents/webdev/appflow
--2017-01-11 11:13:54-- http://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box
Resolving cloud-images.ubuntu.com...
we should avoid this in order to break things, so maybe appflow vagrant
should be an alias for appflow vm
.
/etc/cron.d/wp-cron
entries should be HTTPS.
It would be handy if I could specify the URLs of a site somewhere so they get automatically added to my hosts file. This requires the Vagrant::Hostsupdater or Vagrant Host Manager plugin. (The latter one also updates the hosts file on the vm.)
Varying Vagrant Vagrants uses a file named vvv-hosts
for this.
We want to have a tenant config skeleton called mrrobot
which acts as reference for latest AppFlow variables, it should populate ~/.appflow/tenant/mrrobot
and ~/.appflow/vault/mrrobot
.
In playbooks/roles/web/tasks/main.yml
, mod_pagespeed
is currently commented out.
Leveraging some filters of this Apache module would be very beneficial for overall performance of the websites running on Appflow. mod_pagespeed
has filters to minimize request overhead, minimize payload size, and even filters to improve rendering time in the browser.
we want to add vhost
support for Typo3 CMS, this is similar to vhost-dev-wp_bedrock.conf.j2:
...
DocumentRoot /var/www/vhosts/typo3/web
<Directory /var/www/vhosts/typo3/>
Options FollowSymLinks
AllowOverride All
# Apache 2.4 only syntax, see: http://httpd.apache.org/docs/2.4/upgrading.html
# Require all granted
Order allow,deny
Allow from all
</Directory>
...
Initial setup issues have been following but should have been fixed through 930744b and aa094f7:
as suggested by @grappler we should provide atlantis
with adequate monitoring, options are:
monit
(bottom-up)icinga2
(top-down)munin
(top-down)Eventual usage of https://github.com/gosuri/vagrant-env to be more flexible.
In case WP can't reach the backend DB for whatever reason it returns an HTTP 500 status
code.
It currently seems that HAProxy will not see such an error code as an issue and does not mark the node as down. The HAProxy section for backend is defined here: https://github.com/ttssdev/appflow/blob/master/playbooks/roles/lb/templates/etc/haproxy/haproxy-cfg.j2#L103 in the backend HAProxy checks Varnish then, like:
server compute-01 xxx.xxx.xxx.xxx:6081 cookie control01 check port 6081 inter 12000 rise 3 fall 3
Varnish should return the 500
from Apache2, we need to check this.
Better solution:
Another issue is that each node has 127.0.0.1 mysql.local
as MySQL backend reference, here we should set the IP of the VIP keepalived
address (port 3307
), if possible, or the IP of the load balancer, this way each node will use HAProxy for MySQL queries and HAProxy always correctly knows which PXC is working and which not -- This allows us to not exclude a webserver service but just a PXC service.
wp-includes
and web/app
.The current commit message is "Auto commit".
I think it would be nice to be able to define a custom message. Example: appflow checkin env=production -m "Added new host"
we need to define a skeleton with example basic configurations, something that can be referenced in the documentation and allows a first easy AppFlow run. Something called mrrobot
tenant is already defined here: https://github.com/ttssdev/appflow/tree/master/examples, this has to be cleaned-up and get up running with something like appflow demo
or similar. Another idea is to startup AppFlow after the first run e show directly a link to https://atlantis with demo content.
it may happen from time to time that some services on atlantis
freeze or hang, this issue is here for starting to debug such situations. So actually when atlantis
freezes we should check this way:
ping atlantis
, does the VM respond?ssh atlantis
, possible to login via ssh
?haproxy
, varnish
and apache2
running (note: no HTTPS here)?apache2
only running (note: no HTTPS here)?at this point we can debug via ssh to see if some process is not running anymore:
ssh atlantis "ps aux | grep apache2"
root 7025 0.0 1.2 485748 26236 ? Ss 16:46 0:00 /usr/sbin/apache2 -k start
www-data 7036 0.0 2.1 489536 44856 ? S 16:46 0:00 /usr/sbin/apache2 -k start
www-data 7037 0.1 2.9 572452 59940 ? S 16:46 0:03 /usr/sbin/apache2 -k start
www-data 7038 0.1 3.1 502332 64296 ? S 16:46 0:04 /usr/sbin/apache2 -k start
www-data 7039 0.1 3.9 573760 81392 ? S 16:46 0:04 /usr/sbin/apache2 -k start
www-data 7040 0.2 5.0 597660 103220 ? S 16:46 0:06 /usr/sbin/apache2 -k start
www-data 7868 0.0 2.5 501964 51756 ? S 16:50 0:01 /usr/sbin/apache2 -k start
www-data 7937 0.0 1.6 489208 32884 ? S 16:50 0:00 /usr/sbin/apache2 -k start
www-data 7938 0.0 1.5 489520 32168 ? S 16:50 0:01 /usr/sbin/apache2 -k start
www-data 9159 0.2 2.5 502324 52408 ? S 16:55 0:05 /usr/sbin/apache2 -k start
eim 18602 0.0 0.0 12828 992 ? Ss 17:29 0:00 grep apache2
ssh atlantis "ps aux | grep varnish"
root 2339 0.0 0.2 126900 5288 ? Ss 16:45 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:60821 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
nobody 2340 0.1 0.9 292884 19884 ? Sl 16:45 0:05 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:60821 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
eim 18612 0.0 0.0 12828 992 ? Ss 17:29 0:00 grep varnish
ssh atlantis "ps aux | grep haproxy"
haproxy 1362 0.0 0.3 42196 7068 ? Ss 16:45 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid
eim 18623 0.0 0.0 12832 992 ? Ss 17:29 0:00 grep haproxy
the basic idea behind this is to understand which component has crashed in order to start investigation.
This is definitely a hot topic, would be great to add some kind of https://www.loggly.com support or similar, this way logs could be shipped directly to a target log server or cloud solution. It's important for devs in order to be able to get their logs in case of troubles or performance analysis.
ssdh
settings must be tuned for higher security standards, will be developed in branch 38-sshd-hardening. Features we want to add:
AllowUsers
.fail2ban
.DSA
and ECDSA
.DH 1024 Bit
and ECDH
.Arcfour
, 3DES
and CBC-Ciphers
.MD5
, SHA1
, <128 Bit
.3DES
, !3DES in OpenSSL
.Test can be done with ssh-audit
.
Example settings:
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
MACs [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
DebianBanner no
# Uncomment the two entries below after configuring and testing SSH pubkey authentication
# PasswordAuthentication no
# ChallengeResponseAuthentication no
# AllowUsers user1 user2 user3
PermitRootLogin no
As an administrator I want to be able to manage the available root certificates on my servers.
E.g. I want to add self signed root certificates.
To do this I want to be able to do the following manual steps with appflow:
We need a small tmate daemon on atlantis to be easly accessible for remote assistance
Considering the today most devs have machines with at least 8 or 16GB of physical RAM we could start considering to increase the default Vagrant memory value to 4GB, actually it's set to 2GB as defined in https://github.com/ttssdev/appflow/blob/master/Vagrantfile#L44.
Especially MySQL intensive tasks and queries could take advantage of more memory, actually some atlantis
nodes crash during while executing some heavy queries, like reported by @hubeRsen and @grappler.
Still some issues with GlusterFS peering and volume creation in https://github.com/ttssdev/appflow/blob/master/playbooks/roles/web/tasks/glusterfs.yml
to test from scratch (Debian Family):
% ssh testing "sudo apt purge percona-xtradb-cluster-client-5.6 percona-xtradb-cluster-common-5.6 percona-xtradb-cluster-galera-3.x percona-xtradb-cluster-server-5.6 percona-xtrabackup ; sudo rm -r /root/.my.cnf /etc/mysql ; sudo apt-get autoremove"
then:
% make provision limit=testing tags=mysql
since ansible 2.2.0.0
the make provision
does not automatically decrypt tenant's configs, seems to be related to ansible/ansible#18364.
In order to test AppFlow installations on a fresh macOS environment we need some kind of virtualization for starting tests in a clean vm, this link can help: https://github.com/geerlingguy/macos-virtualbox-vm/blob/master/README.md
When running vagrant reload atlantis
I get this error. I am running reload because the server is unresponsive ref #31
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.
Command: ["controlvm", "b206ffb4-a127-4b0c-bd46-18bbb02354e4", "poweroff"]
Stderr: 0%...10%...20%...30%...40%...50%...
Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to power off machine
VBoxManage: error: The VM session was aborted
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component SessionMachine, interface ISession
VBoxManage: error: Context: "RTEXITCODE handleControlVM(HandlerArg *)" at line 234 of file VBoxManageControlVM.cpp
```
would be interesting to replace atlantis
with a Docker based solution. We need to create different containers for each service like nginx
, PHP
, etc. A valid approach could be docker-compose
or Docker definition inside ansible
. This task also covers complete rewrite of AppFlow CLI app using Symfony, branch: https://github.com/ttssdev/appflow/tree/55-appflow-dev-environment, bin: https://github.com/ttssdev/appflow/blob/55-appflow-dev-environment/bin/appflow.
Symfony:
re-enable RBL check in Icinga
As @hubeRsen reported we nedd to check if all files are encrypted before making a push
Enable provisioning of the local node via ansible_connection=local
as described in http://docs.ansible.com/ansible/intro_inventory.html#non-ssh-connection-types -- could be a new tenant called local
or example tenant mrrobot
, open to ideas.
Right now, Apache in Appflow does not set any Expires
HTTP headers for resources like images, SVGs, etc. In my opinion this is basic 101 of performance optimization and easy to achieve at the same time.
Docs: http://httpd.apache.org/docs/current/mod/mod_expires.html
We want to add continuous integration
via Jenkins to the AppFlow development process.
This means when we push to master
or some else to-define
branch we automatically:
atlantis
or atlantis.centos
from scratch each time.AppFlow
provisioning for that new node.all this via Jenkins which can run remotely or locally.
I know this is just the beginning of the appflow setup, but it would be tremendously helpful to have some sort of documentation. The following parts would be great to have a documentation on:
make
commands of appflowI probably can help with the documentation or at least testing it out. Let me know if you'd need any help here.
Devs want to be able to protect a given vhost
with Apache2 Basic Authentication. This will implemented using a new var called htpasswd_password
, if set to false
no authentication will occur, if set to anything else this string will be the password, example:
Disable basic authentication:
foo:
state: enabled
...
htaccess:
htpasswd_password: false
...
Enable basic authentication:
foo:
state: enabled
...
htaccess:
htpasswd_password: my-secret-password
...
in this case when opening http://foo basic auth requires username
foo
and password
my-secret-password
. The username
will always default to the vhost
item (as in this example foo
).
The appflow bin from #25 works great but it doesn't support autocomplete for arguments.
➜ appflow git:(master) make [tab]
all debug firstrun jenkins provision-debug status tenant vagrant
args decrypt help limit reset support uninstall vagrant-centos
checkin encrypt init local skip_tags syntax-check update vault
checkout env install provision ssh tags url verbose
➜ appflow git:(master) appflow [tab]
LICENSE Vagrantfile* config.example playbooks/
Makefile Vagrantfile.local.yml docker-compose.yml utils/
README.md Vagrantfile.local.yml.sample.yaml docs/ \~pwd/
Vagrant-Boxes@ appflow* examples/
We don't want to show, per default, Apache version, PHP version and OS Version, this implies:
expose_php = Off
ServerTokens Prod
ServerSignature Off
we want to move config VM settings like:
atlantis.vm.provider "virtualbox" do |v|
v.customize ["modifyvm", :id, "--cpus", 2, "--memory", 2048, "--name", "vagrant-atlantis", "--natdnshostresolver1", "on"]
end
to ~/.appflow/config
in order to allow some more personalisation.
would ne nice to have a one-liner installation of AppFlow like brew
: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
. We can then reference this in https://github.com/ttssdev/appflow/wiki/Walkthrough.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.