osm-fr/ansible-scripts

Ansible commands for servers handled by Openstreetmap-France

Ansible scripts to handle OSM-FR machines

This repository contains various scripts to setup and configure machines handled by OSM-FR association (a french association for OpenStreetMap). These scripts are used with ansible.

Installing dependencies

Installing on Debian

A version of ansible >= 2.3 is sufficient, and is available in Debian, or from git repository. To install all dependency on Debian, use:

apt-get install ansible

To install VM through proxmox, python promoxer module is necessary, It can be installed with:

apt-get install python3-proxmoxer

Installing using venv

venv module creates isolated Python environments.

First of all, install the required packages. On Debian, use:

apt-get install python3 python3-venv

You can jump into a new isolated environments using:

python3 -m venv <folder>      # Create the environment, only needs to be done once
source <folder>/bin/activate  # Run a subshell using the environment 

Then, install the required packages:

pip install -r requirements.txt

Note: The created environment folder should be placed outside this project folder to prevent ansible-lint from looking at it.

Using ansible scripts

Installation and configuration of a new VM with proxmox

1. launch ./tools/add-vm.py to update file hosts and create proxmox configuration. Command line options are available to change parameters like number of cpus or size of disk.
2. set shell variables PROXMOX_PASSWORD and PROXMOX_SSHPUBKEY
3. launch following command to create VM, configure network on host for ipv6 access, and update local /etc/hosts to access VM before DNS update:

   ansible-playbook -u root -l <hostname> common.yml

Configuration of a new machine with default configuration

1. add the machine to file hosts, in section [vm] if it is a virtual machine, at the top otherwise.
2. launch following command:

   ansible-playbook -l <hostname> common.yml

Adding a new user to a specific machine

1. add the machine to file hosts, in the relevant section [user]
2. if necessary, add the user to group_vars/all/accounts.yml, with the public ssh key in public\_keys/<user>
3. launch following command:

   ansible-playbook -l <hostname> accounts.yml

Adding a service to a specific machine

1. add the machine to file hosts, in the relevant section [service]
2. launch following command:

   ansible-playbook -l <hostname> <service>.yml

ansible-scripts's Issues

role renderd should install munin plugins

role renderd should install munin plugins

add favicon.ico to ngnix/apache default vhost

add favicon.ico to ngnix/apache default vhost
maybe get it from osm-fr website or osm.org website

ajouter real_ip dans le rôle nginx pour les backend

ajouter quelque chose comme

set_real_ip_from 10.0.0.0/24;
real_ip_header X-Forwarded-For;

afin que tous les backend aient l'ip réel dans les logs nginx

python2 encore nécessaire pour osmose ?

https://github.com/osm-fr/ansible-scripts/blob/master/roles/osmose-backend/files/README.deb#L7
selon @freed osmose n'utilise plus python2
@jocelynj on vire cette ligne ?

taginfo : improve count-key.sh

• move the "rm france.o5m" inside the script for better readability
• add a debug mode (if DEBUG=1) that doesn't do the rm, this is usefull to avoid pbf -> o5m when coding

taginfo : add initial osh download

• initial osh download

change munin osm label

https://munin.openstreetmap.fr/osm26.openstreetmap.fr/osm147.openstreetmap.fr/osm_db_lag.html
change from : label "osm_base"
to "db replication lag" or something like that

load des vm <> load du host

Mettre dans ansible la modif lue sur irc comme faite à la main
modifier la config proxmox pour que le load average vu des VM ne soit pas celui du host mais que celui de la VM sed -i 's/ExecStart=/usr/bin/lxcfs /var/lib/lxcfs$/ExecStart=/usr/bin/lxcfs /var/lib/lxcfs -l/' /lib/systemd/system/lxcfs.service mais pour que ce soit actif, il faut stopper toutes les VM, relancer ce service puis les VM... donc bon pour le prochain reboot

update taginfo role

last run fail

name: compile OSM-binary
command: chdir=/data/project/taginfo/OSM-binary/src make

role osmose-frontend is out-of-sync with server

one unit in the role <> 2x2 in the server

create a empty robots.txt for ngnix and/or apache default virtual host

create a empty robots.txt for ngnix and/or apache default virtual host

Monitoring de backuppc

Comme demandé dans l'issue 77 d'infrastructure, il faudrait rajouter un script de monitoring des backups dans le role backuppc-server.

task "configure hostname" fail with ansible 2.8.0.dev0

$ ansible-playbook -u root -l osm312.openstreetmap.fr common.yml
<...>
TASK [common : Configure hostname] *********************************************************************************************************************************************
fatal: [osm312.openstreetmap.fr]: FAILED! => {"changed": false, "msg": "Command failed rc=1, out=, err=Could not get property: Failed to activate service 'org.freedesktop.hostname1': timed out\n"}
$ ansible --version
ansible 2.8.0.dev0

Add changes to ansible (hugepages, munin plugins)

Just a reminder for me to not forger to add these changes in ansible:

• memory plugin on proxmox host
• Add zfs plugin from munin contrib
• Configure nr_overcommit_hugepages on host
• Maybe over kernel tuning (vm.swappiness =1 ...)
• enable atop and sysstats
• add zfsiostat

role renderd doesn't set tile_dir=/var/lib/mod_tile

it's the conf by default tile_dir=/var/cache/renderd

proxmox template

la tache download proxmox template donne l'impression de télécharger le template à chaque exécution
par ailleurs il est sans doute préférable de télécharger le templace directement sur le hosteur plutôt que télécharger en local puis l'envoyer au hosteur

We should fix postgres version

Due to this task:

ansible-scripts/roles/postgresql/tasks/main.yml

Line 16 in b977744

- postgresql

When the playbook is executed, it installs the latest postgres version. We should indicate postgres'version to avoid that.

role common produce "changed" status for no-change

• TASK [common : enable nesting support on VM, required for systemd]
• TASK [common : bootstrap python3-apt package]
• TASK [common : add authorized_keys for user backuppc]
• TASK [common : add ssh key to user backuppc]
• TASK [common : check if apache installed]

role renderd produce inreadable style file

An error occurred while loading the map layer 'admin8': boost::filesystem::status: Permission denied: "/data/project/layers/layers-mapnik-styles/admin8.xml"

issue is at drwxr-x--- 3 layers layers 8 Jul 10 11:33 /data/project/layers/