chrismeyersfsu/provision_docker

Test Ansible roles and playbooks using Docker

provision_docker

An Ansible role to help you test your roles. Automagically create docker containers for each of your inventory hosts. Use your production inventory file to create docker containers for development and test.

Resource	Description
Blog post	Example provision_docker uses
.travis.yml	Example TravisCI
Dockerfiles	Curated Docker images with init system (so the service module works) and ssh daemon. chrismeyers/centos6 chrismeyers/centos7 chrismeyers/ubuntu12.04 ubuntu-upstart:14.04
test/playbook_*.yml role-install_mongod role-ansible_deps role-iptables	Example provision-docker projects and uses.

NEW docker_connection

Works with docker for Mac, VirtualBox, VMware Fusion, docker native. Using docker_connection does not require any routing rules.

# inventory
[robots]
optimus image="chrismeyers/ubuntu12.04"
bumblebee image="ubuntu-upstart:14.04"

# test.yml
- name: Bring up docker containers for docker connection inventory iface
  hosts: localhost
  roles:
    - role: provision_docker
      provision_docker_privileged: true,
      provision_docker_inventory_group: "{{ groups['robots'] }}"
      provision_docker_use_docker_connection: true

- hosts: robots
  tasks:
    - name: "Say hello to my new containers"
      ping:

parameter	required	default	choices	comments
provision_docker_image	no	chrismeyers/centos6	chrismeyers/centos6 chrismeyers/centos7 chrismeyers/ubuntu12.04 ubuntu-upstart:14.04 other	Docker image to use when starting the container. The containers listed to the left are special. The init system put back in and ssh is started. This allows for starting/stopping service via the service module as well as ssh.
provision_docker_privileged	no	true	true/false	Start Docker container in privileged mode.
provision_docker_inventory_group	no			List of host names that are in the inventory for which to bring up a Docker container. Note that the Docker image that you wish to bring up should be a hostvar associated with the hostname.
provision_docker_inventory	no			List of <name, image> pairs for which to bring up a Docker container.
provision_docker_use_docker_connection	no	false	true/false	Use docker_connection plugin to connect to Docker containers instead of the default ssh.
provision_docker_network	no		Some name from available networks as listed with $ docker network ls	Specify the network that the Docker container should connect to.
provision_docker_volumes	no		List of volumes to mount within the container.	Use docker CLI-style syntax: /host:/container[:mode].
provision_docker_volumes_from	no		List of container names or to get volumes from.

Mac OS X + docker-machine + VMware Fusion

sudo /sbin/route -n add -net 172.17.0.0 -netmask 255.255.0.0 -gateway $(docker-machine ip default)

Mac OS X + docker-machine + VirtualBox

provision_docker relies on being able to ssh to containers. Thus, the ip of the container must be accessible (a route must exist). If your using docker toolbox on OS X + virtualbox containers are not routed to the host. Run the below commands to be add a route to the containers in the guest VM.

/usr/sbin/scutil -w State:/Network/Interface/vboxnet0/IPv4 -t 0
sudo /sbin/route -n add -net 172.17.0.0 -netmask 255.255.0.0 -gateway $(docker-machine ip)

The route does not persist across reboots. To persist the changes edit /Library/LaunchDaemons/com.docker.route.plist

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE plist PUBLIC '-//Apple//DTD PLIST 1.0//EN' 'http://www.apple.com/DTDs/PropertyList-1.0.dtd'>
<plist version='1.0'>
<dict>
  <key>Label</key>
  <string>com.docker.route</string>
  <key>ProgramArguments</key>
  <array>
    <string>bash</string>
    <string>-c</string>
    <!-- You need to adapt the vboxnet0 to the interface that suits your setup, use ifconfig to find it -->
    <string>/usr/sbin/scutil -w State:/Network/Interface/vboxnet0/IPv4 -t 0;sudo /sbin/route -n add -net 172.17.0.0 -netmask 255.255.0.0 -gateway 192.168.99.100</string>
  </array>
  <key>KeepAlive</key>
  <false/>
  <key>RunAtLoad</key>
  <true/>
  <key>LaunchOnlyOnce</key>
  <true/>
</dict>
</plist>

Expose and publish ports

Set expose and ports lists in your inventory per container. Checkout the docker_container Ansible module for reference.

Similar Work

• https://github.com/metacloud/molecule
• https://github.com/AerisCloud/ansible-role-test
• http://www.jeffgeerling.com/blog/testing-ansible-roles-travis-ci-github
• https://github.com/geerlingguy/ansible-role-apache
• https://github.com/neillturner/kitchen-ansible

Projects Using provision_docker

• https://github.com/mbreisch/ssh-role
• https://github.com/mbreisch/ufw-role
• https://github.com/mbreisch/ssmtp-role
• https://github.com/mbreisch/deploy-user-role
• https://github.com/mbreisch/s3cmd-role
• https://github.com/mbreisch/unattended-upgrades-role/tree/master/tests
• https://github.com/Maarc/ansible-role-redhat-jboss-common
• https://github.com/sderen/ansible-grafana
• https://github.com/sderen/ansible-nginxgunicorn
• https://github.com/Maarc/ansible-role-redhat-jboss-web-server-httpd
• https://github.com/Maarc/ansible-role-redhat-jboss-web-server-tomcat
• https://github.com/Maarc/ansible-role-redhat-jboss-eap
• https://github.com/rhtconsulting/jboss_fuse
• https://github.com/rhtconsulting/jboss_bxms
• https://github.com/mlanin/ansible-laravel5/tree/master/roles/carlosbuenosvinos.ansistrano-deploy
• https://github.com/Maarc/ansible-role-redhat-jboss-common
• https://github.com/turkenh/ansible-role-parse-mongodb
• https://github.com/tomashavlas/ansible-role-users_profiles
• https://github.com/tomashavlas/ansible-role-system_users
• https://github.com/tomashavlas/ansible-role-authorized_keys
• https://github.com/tomashavlas/ansible-role-sudo
• https://github.com/tomashavlas/ansible-role-system_groups
• https://github.com/rhevm-qe-automation/ovirt-ansible
• https://github.com/rhevm-qe-automation/ansible-role-seal

provision_docker's Issues

Documentation configuration variables for docker

Explain all this

      provision_docker_image: myimage
      provision_docker_privileged: true,
      provision_docker_inventory_group: "{{ groups['robots'] }}"
      provision_docker_use_docker_connection: true

esp provision_docker_image is missing completely

Rpmdb checksum is invalid: dCDPT(pkg checksums)

When installing packages, the error in the title appears for CentOS7 and CentOS6 images.

Error with Arch Linux because of python3

Hi there,

this role won't work on arch linux because /usr/bin/python (which is used by ansible-playbook) is version 3.6.1 currently.

This leads to the following error using this role:

TASK [chrismeyersfsu.provision_docker : Make sure able to connect to hosts] ******************

The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 97, in run
    item_results = self._run_loop(items)
  File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 290, in _run_loop
    res = self._execute(variables=task_vars)
  File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 472, in _execute
    self._connection = self._get_connection(variables=variables, templar=templar)
  File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 729, in _get_connection
    connection = self._shared_loader_obj.connection_loader.get(conn_type, self._play_context, self._new_stdin)
  File "/usr/lib/python3.6/site-packages/ansible/plugins/__init__.py", line 373, in get
    obj = obj(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/docker.py", line 78, in __init__
    docker_version = self._get_docker_version()
  File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/docker.py", line 138, in _get_docker_version
    for line in cmd_output.split('\n'):
TypeError: a bytes-like object is required, not 'str'

fatal: [localhost]: FAILED! => {
    "failed": true,
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

Manually setting /usr/bin/python2 as the executor for ansible-playbook works but that's not a neat soloution.

Also setting ansible_python_interpreter=/usr/bin/python2 won't work here too.

I am not too sure if the problem lies in ansible itself through.

Ansible version is:

ansible 2.3.0.0
config file = /etc/ansible/ansible.cfg
configured module search path = Default w/o overrides
python version = 3.6.1 (default, Mar 27 2017, 00:27:06) [GCC 6.3.1 20170306]

docker_inspect.sh not found when provisioning on remote machine

 "stderr_lines": [
        "/bin/sh: /projects/_48__ansible_engineering/roles/provision_docker/files/docker_inspect.sh: No such file or directory"
    ],

docker_inspect.sh exists because you can't call register: on a module that takes parameters with {{ }} in the value. For remote nodes, we need to call copy to the remote node before running docker_inspect.sh.

Open-source licence

Would it be possible to get this module released under an explicit open source licence?
At the moment it appears we need to rely on GitHub's general Terms & Conditions for rights.

Explain how the inventory should look like and OSX quirks

You should explain that the inventory needs at least

localhost ansible_connection=local

And under osx, most probably some kind of brew / boxen environment, that users need to add the python binary they installed ansible for

Thats for boxen + pyenv. People should use where python to find out what they need

localhost ansible_connection=local ansible_python_interpreter=/opt/boxen/pyenv/shims/python

Feature: Passing host vars to container host?

It would be nice to pass along every variable that were defined in group_vars and host_vars and in the inventory file to the host created with add_host. This way all existing playbooks could be run against the container without change.

There are comments about this in tasks/inc_inventory_iface.yml and tasks/inc_cloud_iface.yml:

    # TODO: copy ALL host vars in the inventory

Any suggestions on how to do this properly?

Does this work with swarm?

Does our design work with docker swarm?

Specifically, I am thinking of how we identify/route to the docker containers, by their internal ip address. Maybe we should be using their hostname?

default variables fail

When running this playbook I get an error with not defining the provision docker inventory and group variables.

TASK [Ensure required vars are defined] ********************************************************************************* fatal: [localhost]: FAILED! => {"changed": false, "msg": "provision_docker_inventory_group or provision_docker_inventory is required"}

Here is what I have set in the defaults variables. Im not sure if how the variables should be set but I figured if I throw anything as a variable this check should work and then fail later down the playbook.
``
provision_docker_image_default: "centos/systemd"
provision_docker_ip: "127.0.0.1"
provision_docker_use_tls: true
provision_docker_state: "started"
provision_docker_inventory: "{{ inventory }}"
provision_docker_inventory_group: [net-dock]

Inventory file
`[net-docks]
docknode1 env=prod ansible_host=localhost ansible_port=2123
docknode2 env=prod ansible_host=localhost ansible_port=2123
docknode3 env=prod ansible_host=localhost ansible_port=2123

`I would like to point the variable to the inventroy file and the inventory_group variable to the group in the inventory file.

Allow assigning port mappings on docker containers

Crashes my Xserver

I probably just don't have my environment configured correctly, but when I run the ansible-playbook, which is identical to the example that is on the blog post. It starts running, brings up a docker host, and when it gets to the 'Wait for ssh' task, my Xserver appears to crash. I'm not really sure exactly, but I lose my X Desktop, and then promptly brought back to my Display manager login window.

After logging in, I can see that a docker image is running wtih docker ps, but I can't attach to it.

I have attached an ansible.log file of the playbook running.

ansible.txt

Any ideas where I should start looking? I did outside of ansible, execute a docker run on the chrismeyers/centos6 image and it starts and I can attach to it. I also can docker inspect the image, and ping the given IP address.

copying script on system with selinux reqires libselinux-python package

TASK [provision_docker : Copy inspect script to server] ************************
Friday 09 June 2017  10:52:50 +0300 (0:00:00.779)       0:00:00.865 *********** 
fatal: [localhost]: FAILED! => {"changed": false, "checksum": "eee1fdc007805abc8a5a583588da6162463342bc", "failed": true, "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"}
	to retry, use: --limit @/root/workspace/gate-ansible/ansible-playbooks/playbooks/rhv-deploy/tests/config.retry

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=1   

Role execution fails if not running under root user

TASK [provision_docker : Install libselinux-python package for 'copy' task] ******************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "This command has to be run under the root user."}

I have the libselinux package already installed on my machine, but even for checking if the package is installed it requires root, which is not expected behavior.

I tried to replace the package module with yum module which works on my system and it succeeds as expected, so I guess we can replace it here as well.

Affects only system where ansible_os_family== "Redhat"

Test Issue

xyz a test test test

impossible to test idempotency of a role

re-running the test that makes use of provision_docker would recreate a docker container making impossible to test idempotency of a role (or set of roles) over the same container
I have tried to set the item's restart attribute to false from the inventory, host_vars file and event as vars in the same playbook but the container would always be rebuilt (with a new Container ID) every time the "Bring up inventory group of hosts" task is executed

Feature - Make this role an Ansible module?

Should this role be a module? Maybe an action module ? ... pros/cons?

Pros

• Could more easily and generically handle the forwarding of "extra" docker parameters
• Could bundle the plugin into the provision_docker role namespace so that it continues to support - role: provision_docker

Cons

• If it breaks any backwards support

Inventory Description Design Decision

Currently, the provision_docker role exposes an interface much like an Ansible cloud module. The role variable provision_docker_inventory is used to express the desired set of docker containers to bring up. One alternative to this interface that I have toyed with is to use "dummy inventory" entries to express the set of docker containers that you want to bring up/operate on. For example:

# inventory file
[group1]
host1 image="chrismeyers/ubuntu12.04"
host2

[group2]
host1
host2
host3

The "dummy inventory" method allows the user to add more cleanly add hostvars. I say more cleanly because with the existing cloud-module-like interface you can still hadd hostvars. However, you have to do so in code using set_fact, which is not as clean.

With the cloud-module-like interface I've already added and exposed a groups parameter so the user doesn't have to add the host to a group in code. I don't want to go down the path of adding more one-off features like this when we can get it for free with the dummy-inventory method.

So the items up for discussion becomes, should we ...

• transition to a dummy-inventory interface?
• support both the dummy-inventory and cloud-module-like interface?

If we stick with the cloud-module-like interface, what features should we add ...

• parameter to specify arbitrary facts to attach to docker container/hosts
• other?

Cannot ping container

Hi,

Trying to follow https://www.ansible.com/blog/testing-ansible-roles-with-docker, I run into this issue:

TASK [provision_docker : Make sure ssh is really up] ***************************
task path: /home/rene/.ansible/roles/provision_docker/tasks/inc_cloud_iface.yml:50
<172.17.0.3> ESTABLISH SSH CONNECTION FOR USER: root
<172.17.0.3> SSH: EXEC sshpass -d13 ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/home/rene/.ansible/cp/ansible-ssh-%h-%p-%r 172.17.0.3 '/bin/sh -c '"'"'LANG=de_DE.UTF-8 LC_ALL=de_DE.UTF-8 LC_MESSAGES=de_DE.UTF-8 /usr/bin/python && sleep 0'"'"''
fatal: [localhost]: FAILED! => {"failed": true, "msg": "Using a SSH password instead of a key is not possible because Host Key checking is enabled and sshpass does not support this.  Please add this host's fingerprint to your known_hosts file to manage this host."}

Should we disable SSH host key checking? Or maybe it would be better to just include an SSH key for the purpose of running the tests and bake it into the containers?

Care about container-shutdown

Using this playbook:

- name: Bring up docker containers for docker connection inventory iface
  hosts: localhost
  roles:
    - role: provision_docker
      provision_docker_privileged: true,
      provision_docker_inventory_group: "{{ groups['robots'] }}"
      provision_docker_use_docker_connection: true

- name: Run tests for docker connection inventory iface
  hosts: robots
  gather_facts: false
  tasks:
    - name: "test command"
      command: cat /var/log/dmesg

and running it 2 times, i will stall on the second time, since the container is already started. You should debootstrap the containers, so remove (stop) them and all their volumes ( dangling? ) so the test can be repeatet

Error connecting: load_config() got an unexpected keyword argument 'config_dict'

When executing the tests with, I'm getting an error (debug log: load_config_error.txt)

I'm using docker-py 3.0.1. I also tried with 2.7.0 and 2.6.0 getting a different error in another file, but also related to authentication (debug log:
load_config_error_dockerpy27.txt).

If I'm using docker-py 3.0.1, but comment out the line reported to be faulty in the first paragraph, the same error as if using docker-py 2.7.0/2.6.0 appears.

I have no special authentication configuration and I'm not really educated in python :-) Looks like an issue of docker-py to me? Any Ideas?

Cheers,
Mario

Using CentOS 7 image

Hello, I am trying to use this ansible role with your CentOS 7 docker image, chrismeyers/centos7, and I am receiving some errors. It seems that sshing into the container is not working. I have tried the same configuration on your CentOS 6 image, chrismeyers/centos6, and it works properly. The errors I am getting are:

TASK [provision_docker : Make sure ssh is really up] *************************** failed: [localhost] (item={u'image': u'chrismeyers/centos7', u'name': u'test1'}) => {"item": {"image": "chrismeyers/centos7", "name": "test1"}, "msg": "Failed to connect to the host via ssh.", "unreachable": true} failed: [localhost] (item={u'image': u'chrismeyers/centos7', u'name': u'test2'}) => {"item": {"image": "chrismeyers/centos7", "name": "test2"}, "msg": "Failed to connect to the host via ssh.", "unreachable": true}

test: network_mode target causes problem to resolve fqdn names from containers

When executing network_mode test, it causes for future containers which are being created afterwards.

I hit this issue in PR #51 , the installation of nginx failed on error to resolve nginx.com.
When I execute my test (docker_ports) before executing network_mode everything works ok.

Another problem is that issue persist between runs, the easy fix is restart of docker service.

I think this side effect deserve some attention to be treated properly.

How to reproduce:

• move network_mode in test/Makefile to be executed before docker_ports target

all: cloud inventory docker_inventory docker_cloud groups docker_links docker_volumes_inventory docker_volumes_cloud network_mode docker_ports

• run make

$ make

Updating README

Hi,

I think the README could use a few updates:

• using the docker ansible_connection removes any need for ssh, (apart from the readme: IMO this should be used by default). this also removes any need for the whole custom routing part on mac os x.
• actually, the service module does already work atleast with the minimal sysvinit-utils debian:jessie ships with. I would assume that this is also the case for ubuntu:16.04, but this will need verification.
• additional documentation about the parameters the role accepts for a host should be added
• perhaps you'd like to link my role, from what I have seen, none of the other roles that are currently linked actually make use of separating services through different container types and linking these.
• the defaults should be documented (i.e. provision_docker_groups: ['docker_containers'] is really useful to know.)

stop containers on SIGINT

first of all: it's an awesome role to test roles!!
only thing I see - when pressing CTRL-C while running ansible the containers keep running in background
Not sure how and if ansible can handle signals and add hooks to them

Error: template error while templating string: unexpected '.'

When spinning up a test following the instructions I get this error:

TASK [provision_docker : Add docker hosts with connection docker] **************
task path: /Users/micah/code/ansible/provision_docker/tasks/inc_inventory_iface.yml:55
fatal: [localhost]: FAILED! => {
    "failed": true,
    "msg": "{'msg': u'All items completed', 'changed': False, 'results': [{'_ansible_parsed': True, '_ansible_item_result': True, u'end': u'2017-02-21 16:22:11.684872', '_ansible_no_log': False, u'stdout': u'172.17.0.2', u'cmd': [u'docker', u'inspect', u'--format', u'{{ .NetworkSettings.IPAddress }}', u'optimus'], u'rc': 0, 'invocation': {'module_name': u'command', u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'_raw_params': u\"docker inspect --format '{{ .NetworkSettings.IPAddress }}' optimus\", u'removes': None, u'creates': None, u'chdir': None}}, 'item': u'optimus', u'delta': u'0:00:00.060217', u'stderr': u'', u'changed': False, '_ansible_delegated_vars': {'ansible_host': u'localhost'}, 'stdout_lines': [u'172.17.0.2'], u'start': u'2017-02-21 16:22:11.624655', u'warnings': []}]}: template error while templating string: unexpected '.'. String: docker inspect --format '{{ .NetworkSettings.IPAddress }}' optimus"
}

Rerun ansible playbook

I have created a testing playbook which mimics my production playbook but start up docker containers first. If I want to test that I can rerun the playbook to check the state, the testing playbook will recreate the docker containers from fresh and thus it's not a rerun anymore.

Is there a way (like saying state: present) to only create the docker containers when they aren't there?

No package libselinux-python available (Fedora 31)

Running the provision_dockerrole gives me:

fatal: [localhost]: FAILED! => {"changed": false, "failures": ["No package libselinux-python available."], "msg": "Failed to install some of the specified packages", "rc": 1, "results": []}

This is because libselinux-python is no longer available; there is libselinux-python3 for python3installations.

Use `docker exec` rather than rely on SSH?

Seeing in the readme "provision_docker relies on being able to ssh to containers" and the klunky workaround for Macs running Docker made me wonder why provision_docker cannot use the docker exec command instead? That way it would only need the IP of the Docker host and not the IP of the individual containers to target a command on them?

Does this role work with docker-machine?

I'm running docker-machine on Mac OS X (El Capitan) and I'm seeing the following error when running my playbook:

TASK [chrismeyersfsu.provision_docker : Wait for ssh] **************************
failed: [localhost] => (item={u'image': u'ubuntu-upstart:14.04', u'name': u'statsd_host_1'}) => {"elapsed": 30, "failed": true, "item": {"image": "ubuntu-upstart:14.04", "name": "statsd_host_1"}, "msg": "Timeout when waiting for 172.17.0.2:22"}

Is this method deprecated in ansible 2.2?

I am fairly new to ansible, so i might really add FUD here, so just punch me.

Could it be that this playbook is kinda outdated due to ansible 2.2. introducing native docker features?

running the playbook i also get

[DEPRECATION WARNING]: docker is kept for backwards compatibility but usage is discouraged. The module documentation details page may explain more about this rationale..
This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.

But in general i run into issues like

failed: [localhost -> localhost] (item=localhost) => {"failed": true, "item": "localhost", "msg": "`docker-py` doesn't seem to be installed, but is required for the Ansible Docker module."}

where i am not really sure, that is an issue on the client or on the host running ansible ( osx / brew based installation of ansible 2.2