oguya/ansible-ckan

Ansible playbook to deploy CKAN

ansible-ckan's Issues

user provisioning is not compatible with ansible 2.1 security

In order for this to run under ansible 2.1 you either have to create ansible.cfg with the following content (Strongly advised against in Ansible docs):

[defaults]
allow_world_readable_tmpfiles = True

What is more in line with ansible best practices is to run ansible as remote user root using pub-key auth, and don't bother with setting up the provisioning user on the target machine
This also makes the param --ask-become-pass obsolete.

1. add to vars/all.yml
   ansible_user: root

2. from ckan.yml, db.yml & solr.yml remove the line user: provisioning

Add support for Travis CI

We can use travis CI to test these playbooks & even deploy CKAN!

Besides that, we should also test these playbooks on Ubuntu 12.04 since we actively developed & tested them on Ubuntu 14.04.

Use nip.io for ckan site name

Instead of using ckan.localhost.localdomain as the default ckan site name & adding it to /etc/hosts, we can use nip.io[1] wildcard DNS service which is clean, easier to provision & manage!

For instance: The CKAN site name 192.168.1.2.nip.io would map to 192.168.1.2.

1. http://nip.io/

Internal Server Error(code 500) returned by apache

Deploying ckan with vanilla configs on Ubuntu 12.04 installs & configures all the components successfully.
But when you access the default ckan URL - http://hostname/ you get a HTTP code 500 error from Apache.

• apache version

  root@ubuntu-VirtualBox:~# apachectl -v
  Server version: Apache/2.2.22 (Ubuntu)
  Server built:   Jul 24 2015 17:25:52
  

• OS type & version

  root@ubuntu-VirtualBox:~# lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 12.04.5 LTS
  Release:        12.04
  Codename:       precise
  root@ubuntu-VirtualBox:~# 
  

• apache error logs

  root@ubuntu-VirtualBox:~# cat /var/log/apache2/ckan.localhost.localdomain.error.log 
  [Thu Aug 13 09:39:47 2015] [crit] [client 172.26.23.23] configuration error:  couldn't perform authentication. AuthType not set!: /
  [Thu Aug 13 09:39:47 2015] [crit] [client 172.26.23.23] configuration error:  couldn't perform authentication. AuthType not set!: /favicon.ico, referer: http://172.26.23.225/
  [Thu Aug 13 09:41:33 2015] [crit] [client 172.26.23.225] configuration error:  couldn't perform authentication. AuthType not set!: /
  [Thu Aug 13 09:42:11 2015] [crit] [client 172.26.23.23] configuration error:  couldn't perform authentication. AuthType not set!: /
  [Thu Aug 13 09:42:11 2015] [crit] [client 172.26.23.23] configuration error:  couldn't perform authentication. AuthType not set!: /favicon.ico, referer: http://172.26.23.225/
  [Thu Aug 13 09:42:11 2015] [crit] [client 172.26.23.23] configuration error:  couldn't perform authentication. AuthType not set!: /
  [Thu Aug 13 09:42:11 2015] [crit] [client 172.26.23.23] configuration error:  couldn't perform authentication. AuthType not set!: /favicon.ico, referer: http://172.26.23.225/
  

Test CKAN 2.4

At the moment we are only deploying CKAN 2.3. CKAN 2.4[1] was released a few months ago, maybe we should test it!

1. http://docs.ckan.org/en/latest/changelog.html#v2-4-0-2015-07-22

Switch to Travis trusty beta build environment

It would be cool to test Travis' Trusty beta build environment just to see what breaks & what doesn't 😉

Only initialize CKAN database if necessary

Currently, the paste command to initialize the CKAN database is always run. As your comment above it already states the initialization should be idempotent, i.e. we should only initialize the database if that hasn't been done before. Your comment suggests touching a file to check this, however I think it's better to simply check whether the database contains any tables.

Here's the code I use to do this. My setup is different enough from yours that I'm currently writing my own playbook (that I might try to merge with yours once I figured it out), but you'll get the principal idea.

- name: Check if CKAN database has already been initialized
  command: psql -h {{ db_server }} -U {{ ckan_db_user }} -d {{ ckan_db_name }} -c '\d'
  environment:
    PGPASSWORD: "{{ ckan_db_password }}"
  register: ckan_tables_result

- name: Initialize CKAN database if necessary
  command: '{{ ckan_virtualenv_path }}/bin/paster --plugin=ckan db init -c "{{ ckan_config_path }}/production.ini"'
  when: ckan_tables_result.stdout_lines|length <= 1

Basically I use psql to execute the command \d (list all tables) in the CKAN database. If that returns more than one line then the database is already initialized (psql prints a single result line if the database is empty).

Ansible error when starting/enabling solr service

Ansible returns an error & terminates playbook execution when attempting to start & enable solr service on a clean Ubuntu 15.04 systemd container.

• Error returned by ansible

  james@vast ansible-ckan (master) $ ansible-playbook solr.yml -i tests/hosts -K
  ...
  TASK: [solr | Start & enable service] ***************************************** 
  failed: [localhost] => {"failed": true}
  msg: Error when trying to enable solr: rc=1 Synchronizing state for solr.service with sysvinit using update-rc.d...
  Executing /usr/sbin/update-rc.d solr defaults
  Executing /usr/sbin/update-rc.d solr enable
  Failed to execute operation: No such file or directory
  
  
  FATAL: all hosts have already failed -- aborting
  
  PLAY RECAP ******************************************************************** 
             to retry, use: --limit @/home/james/solr.retry
  
  localhost                  : ok=14   changed=13   unreachable=0    failed=1   
  

• container details

  root@vast:~# uname -a
  Linux vast 4.1.5-200.fc22.x86_64 #1 SMP Mon Aug 10 23:38:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
  root@vast:~# lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 15.04
  Release:        15.04
  Codename:       vivid
  

Unable to initialize ckan postgresql db

Using paster with plugin option doesn't work in playbook & on shell:

- name: Initialize ckan postgresql database
  sudo_user: "{{ckan_user.name}}"
  command: "{{ ckan_virtualenv }}/bin/paster --plugin=ckan db init -c {{ ckan_conf_dir }}/production.ini"

It produces the following error:

TASK: [ckan | Initialize ckan postgresql database] **************************** 
failed: [ckan01] => {"changed": true, "cmd": ["/usr/lib/ckan/ckan.localhost.localdomain/bin/paster", "--plugin=ckan", "db", "init", "-c", "/etc/ckan/ckan.localhost.localdomain/production.ini"], "delta": "0:00:02.041566", "end": "2015-07-14 14:39:22.889204", "rc": 1, "start": "2015-07-14 14:39:20.847638", "warnings": []}
stderr: Traceback (most recent call last):
  File "/usr/lib/ckan/ckan.localhost.localdomain/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/ckan.localhost.localdomain/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/ckan.localhost.localdomain/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/ckan.localhost.localdomain/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/lib/cli.py", line 208, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/lib/cli.py", line 148, in _load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/config/environment.py", line 232, in load_environment
    p.load_all(config)
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/plugins/core.py", line 124, in load_all
    unload_all()
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/plugins/core.py", line 181, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/plugins/core.py", line 209, in unload
    plugins_update()
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/plugins/core.py", line 116, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/config/environment.py", line 270, in update_config
    search.check_solr_schema_version()
  File "/usr/lib/ckan/ckan.localhost.localdomain/src/ckan/ckan/lib/search/__init__.py", line 306, in check_solr_schema_version
    % (version, ', '.join(SUPPORTED_SCHEMA_VERSIONS)))
ckan.lib.search.common.SearchError: SOLR schema version not supported: 1.5. Supported versions are [2.3]

FATAL: all hosts have already failed -- aborting

Group hosts file according to roles

By default, this playbook assumes that you're installing CKAN & its dependencies(solr, postgresql, nginx & apache) on one host - which might not be the case in certain production environments.

Therefore, we can break the hosts file into several groups each representing a specific role & then add a top-level play for each role. I think this allows you to deploy a specific role to a specific host.

In cases where you're deploying all the roles to a single host, you can have the same host on multiple groups in the hosts file.

Port Playbooks to Ansible 2.0

There are couple of warnings that I've received when running these playbooks using ansible 2.0.

• Use synchronize module rather than rsync

  TASK [solr : Copy solr.xml config. files] **************************************
  changed: [localhost]
  [WARNING]: Consider using synchronize module rather than running rsync
  

• Remove bare variables—e.g. solr_cores—use full variable syntax instead—e.g. '{{solr_cores}}'

  TASK [solr : Create Solr core data dir] ****************************************
  [DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks
  so that the environment value uses the full variable syntax ('{{solr_cores}}').
  This feature will be removed in a future release. Deprecation warnings can be 
  disabled by setting deprecation_warnings=False in ansible.cfg.
  changed: [localhost] => (item=ckan_default)
  

Have a look at Ansible 2.0 porting guide

Download & Install postgresql from PGDG repo

PostgreSQL Global Development Group (PGDG) maintains an apt repo[1][2] of PostgreSQL packages for Debian and Ubuntu which we can use to download & install postgresql instead of depending on the main distro repo.

Besides that, using pgdg repo allows us to install newer/specific versions of postgresql on any debian badsed distro.

1. http://apt.postgresql.org/pub/repos/apt/
2. https://wiki.postgresql.org/wiki/Apt

Add tags & hosts distro checks

This playbook was actively tested on Ubuntu 14.04 and I'm not sure if it will work different distros. Therefore, we need to add checks to limit these playbooks to only ubuntu 14.04 hosts.
With time, we'll test it on other versions of Ubuntu & maybe other distros.

We also need to add tags to tasks & or roles.