esgf/esgf-ansible

A repository to hold Ansible inventory, playbooks, roles and tasks that specify the details of ESGF Node deployments.

esgf-ansible's Issues

upgrading data node only but care about idx node

Describe the bug
Trying to upgrade existing data node, but esgf-ansible failed when 'Create vesgdev-idx.ipsl.upmc.fr Cert Own CA'

This issue only differs from issue#47 by the flags -tags data --limit vesgdev-data.ipsl.upmc.fr.
Esgf-ansible try to create a certificate for the index node that is not supposed to be upgraded.

Full log:
upgrade_data_only_from_idx_without_cert_paths.log

TASK [certificate : Create vesgdev-idx.ipsl.upmc.fr Cert Own CA] ******************************************
task path: /root/tmp/esgf-ansible/roles/certificate/tasks/cert.yml:47
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1552321721.44-143578833281545/AnsiballZ_openssl_certificate.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1552321721.44-143578833281545/AnsiballZ_openssl_certificate.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1552321721.44-143578833281545/AnsiballZ_openssl_certificate.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 1091, in <module>
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 1070, in main
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 617, in generate
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 487, in check
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 469, in _validate_csr
IndexError: list index out of range
fatal: [vesgdev-data.ipsl.upmc.fr]: FAILED! => {
    "changed": false, 
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1552321721.44-143578833281545/AnsiballZ_openssl_certificate.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1552321721.44-143578833281545/AnsiballZ_openssl_certificate.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1552321721.44-143578833281545/AnsiballZ_openssl_certificate.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 1091, in <module>
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 1070, in main
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 617, in generate
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 487, in check
  File "/tmp/ansible_openssl_certificate_payload_vpLzT2/__main__.py", line 469, in _validate_csr
IndexError: list index out of range


PLAY RECAP ************************************************************************************************
vesgdev-data.ipsl.upmc.fr  : ok=105  changed=56   unreachable=0    failed=1

To Reproduce

## Conda env installation

(Installation de pip (pour python 2.6) => pip search ansible crashes.)

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

wget https://repo.anaconda.com/miniconda/Miniconda2-latest-Linux-x86_64.sh

chmod +x Miniconda2-latest-Linux-x86_64.sh
./Miniconda2-latest-Linux-x86_64.sh

source ${HOME}/.bashrc

conda create -y -n ansible python=2.7

conda activate ansible

pip install ansible==2.7

## esgf-ansible repo

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

TAG_NAME='4.0.0-beta1'
git clone https://github.com/ESGF/esgf-ansible.git && cd esgf-ansible && git checkout $TAG_NAME

## Inventory

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/inventory.txt" <<EOF
[data]
vesgdev-data.ipsl.upmc.fr   ansible_connection=local

[index]
vesgdev-idx.ipsl.upmc.fr    ansible_connection=local

[idp]
vesgdev-idx.ipsl.upmc.fr   ansible_connection=local
EOF

## Host vars for vesgdev-data.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-data.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false
EOF








## Host vars for vesgdev-idx.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-idx.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false
EOF


## Preprocessing 

backup :
- certificate https
- certificate gridftp

* To do for vesgdev-idx

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

* To do for vesgdev-data

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

## Upgrade

export PARENT_DIR='/root/tmp'
script ${PARENT_DIR}/esgf-ansible/upgrade.log
cd ${PARENT_DIR}/esgf-ansible
source activate ansible
export ANSIBLE_NOCOLOR=true
ansible-playbook -vvv -i ${PARENT_DIR}/esgf-ansible/inventory.txt -u root --tags data --limit vesgdev-data.ipsl.upmc.fr install.yml

Expected behavior

upgrading done

ESGF Node (please complete the following information):

• Managed OS: CentOS 6 (packages updated)

• Host OS: CentOS 6 (from vesgdev-idx our 'all' node)

• esgf-ansible version: 4.0.0-beta1

• ansible version: 2.7.8

• Node type:

VM #1: vesgdev-idx, an 'all' esgf node that serves as index and idp node. version v2.8.1-master-release
VM #2: vesgdev-data, an data esgf node. version v2.8.1-master-release

Additional context

Upgrade java to 8-202

Describe the solution you'd like
Java 8 202 was released Jan 15. Upgrade the installer to deploy this latest version.
https://java.com/en/download/faq/release_dates.xml

Bump version of urllib3

Expected Behavior

Need to use urllib3 version higher than 1.24.2.
Check https://nvd.nist.gov/vuln/detail/CVE-2019-11236

Actual Behavior

Uses 1.24.1

Steps to Reproduce the Problem

Specifications

• Version:
• Platform:
• Subsystem:

When installing from scratch, /esg/gridftp_root/etc/grid-security/certificates/ isn’t created if non-existent

While installing from scratch, if you don’t create the directory /esg/gridftp_root/etc/grid-security/certificates/, install.yml fails with this error :

TASK [globus_certs : Copy certificates over if it is] **********************************************************************************************************************************************************
changed: [vesg4cds-data.ipsl.upmc.fr]
fatal: [vesg4cds-idx.ipsl.upmc.fr]: FAILED! => {
    "changed": true,
    "cmd": "cp /etc/grid-security/certificates/* /esg/gridftp_root/etc/grid-security/certificates/",
    "delta": "0:00:00.005728",
    "end": "2019-09-16 15:19:56.689950",
    "rc": 1,
    "start": "2019-09-16 15:19:56.684222"
}

STDERR:

cp: la cible « /esg/gridftp_root/etc/grid-security/certificates/ » n'est pas un répertoire

To Reproduce
Steps to reproduce the behavior :

1. Run installer with options --skip-tags gridftp
2. See error

ESGF Node (please complete the following information):

• Managed OS: CentOS 6 & 7
• Host OS: CentOS 7
• esgf-ansible Version: 4.0.4
• Node type: Index

Additional context
Possible workaround : Using the command mkdir -p /esg/gridftp_root/etc/grid-security/certificates/, then launching the playbook install.yml again, solves the problem.

PostgreSQL has to run before upgrading legacy installation

Describe the bug

Legacy installation makes esgf node starts at boot time, with the command esg-node.
So as to rely only on the start/stop recipes of esgf-ansible, we have to disable this feature.
But disable this feature, reboot and than upgrade the legacy installation makes the upgrade to fail with this message:

TASK [base : Create db super user] ******************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket Unix « /tmp/.s.PGSQL.5432 » ?
[WARNING]: Module remote_tmp /var/lib/pgsql/.ansible/tmp did not exist and was created with a mode of 0700,
this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
fatal: [vesgint-data.ipsl.upmc.fr]: FAILED! => {
"changed": false
}

MSG:

unable to connect to database: n'a pas pu se connecter au serveur : Aucun fichier ou dossier de ce type
Le serveur est-il actif localement et accepte-t-il les connexions sur la
socket Unix « /tmp/.s.PGSQL.5432 » ?

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket Unix « /tmp/.s.PGSQL.5432 » ?
fatal: [vesgint-idx.ipsl.upmc.fr]: FAILED! => {
"changed": false
}

MSG:

unable to connect to database: n'a pas pu se connecter au serveur : Aucun fichier ou dossier de ce type
Le serveur est-il actif localement et accepte-t-il les connexions sur la
socket Unix « /tmp/.s.PGSQL.5432 » ?

To Reproduce
Steps to reproduce the behavior:

1. Disable esg-node start at boot time:

chmod -x /etc/init.d/zesgf
chmod -x /etc/init.d/esgf-httpd

2. reboot the machine

3. Upgrade a legacy installation

Expected behavior

esgf-ansible starts postgreSQL but it must start postgreSQL with the good configuration.

ESGF Node (please complete the following information):

• Managed OS: CentOS 6
• Host OS: CentOS 7
• esgf-ansible Version: v4.0.2
• Node type: a set of idp, index and data nodes

Additional context
legacy installation

Stop all the services with one tag

Is your feature request related to a problem? Please describe.

By default, the recipe stop.yml doesn't stop httpd, postgresl and monitoring. If I would like to shut down these services, I have to add option '--tasks never'. But adding this option, prevents to shut down the other services of ESGF (tomcat, cog, etc.) when managing a set of data, idp and index nodes. The solution is quite straightforward : add --tags "data,idp,index,never" to shut down all the services of a set of data, idp and index nodes. But esgf-ansible may implement new services in the future and I don't want to miss to stop these services.

Describe the solution you'd like

So I would like a tag that makes all the services to stop, whenever esgf-ansible will add new services.

Describe alternatives you've considered

Any other mechanism that aims the same goal.
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Upgrade Tomcat to 8.5.81

Need to upgrade

reboot of the node when selinux is not set to disabled

Describe the bug

While upgrading ESGF nodes, esgf-ansible (4.0.0-beta1) has rebooted my data node during the task "selinux : Disable SELinux" because the SELinux policy of the node was set to "permissive" (which is not compliant with the ESGF settings spec, I admit). Following the FAQ page about the upgrade of existing nodes, ESGF services must be shutdown before starting esgf-ansible. But rebooting the node while running esgf-ansible, restarts the ESGF services (and most of the time, some of the services are not restarted - not deterministic -). This is due to the ansible task configuration:

TASK [selinux : Disable SELinux] 

...

    "policy": "targeted", 
    "reboot_required": true, 
    "state": "disabled"

    Config SELinux state changed from 'permissive' to 'disabled'
    [selinux : Pause for reboot confirmation]

To Reproduce
Steps to reproduce the behavior:

Set the SELinux to "permissive" (SELINUX=permissive in the /etc/selinux/config), reboot, stop ESGF services and start upgrading with esgf-ansible.

Expected behavior

I think esgf-ansible better terminate when dealing with machines with SELinux set to other values than "disabled", and output a clear message that SELinux must be manually set to "disabled".
Don't reboot the machine automatically.

ESGF Node (please complete the following information):

• Managed OS: CentOS 6
• Host OS: CentOS 6
• esgf-ansible Version: 4.0.0-beta1
• Node type: all

Additional context
Full context is described in the issue #48.

inconsistent webapps after upgrading legacy installation

Describe the bug

After a successful upgrade of a legacy installation, orp, idp, esgf-search and thredds webapps don't start. catalina.out shows some inconsistency like:

• Orp & idp
  Configuration problem: You cannot use a spring-security-2.0.xsd or spring-security-3.0.xsd or spring-security-3.1.xsd schema or spring-security-3.2.xsd schema with Spring Security 4.0. Please update your schema declarations to the 4.0 schema.
  Offending resource: class path resource [esg/orp/orp/config/security-context-auth.xml]
• thredds
  org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/thredds]]

Caused by: java.lang.NoSuchMethodError: org.apache.logging.log4j.util.LoaderUtil.isClassAvailable(Ljava/lang/String;)

To Reproduce
Steps to reproduce the behavior:

Upgrade a v2.6.7-master-release node with esgf-ansible 4.0.2

ESGF Node (please complete the following information):

• Managed OS: CentOS 6
• Host OS: CentOS 7
• esgf-ansible Version: v4.0.2
• Node type: upgrading idp, index and data nodes

Workaround

Simply delete the webapps before upgrading a legacy installation and probably esgf-ansi should always do that.

rm -fr /usr/local/tomcat/webapps/*
rm -fr /usr/local/src

esgf-config repo still present

It appears that if installation fails at some point, a clone of the esgf-config repo is left in /tmp with modifications that causes a subsequent clone to fail:

task path: /Users/ames4/git-repos/esgf-ansible/roles/base/tasks/config.yml:2
fatal: [esgf-test-data.llnl.gov]: FAILED! => {
    "before": "c62d6d576374263a375bf1b1436527cb7b618052",
    "changed": false
}

MSG:

Local modifications exist in repository (force=no).


PLAY RECAP *********************************************************************
esgf-test-data.llnl.gov    : ok=78   changed=7    unreachable=0    failed=1

This might be hard to reproduce. If the repo isn't need perhaps the best option is to delete it so nothing is left behind

I had used devel with the most recent commit: ff088ae

Create Conda environments from environment files

Conda can export the setting of an environment so that the environment can be reproduced. I've exported the conda environments that got created on an all node type that was installed on cwt-node and will try to integrate the use of the environment file into the ansible script.

Using CI tests with travis

It would be nice to have continuous-integration tests (on Travis or others) to see if the default deployment is still working.

Here are some examples how to use Travis CI for Ansible playbook testing with docker containers:

• https://github.com/geerlingguy/ansible-role-nginx/blob/master/.travis.yml
• https://github.com/ANXS/postgresql/blob/master/.travis.yml

The nginx role is using molecule:
https://pypi.org/project/molecule/

Drop AccessLogging filter, etc.

Is your feature request related to a problem? Please describe.
The AccessLogging filter has shown to block THREDDS when a server is under a heavy volume of requests. Many sites now

Describe the solution you'd like
remove the schema and filter entries. no need to load the filter jar etc.

While we are at it observed that the esgf_security schema is being deployed in data and this is not correct.

Correct perms on myproxy-ca

The apache user must be able to read the myproxy-ca keypair in order for SLCS to issue certs.
The best solution is to ensure that the the files are group readable by the apache user but still can be owned by root for rw.

esgf-ansible using out-of-date trust sources

The following files which are part of the esgf-ansible release, are out-of-date and need to be urgently replaced with the latest versions available.

1. esg_trusted_certificates.tar
2. esg-truststore.ts
3. esgf-ca-bundle.crt

The versions currently being served have several certificates which are already expired, and don't contain the replacements to these expired certificates. This causes globus transfers to fail for sites using certificates issued by the CAs whose certificates are no longer current in the ESGF trust served by esgf-ansible.
The updated files to replace these are found here:
https://github.com/ESGF/esgf-dist/tree/master/installer/certs

missing ESGF_VERSION setting

Following an installation with index, the cog_settings.cfg file is missing the ESGF_VERSION
setting. This has been added by the bash scripts in the past.

Private keys / certs on local vs remote machines

I have generated a CSR first time running the installer. I got it signed, great, but what to do now? Looks like I need to copy the private key back to my local machine in order to specify in the hosts variables file. In order to do that I need to make it readable by non-root user. Maybe thats ok...but not great practice for private keys....

How about instead of single variables we have local vs remote - we should be clear: "local" is the client, ie. your laptop?

gftphostcert_local: /home/ames4/esgf-data-node.pem
gftphostkey_remote: /root/gridftp.key

In this example we mix and match a remote key (where it was generated, and a .key extension might be helpful) with a signed cert (from the CSR) on my local system. While Prashanth has been putting CSRs responses on his website for download (with wget) may not be a universal practice, so remote or local options might be handy. For instance our web-certs are received by email, but private keys remain on the remote system.

Comparison Table

Create a table that compares ESGF v2 vs v4 feature by feature.

Add playbook to set up a Prometheus query crontab

We are integrating node status information into CoG and need the backend to support that.
(1) Playbook should either fetch the repo or create an environment that runs the "query_prom.py" CLI
(2) Add a crontab to call a shell script that can activate the environment and call the python script with proper items. Need to agree on the frequency (how many minutes) and set cron accordingly

Install trusted certificates for Globus: FAIL

Describe the bug
The following failed: fatal: [esgf-test-data.llnl.gov]: FAILED! => {
"changed": true,
"cmd": "cp /tmp/esg_trusted_certificates/* /etc/grid-security/certificates",
"delta": "0:00:00.006249",
"end": "2019-03-25 11:02:05.553235",
"rc": 1,
"start": "2019-03-25 11:02:05.546986"
}

STDERR:

cp: cannot stat `/tmp/esg_trusted_certificates/*': No such file or directory

MSG:

non-zero return code

Merged from devel branch latest commit: d89c643

Upgrade tomcat version

Expected Behavior

Use tomcat 8.5.40 or better, as older versions are flagged by CVE-2019-10072

Actual Behavior

Uses tomcat 8.5.39

Steps to Reproduce the Problem

Specifications

• Version:
• Platform:
• Subsystem:

upgrading with cert paths

Describe the bug
Trying to upgrade existing data, index and idp nodes, but esgf-ansible failed when 'Create vesgdev-idx.ipsl.upmc.fr Cert Own CA'

The difference between issue#47 and this issue is that the certificate paths are provided in the host_vars files.

Full log:
upgrade_with_cert_paths.log

TASK [certificate : Create vesgdev-idx.ipsl.upmc.fr Cert Own CA] ******************************************
task path: /root/tmp/esgf-ansible/roles/certificate/tasks/cert.yml:47
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1552319967.28-83731132508851/AnsiballZ_openssl_certificate.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1552319967.28-83731132508851/AnsiballZ_openssl_certificate.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1552319967.28-83731132508851/AnsiballZ_openssl_certificate.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 1091, in <module>
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 1070, in main
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 617, in generate
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 487, in check
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 469, in _validate_csr
IndexError: list index out of range
fatal: [vesgdev-idx.ipsl.upmc.fr]: FAILED! => {
    "changed": false, 
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1552319967.28-83731132508851/AnsiballZ_openssl_certificate.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1552319967.28-83731132508851/AnsiballZ_openssl_certificate.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1552319967.28-83731132508851/AnsiballZ_openssl_certificate.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 1091, in <module>
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 1070, in main
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 617, in generate
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 487, in check
  File "/tmp/ansible_openssl_certificate_payload_TOGEfY/__main__.py", line 469, in _validate_csr
IndexError: list index out of range


PLAY RECAP ************************************************************************************************
vesgdev-data.ipsl.upmc.fr  : ok=28   changed=8    unreachable=0    failed=1   
vesgdev-idx.ipsl.upmc.fr   : ok=100  changed=48   unreachable=0    failed=1

To Reproduce

## Conda env installation

(Installation de pip (pour python 2.6) => pip search ansible crashes.)

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

wget https://repo.anaconda.com/miniconda/Miniconda2-latest-Linux-x86_64.sh

chmod +x Miniconda2-latest-Linux-x86_64.sh
./Miniconda2-latest-Linux-x86_64.sh

source ${HOME}/.bashrc

conda create -y -n ansible python=2.7

conda activate ansible

pip install ansible==2.7

## esgf-ansible repo

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

TAG_NAME='4.0.0-beta1'
git clone https://github.com/ESGF/esgf-ansible.git && cd esgf-ansible && git checkout $TAG_NAME

## Inventory

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/inventory.txt" <<EOF
[data]
vesgdev-data.ipsl.upmc.fr   ansible_connection=local

[index]
vesgdev-idx.ipsl.upmc.fr    ansible_connection=local

[idp]
vesgdev-idx.ipsl.upmc.fr   ansible_connection=local
EOF

## Host vars for vesgdev-data.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-data.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false

gftphostcert: /etc/grid-security.bak/hostcert.pem
gftphostkey: /etc/grid-security.bak/hostkey.pem

hostkey_src: /etc/certs.bak/hostkey.pem
hostcert_src: /etc/certs.bak/hostcert.pem
cachain_src: /etc/certs.bak/cachain.pem

EOF








## Host vars for vesgdev-idx.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-idx.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false

myproxycacert: /var/lib/globus/simple_ca/cacert.pem
myproxycakey: /var/lib/globus/simple_ca/private/cakey.pem
myproxy_signing_policy: /var/lib/globus/simple_ca/signing-policy

hostkey_src: /etc/certs.bak/hostkey.pem
hostcert_src: /etc/certs.bak/hostcert.pem
cachain_src: /etc/certs.bak/cachain.pem

EOF


## Preprocessing 

backup :
- certificate https
- certificate gridftp

* To do for vesgdev-idx

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

* To do for vesgdev-data

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

## Upgrade

export PARENT_DIR='/root/tmp'
script ${PARENT_DIR}/esgf-ansible/upgrade.log
cd ${PARENT_DIR}/esgf-ansible
source activate ansible
export ANSIBLE_NOCOLOR=true
ansible-playbook -vvv -i ${PARENT_DIR}/esgf-ansible/inventory.txt -u root install.yml

Expected behavior

upgrading done

ESGF Node (please complete the following information):

• Managed OS: CentOS 6 (packages updated)

• Host OS: CentOS 6 (from vesgdev-idx our 'all' node)

• esgf-ansible version: 4.0.0-beta1

• ansible version: 2.7.8

• Node type:

VM #1: vesgdev-idx, an 'all' esgf node that serves as index and idp node. version v2.8.1-master-release
VM #2: vesgdev-data, an data esgf node. version v2.8.1-master-release

Additional context

Python 3 publisher setup and environment

update the publisher environment for new versions using python3
Depends on pushing dependent packages to pypi.org

Let's Encrypt while upgrading nodes

Describe the bug

When trying to upgrade from 4.04 to devel or 4.05, with Let's Encrypt on.

`TASK [tomcat : Create Keystore] **************************************************************************************************************************************************************
fatal: [vesgint-idx.ipsl.upmc.fr]: FAILED! => {
"changed": false,
"cmd": "/usr/bin/openssl pkcs12 -export -name my_esgf_node -in /tmp/my_esgf_node.crt -inkey /tmp/my_esgf_node.key -out /tmp/keystore.p12 -passout '********'",
"rc": 1
}

STDERR:

No certificate matches private key

MSG:

No certificate matches private key

fatal: [vesgint-data.ipsl.upmc.fr]: FAILED! => {
"changed": false,
"cmd": "/usr/bin/openssl pkcs12 -export -name my_esgf_node -in /tmp/my_esgf_node.crt -inkey /tmp/my_esgf_node.key -out /tmp/keystore.p12 -passout '********'",
"rc": 1
}

STDERR:

No certificate matches private key

MSG:

No certificate matches private key`

Full log: 6_upgrade_int_to_devel_lets.log

To Reproduce

ansible-playbook -i hosts.int -u root install.yml

idx&idp config:

`ansible_user: root

globushostcert: /root/certs/local_certs/hostcert.pem
globushostkey: /root/certs/local_certs/hostkey.pem

myproxycacert: /root/certs/local_certs/cacert.pem
myproxycakey: /root/certs/local_certs/cakey.pem
myproxy_signing_policy: /root/certs/local_certs/globus_simple_ca_47671b99_setup-0/47671b99.signing_policy

try_letsencrypt: true

globus_user: [NOT SHOWN]
globus_pass: [NOT SHOWN]
register_gridftp: false
register_myproxy: false

configure_centos6_iptables: false
configure_centos7_firewalld: false

mirror_host: distrib-coffee.ipsl.jussieu.fr/pub`

data config:

`ansible_user: root

globushostcert: /root/certs/local_certs/hostcert.pem
globushostkey: /root/certs/local_certs/hostkey.pem

try_letsencrypt: true

globus_user: [NOT SHOWN]
globus_pass: [NOT SHOWN]
register_gridftp: false
register_myproxy: false

configure_centos6_iptables: false
configure_centos7_firewalld: false

mirror_host: distrib-coffee.ipsl.jussieu.fr/pub`

Expected behavior

Installation to complete and working Let's Encrypt certificats and idx and data nodes.

ESGF Node (please complete the following information):

• Managed OS: Up to date CentOS 7
• Host OS: Up to date CentOS 7
• esgf-ansible Version: 4.04
• Node type: idx, idp and data

sqlparse-0.4.1 does not work with schema_migrate of esgf-security

Describe the bug
TASK [schema_migrate : Easy Install schema migration script esgf_security_initialize] failed with sqlparse-0.4.1.

To Reproduce
Steps to reproduce the behavior:
$ sudo sh -c "source /usr/local/conda/bin/activate schema-migrate && easy_install /tmp/esgf_security-0.1.7-py2.7.egg"
WARNING: The easy_install command is deprecated and will be removed in a future version.
Processing esgf_security-0.1.7-py2.7.egg
removing '/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/esgf_security-0.1.7-py2.7.egg' (and everything under it)
creating /usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/esgf_security-0.1.7-py2.7.egg
Extracting esgf_security-0.1.7-py2.7.egg to /usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages
esgf-security 0.1.7 is already the active version in easy-install.pth
Installing esgf_security_initialize script to /usr/local/conda/envs/schema-migrate/bin
Installed /usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/esgf_security-0.1.7-py2.7.egg
Processing dependencies for esgf-security==0.1.7
Searching for sqlparse
Reading https://pypi.org/simple/sqlparse/
Downloading https://files.pythonhosted.org/packages/a2/54/da10f9a0235681179144a5ca02147428f955745e9393f859dec8d0d05b41/sqlparse-0.4.1.tar.gz#sha256=0f91fd2e829c44362cbcfab3e9ae12e22badaa8a29ad5ff599f9ec109f0454e8
Best match: sqlparse 0.4.1
Processing sqlparse-0.4.1.tar.gz
Writing /tmp/easy_install-6ZGABz/sqlparse-0.4.1/setup.cfg
Running sqlparse-0.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6ZGABz/sqlparse-0.4.1/egg-dist-tmp-KfKl7V
Traceback (most recent call last):
File "/usr/local/conda/envs/schema-migrate/bin/easy_install", line 11, in
sys.exit(main())
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 2321, in main
**kw
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/init.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 424, in run
self.easy_install(spec, not self.no_deps)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 666, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 713, in install_item
self.process_distribution(spec, dist, deps)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 758, in process_distribution
[requirement], self.local_index, self.easy_install
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/pkg_resources/init.py", line 782, in resolve
replace_conflicting=replace_conflicting
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/pkg_resources/init.py", line 1065, in best_match
return self.obtain(req, installer)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/pkg_resources/init.py", line 1077, in obtain
return installer(requirement)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 685, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 711, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 896, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1164, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1150, in run_setup
run_setup(setup_script, args)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
raise
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
saved_exc.resume()
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
six.reraise(type, exc, self._tb)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-6ZGABz/sqlparse-0.4.1/setup.py", line 12, in
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/init.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/distutils/core.py", line 124, in setup
dist.parse_config_files()
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/dist.py", line 702, in parse_config_files
ignore_option_errors=ignore_option_errors)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/config.py", line 121, in parse_configuration
meta.parse()
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/config.py", line 426, in parse
section_parser_method(section_options)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/config.py", line 399, in parse_section
self[name] = value
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/config.py", line 184, in setitem
value = parser(value)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/config.py", line 515, in _parse_version
version = self._parse_attr(value, self.package_dir)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/site-packages/setuptools/config.py", line 349, in _parse_attr
module = import_module(module_name)
File "/usr/local/conda/envs/schema-migrate/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/tmp/easy_install-6ZGABz/sqlparse-0.4.1/sqlparse/init.py", line 11, in
File "/tmp/easy_install-6ZGABz/sqlparse-0.4.1/sqlparse/sql.py", line 192
.format(**locals()), file=f)
^
SyntaxError: invalid syntax
Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

ESGF Node (please complete the following information):

• Managed OS: RedHat7
• Host OS:
• esgf-ansible Version: [e.g. v4.0.0]
• Node type: [e.g. data-only with own IDP]

Additional context
Add any other context about the problem here.

Lets encrypt certificate retrieval fails (support of ACME v1 is finished)

Describe the bug
I was using Lets encrypt web certificate for data node. Since the certificate in our node is expired, so I tried to renew the certificate through ESGF ansible. But it failed with the following message.

TASK [httpd : Create ACME Challenge] ************************************************************************************************
fatal: [dist.nmlab.snu.ac.kr]: FAILED! => {
"changed": false,
"other": {}
}

MSG:

Error registering: 403 {u'status': 403, u'type': u'urn:acme:error:unauthorized', u'detail': u'Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.'}

---

It seems like ACME client using in ESGF ansible does not support ACMEv2. I tried updating ESGF ansible to the latest version but it did not help.

To Reproduce
Steps to reproduce the behavior:

ansible-playbook -v -i hosts.test --tags data --limit dist.nmlab.snu.ac.kr install.yml
OR
ansible-playbook -v -i hosts.test --tags data --limit dist.nmlab.snu.ac.kr web_certs.yml

ESGF Node (please complete the following information):

• Managed OS: [CentOS 6]
• Host OS: [CentOS6]
• esgf-ansible Version: [v4.0.4]
• Node type: [data]

Using Vagrant for local testing and development

Vagrant setups a new local VM to be used for ansible deployment. See example:
https://github.com/ANXS/postgresql

It would be nice to have a default configuration which works with the Vagrant settings.

temporary files and directories (/tmp/esgf-config)

Describe the bug

While upgrading an idp, an index and a data node from version 4.0.2 to 4.0.3 on machines that have never been rebooted since the last update (to version 4.0.2), esgf-ansible fails to upgrade because /tmp/esgf-config exists and is not empty. Full log of the upgrade is attached. Deleting this directory fixes the problem.
upgrade.log

To Reproduce
Steps to reproduce the behavior:

1. Run installer with options:
   INVENTORY var contains the informations for an idp, an index and a data node.
   ansible-playbook -i ${INVENTORY} -u root --skip-tags gridftp install.yml

2. See error

TASK [base : Create config directory] *****************************************************************************************************************************************************************************
ok: [vesgint-data.ipsl.upmc.fr]
ok: [vesgint-idx.ipsl.upmc.fr]
/home/esgf-watch-dog/miniconda2/envs/ansible/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.ecdsa_curve.curve_class(), pointinfo
/home/esgf-watch-dog/miniconda2/envs/ansible/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.ecdsa_curve.curve_class(), pointinfo

TASK [base : Install properties] **********************************************************************************************************************************************************************************
ok: [vesgint-data.ipsl.upmc.fr]
ok: [vesgint-idx.ipsl.upmc.fr]
/home/esgf-watch-dog/miniconda2/envs/ansible/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.ecdsa_curve.curve_class(), pointinfo
/home/esgf-watch-dog/miniconda2/envs/ansible/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.ecdsa_curve.curve_class(), pointinfo

TASK [base : Install config_type] *********************************************************************************************************************************************************************************
ok: [vesgint-data.ipsl.upmc.fr]
ok: [vesgint-idx.ipsl.upmc.fr]
/home/esgf-watch-dog/miniconda2/envs/ansible/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.ecdsa_curve.curve_class(), pointinfo
/home/esgf-watch-dog/miniconda2/envs/ansible/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.ecdsa_curve.curve_class(), pointinfo

TASK [base : Clone ESGF config repository] ************************************************************************************************************************************************************************
fatal: [vesgint-data.ipsl.upmc.fr]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/git clone --origin origin https://github.com/ESGF/config.git /tmp/esgf-config",
    "rc": 128
}

STDERR:

fatal: destination path '/tmp/esgf-config' already exists and is not an empty directory.



MSG:

fatal: destination path '/tmp/esgf-config' already exists and is not an empty directory.
fatal: [vesgint-idx.ipsl.upmc.fr]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/git clone --origin origin https://github.com/ESGF/config.git /tmp/esgf-config",
    "rc": 128
}

STDERR:

fatal: destination path '/tmp/esgf-config' already exists and is not an empty directory.



MSG:

fatal: destination path '/tmp/esgf-config' already exists and is not an empty directory.

Expected behavior

Delete temporary files and directories before installing/upgrading or generating unique directory names in /tmp.

ESGF Node (please complete the following information):

• Managed OS: CentOS 7
• Host OS: CentOS 7
• esgf-ansible Version: 4.0.3
• Node type: all

update CoG

The version of CoG deployed with Ansible is out-of-date. Is the change in the @William-Hill William-Hill fork still necessary? I don't remember if this was only needed for Python (ESGFv3) If so we should merge into CoG devel and create a new tag.

upgrading without cert paths

Describe the bug
Trying to upgrade existing data, index and idp nodes, but esgf-ansible failed when 'Create vesgdev-idx.ipsl.upmc.fr Cert Own CA'

Full log: upgrade_without_cert_paths.log

TASK [certificate : Create vesgdev-idx.ipsl.upmc.fr Cert Own CA] ******************************************
task path: /root/tmp/esgf-ansible/roles/certificate/tasks/cert.yml:47
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1552319445.47-252336077670662/AnsiballZ_openssl_certificate.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1552319445.47-252336077670662/AnsiballZ_openssl_certificate.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1552319445.47-252336077670662/AnsiballZ_openssl_certificate.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 1091, in <module>
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 1070, in main
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 617, in generate
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 487, in check
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 469, in _validate_csr
IndexError: list index out of range
fatal: [vesgdev-data.ipsl.upmc.fr]: FAILED! => {
    "changed": false, 
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1552319445.47-252336077670662/AnsiballZ_openssl_certificate.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1552319445.47-252336077670662/AnsiballZ_openssl_certificate.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1552319445.47-252336077670662/AnsiballZ_openssl_certificate.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 1091, in <module>
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 1070, in main
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 617, in generate
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 487, in check
  File "/tmp/ansible_openssl_certificate_payload_q9dn8H/__main__.py", line 469, in _validate_csr
IndexError: list index out of range


PLAY RECAP ************************************************************************************************
vesgdev-data.ipsl.upmc.fr  : ok=105  changed=56   unreachable=0    failed=1   
vesgdev-idx.ipsl.upmc.fr   : ok=4    changed=0    unreachable=0    failed=1 

To Reproduce

## Conda env installation

(Installation de pip (pour python 2.6) => pip search ansible crashes.)

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

wget https://repo.anaconda.com/miniconda/Miniconda2-latest-Linux-x86_64.sh

chmod +x Miniconda2-latest-Linux-x86_64.sh
./Miniconda2-latest-Linux-x86_64.sh

source ${HOME}/.bashrc

conda create -y -n ansible python=2.7

conda activate ansible

pip install ansible==2.7

## esgf-ansible repo

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

TAG_NAME='4.0.0-beta1'
git clone https://github.com/ESGF/esgf-ansible.git && cd esgf-ansible && git checkout $TAG_NAME

## Inventory

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/inventory.txt" <<EOF
[data]
vesgdev-data.ipsl.upmc.fr   ansible_connection=local

[index]
vesgdev-idx.ipsl.upmc.fr    ansible_connection=local

[idp]
vesgdev-idx.ipsl.upmc.fr   ansible_connection=local
EOF

## Host vars for vesgdev-data.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-data.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false
EOF








## Host vars for vesgdev-idx.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-idx.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false
EOF


## Preprocessing 

backup :
- certificate https
- certificate gridftp

* To do for vesgdev-idx

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

* To do for vesgdev-data

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

## Upgrade

export PARENT_DIR='/root/tmp'
script ${PARENT_DIR}/esgf-ansible/upgrade.log
cd ${PARENT_DIR}/esgf-ansible
source activate ansible
export ANSIBLE_NOCOLOR=true
ansible-playbook -vvv -i ${PARENT_DIR}/esgf-ansible/inventory.txt -u root install.yml

Expected behavior

upgrading done

ESGF Node (please complete the following information):

• Managed OS: CentOS 6 (packages updated)

• Host OS: CentOS 6 (from vesgdev-idx our 'all' node)

• esgf-ansible version: 4.0.0-beta1

• ansible version: 2.7.8

• Node type:

VM #1: vesgdev-idx, an 'all' esgf node that serves as index and idp node. version v2.8.1-master-release
VM #2: vesgdev-data, an data esgf node. version v2.8.1-master-release

Additional context

Replica shards disappear from CoG after 4.0.3 upgrade.

Describe the bug
After trying to upgrade our production node (esgf-node.ipsl.upmc.fr) from 2.6.7 to 4.0.3.
CoF Data Searches and the search API itself doesn't return any data from the replica shards.
esgf_shards_static.xml and other configuration files and CoG seetings have been checked without any changes.

Note that the upgrade failed first when cloning the CoG repository. We had to force it using "force :yes" in the appropriate ansible roles.
Then the node is upgraded and starts correctly. But the whole CoG config was reset and we observe that the "TestProject" build by default redirects to the LIU node instead of IPSL which is really strange behavior.

To Reproduce
Steps to reproduce the behavior:
From ESGF 2.6.7 :
ansible-playbook -i hosts.prod -u root --skip-tags gridftp --tags 'index,idp' --limit esgf-node.ipsl.upmc.fr install.yml

Expected behavior
Having the node installed with legacy CoG configuration and project pages. Also replica shards always accessible.

ESGF Node (please complete the following information):

• Managed OS: CentOS 6
• Host OS: CentOS 6
• esgf-ansible Version: 4.0.3
• Node type: Index (full node)

Additional context
Add any other context about the problem here.

Tag for certificate checking (status)

Is your feature request related to a problem? Please describe.

I would like to skip the certificate checking when executing the status.yml recipe. Some of my nodes have self signed SSL certificates. These nodes are for development purpose.

Describe the solution you'd like

Add a tag for the tasks that check certificates so as to skip them with --skip-tags

Failing download from rainbow.llnl.gov

Describe the bug
There are attempts to download something from rainbow.llnl.gov during the easy_install of schema migration scripts.

To Reproduce
Steps to reproduce the behavior:

1. Run easy_install with the esgf_security .egg file.

Expected behavior
The script is installed without an error about downloading from rainbow.llnl.gov

Restart playbook

If I need to stop and start services, this needs to be issued in 2 commands. This is not great because (1) the stop may complete and leave the node offline for the time it takes me to notice, or I have to watch it for the period. (2) for remote Sudo I have to enter credentials twice. That would make calling both playbooks in a script impractical. Could the playbooks be wrapped into a single restart.yml, hopefully preserving order?

deprecation warning on selinux role

Describe the bug
I upgrade today my data-node to 4.0.3 using esgf-ansible, and I notice a deprecation warning in ansible logs

To Reproduce
Steps to reproduce the behavior:

1. Run installer with : ansible-playbook -i "${INVENTORY}" -u root --tags data --limit "${HOST_NAME}" --skip-tags gridftp install.yml
2. see error
   2019-10-01 07:41:29,994 p=65375 u=root | [DEPRECATION WARNING]: evaluating (is_7 and firewalld_enabled is defined and firewalld_enabled is changed) as a bare variable, this behaviour will go away and you might need to add |bool to the
   expression in the future. Also see CONDITIONAL_BARE_VARS configuration toggle.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
   deprecation_warnings=False in ansible.cfg.

ESGF Node :

• Managed OS: centos 7
• ansible : ansible-2.8.4-1.el7.noarch
• esgf-ansible Version:4.0.3
• Node type: data-only
• node name : esg1.umr-cnrm.fr

unexplicite db connection required when upgrading

Describe the bug
Trying to upgrade existing data, index and idp nodes, but esgf-ansible failed when 'Create db super user'.

From the data node (vesgdev-data ; issue #47-49 were from index node), esgf-ansible fails because it needs to connect to the index db.
The wiki doesn't give any requirements about db connections.

Full log:
upgrade_from_data_node.log

TASK [base : Create db super user] ************************************************************************
task path: /root/tmp/esgf-ansible/roles/base/tasks/postgres.yml:28
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_postgresql_user_payload_RlmlRZ/__main__.py", line 797, in main
    db_connection = psycopg2.connect(**kw)
OperationalError: n'a pas pu se connecter au serveur : Aucun fichier ou dossier de ce type
	Le serveur est-il actif localement et accepte-t-il les connexions sur la
 	socket Unix « /tmp/.s.PGSQL.5432 » ?

fatal: [vesgdev-idx.ipsl.upmc.fr]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "conn_limit": null, 
            "db": "", 
            "encrypted": true, 
            "expires": null, 
            "fail_on_user": true, 
            "login_host": "", 
            "login_password": "", 
            "login_unix_socket": "", 
            "login_user": "postgres", 
            "name": "dbsuper", 
            "no_password_changes": false, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": "5432", 
            "priv": null, 
            "role_attr_flags": "SUPERUSER", 
            "ssl_mode": "prefer", 
            "ssl_rootcert": null, 
            "state": "present", 
            "user": "dbsuper"
        }
    }
}

MSG:

unable to connect to database: n'a pas pu se connecter au serveur : Aucun fichier ou dossier de ce type
	Le serveur est-il actif localement et accepte-t-il les connexions sur la
 	socket Unix « /tmp/.s.PGSQL.5432 » ?


PLAY RECAP ************************************************************************************************
vesgdev-data.ipsl.upmc.fr  : ok=33   changed=5    unreachable=0    failed=1   
vesgdev-idx.ipsl.upmc.fr   : ok=80   changed=44   unreachable=0    failed=1  

To Reproduce

## Conda env installation

(Installation de pip (pour python 2.6) => pip search ansible crashes.)

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

wget https://repo.anaconda.com/miniconda/Miniconda2-latest-Linux-x86_64.sh

chmod +x Miniconda2-latest-Linux-x86_64.sh
./Miniconda2-latest-Linux-x86_64.sh

source ${HOME}/.bashrc

conda create -n -y ansible python=2.7

conda activate ansible

pip install ansible==2.7

## esgf-ansible repo

PARENT_DIR='/root/tmp'
cd ${PARENT_DIR}

TAG_NAME='4.0.0-beta1'
git clone https://github.com/ESGF/esgf-ansible.git && cd esgf-ansible && git checkout $TAG_NAME

## Inventory

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/inventory.txt" <<EOF
[data]
vesgdev-data.ipsl.upmc.fr   ansible_connection=local

[index]
vesgdev-idx.ipsl.upmc.fr    ansible_connection=local

[idp]
vesgdev-idx.ipsl.upmc.fr   ansible_connection=local
EOF

## Host vars for vesgdev-data.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-data.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false
EOF








## Host vars for vesgdev-idx.ipsl.upmc.fr

PARENT_DIR='/root/tmp'
cat > "${PARENT_DIR}/esgf-ansible/host_vars/vesgdev-idx.ipsl.upmc.fr.yml" <<EOF
### All paths in this file are on the machine
### on which Ansible itself will be run. That is,
### they are on the 'control' machine.

### Mirror Host
##
### The ESGF mirror to be used. Defaults to
### aims1.llnl.gov
##
### Choices:
## distrib-coffee.ipsl.jussieu.fr/pub
## dist.ceda.ac.uk
## aims1.llnl.gov
## esg-dn2.nsc.liu.se
##
mirror_host: distrib-coffee.ipsl.jussieu.fr/pub

### Prometheus Hosts and/or IPs
##
### This deployment deploys Prometheus exporters. These
### exporters report information and metrics about the node
### to a Prometheus server. The Prometheus server(s) must be
### specified so that it can be trusted to communicate with
### the exporters. Place your own Prometheus server here if
### applicable. This trust is at the webserver level.
##
### Information will not be collected until your host is added
### to the Prometheus server's target list. Remove the following
### lines if you would not like to participate in this, in which
### case only localhost will be trusted to access the exporters'
### information.
##
#prometheus_hosts:
#  - aims1.llnl.gov
#prometheus_ips:
#  - 10 # Trust your whole internal network

### Globus Services
##
### This information is used to connect the node
### to the Globus system. 
##
### Leave unspecified and the Globus Connect Server 
### setup steps will be skipped.
##
#globus_user:
#globus_pass:
#register_gridftp: true|false
#register_myproxy: true|false

### Keys and Certificates
# The priority for installing certs and keys goes like this:
# 1. Whatever is specified in the host_vars file will be installed,
#    whether or not the destination path is already present on the 
#    managed machine.
#
# 2. If the files are already present on the managed machine and 
#    nothing is specified in the host_vars file, the files that 
#    are present will be checked for validity. If the certificate is
#    found to be invalid, it will be removed and regenerated.
#
# 3. If no key and certificate files on the managed machine are
#    present and nothing is specified in the host_vars file, self-signed,
#    Ansible generated, certificates will be installed.

### Keys and Certificates for Globus Services
##
### The paths on the local machine to the respective 
### key/cert that have been signed by ESGF. If these
### have not been obtained for a node, leave unspecified.
##
### Leave unspecified and certificate signing requests for
### valid certs will be generated in the HOME directory
### of the root user.
##
#gftphostcert:
#gftphostkey:
#myproxycacert:
#myproxycakey:
#myproxy_signing_policy:

### Keys and Certificates for Web Services, httpd and tomcat
##
### The paths on the local machine to the respective 
### key/cert that have been signed by a commonly trusted 
### certificate authority.
##
### Leave unspecified and temporary, or LetsEncrypt (see below), 
### certificates will be generated.
##
#hostkey_src:
#hostcert_src:
#cachain_src:

### LetsEncrypt Certificates for Web Services
##
### LetsEncrypt certificates are browser trusted certificates
### that can be obtained in an automated fashion by proving
### you control the domain.
### In order for this to work your host must be publicly
### available at deployment time.
### This variable has no effect if the above host key/cert/chain
### variables for web services are specified.
##
### Leave unspecified to not attempt to retrieve LetsEncrypt certs.
##
#try_letsencrypt: true|false

### CentOS 6 iptables
##
### iptables is a tool that controls the Linux kernel's packet firewall.
### Many sites will likely wish to manage this on their own. If you are
### not familiar with this tool and would like this deployment to
### takes steps to configure the iptables for you on CentOS 6 then
### uncomment the below variable and set it to true. Otherwise your
### site will be responsible for ensuring the proper ports are open.
##
### Leave unspecified to manage iptables on CentOS 6 yourself.
##
#configure_centos6_iptables: true|false
EOF


## Preprocessing 

backup :
- certificate https
- certificate gridftp

* To do for vesgdev-idx

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

* To do for vesgdev-data

esg-node stop

rm -fr /etc/tempcerts
rm -fr /etc/certs.bak /etc/esgfcerts.bak /etc/grid-security.bak /var/lib/globus/simple_ca.bak ; cp -rp /etc/certs /etc/certs.bak ; cp -rp /etc/esgfcerts /etc/esgfcerts.bak ; cp -rp /etc/grid-security /etc/grid-security.bak ; cp -rp /var/lib/globus/simple_ca /var/lib/globus/simple_ca.bak

## Upgrade

export PARENT_DIR='/root/tmp'
script ${PARENT_DIR}/esgf-ansible/upgrade.log
cd ${PARENT_DIR}/esgf-ansible
source activate ansible
export ANSIBLE_NOCOLOR=true
ansible-playbook -vvv -i ${PARENT_DIR}/esgf-ansible/inventory.txt -u root install.yml

Expected behavior

upgrading done

ESGF Node (please complete the following information):

• Managed OS: CentOS 6 (packages updated)

• Host OS: CentOS 6 (from vesgdev-data, a data node)

• esgf-ansible version: 4.0.0-beta1

• ansible version: 2.7.8

• Node type:

VM #1: vesgdev-idx, an 'all' esgf node that serves as index and idp node. version v2.8.1-master-release
VM #2: vesgdev-data, an data esgf node. version v2.8.1-master-release

Additional context

Deployment testing of Python3 CoG with Ansible playbook

We have tested a deployment of the following branch: python3_cog
The branch contains updates to the playbook that point to the Python3 CoG fork/branch. Will comment on status of testing...

Upgrade to esg-publisher 3.7.0 ; add call to esgfetchini -k

esg-publisher will be updated to 3.7.0. This will remove the calls to esgfetchini and esgprep in setup.py effectively decoupling the modules.
To accommodate this, a call to esgfetchini -k should be made by the playbook prior to esgsetup is run. That call depends on esgprep installed in the environment. For now same esgf-pub as esgcet, but may consider changing near future.

Upgrading node GLIBC issue

In collaboration with @pabretonniere, we are trying to update esgf.bsc.es data node (v2.8.1-master-release) with esgf-ansible procedure but we are having a major issue.

This method seems to be currently broken for a CentOS 6.9 node. Conda installs a python version without glibc 2.12 support and consequently, fails.

We tried to upgrade from 6.9 to 7 but the upgrade crashed (we add a snapshot of the VM so we can easily roll back).

We need to update the node to solve a publisher issue but we are currently stuck. Any advice?

Thank you!

update comments about globus_user and _pass in host_vars template files

Describe the bug

Comments in the host_vars templates like myhost.my.org.yml, specifies that the globus_user and globus_pass variables are not mandatories. But upgrading to the version 4.0.4 fails when these variables are missing.

Globus Services

This information is used to connect the node
to the Globus system. Note globus_user should
not include the "@globusid.org" portion.

Leave unspecified and the Globus Connect Server
setup steps will be skipped.

#globus_user:
#globus_pass:

To Reproduce
Steps to reproduce the behavior:

1. Upgrade a node to the version 4.0.4
2. See error

ESGF Node (please complete the following information):

• Managed OS: CentOS 6
• Host OS: CentOS 7
• esgf-ansible Version: 4.0.4
• Node type: all