icatproject-contrib/icat-ansible

icat-ansible's Issues

Check version number when deciding whether to install

Currently, roles check for the existence of a directory under ~/install to decide whether to unzip the recently-downloaded component or not. If the directory exists, the unzip step is skipped. For example, see the icat.server step here.

This behaviour prevents the component being upgraded when the version number is updated: the directory already exists so the upgraded component is never unzipped. Instead, the role should check the version number stored in the local fact file and unzip if the current version is different. Here is where the icat server version number is set at the end of the installation.

When unzipping the new component in-place, the new configuration files (eg. setup.properties.example) will overwrite the old configuration files. However, the component .war file contains the version number in its file name so will be unzipped alongside the old .war file. The installation script will fail if there is more than one .war file. So, in addition to unzipping the new component, the role will have to remove the old .war file or delete the old directory.

To delete the entire installation directory before unzipping the new component feels like the ansible way - being idempotent and all - but also a bit risky in a real production environment. I'm not sure if I am brave enough for this approach. Alternatively, since we will know the currently installed version number from the local fact file, we should be able to delete the .war file directly.

Install ICAT v5 size/count database triggers

In ICAT 5, the new fileSize/fileCount fields were added. To go with this, ICAT 5 comes with a script which installs database triggers to autocalculate these values. It would be good to have an optional step to install these triggers if the user is installing ICAT 5 or above. This is needed for DG e2e tests in this PR: ral-facilities/datagateway#1499

Upgrade Ansible Version for Python 3.10 Compatibility

The current version of Ansible is not compatible with Python 3.10. When installing Ansible via requirements.txt on Python 3.10, it also tries to install ciffi which fails because the version it picks up isn't compatible with Python 3.10. As a result, the version of Ansible needs to be upgraded.

Upgrade ICAT Lucene Version

ICAT Lucene 1.1.1 has recently been released. This new version brings a (small) fix which is required for DataGateway. The version number should be updated in this repo.

Can icat-ansible run the schema upgrade scripts?

As noted in icatproject/icat.server#207, the travis build fails because the schema upgrade script is not run. Is there a way to build a string from the version number and look for the matching sql file and run it if it exists?

Upgrade Default ICAT Server Version

ICAT Server 5.0.0 has been released so this should now be the new default.

As well as the CI on this repo, it might be worth also testing the change on DataGateway API's CI

Not compatible with Mariadb 10.x

The version of mariadb is not pinned so the mariadb role installs the latest version. We could/should pin the version to whichever one we have tested successfully and/or make some changes so that the installation works with Mariadb 10.x. I have found that this script does not work with mariadb 10.x on Centos 7. We could replace it with individual SQL commands as described here or here.

We may need to make other changes to run on Mariadb 10 on Ubuntu. @kevinphippsstfc reports:

For me, ICAT failed to deploy with mysql errors about "Specified key was too long"

I found this page:
https://srsoftware.de/mariadb-innodb-keycloak

and added the following lines to /etc/mysql/mariadb.conf.d/50-server.cnf under [mysqld]:

innodb_file_format = Barracuda
innodb_file_per_table = on
innodb_default_row_format = dynamic
innodb_large_prefix = 1
innodb_file_format_max = Barracuda

Add user db/root to icat.server's rootUserNames

The test suite for icat.client expects the 'db/root' user to have permission to create a Facility. This would usually only be possible when logged in as the administrator. Therefore we need to add 'db/root' the 'rootUserNames' field of the icat.server run.properties file (and reinstall icat.server).

install epel-release when running on yum-based system

epel-release is installed already on SL7, I think, but not on CentOS7.

Handler not found error

Running the icat-ansible playbook stalls with this error:

"The requested handler 'payara-handler' was not found in either the main handlers list nor in the listening handlers list"

Add authn.ldap role

Add role that installs and sets up the LDAP authenticator.

Allow ids.server reader account to be configured on templated install

See discussion in #24

Currently, the IDS reader account is configured to be the authn.simple reader account. If a templated install is ran without the authn-simple role, this will cause the install to fail because there is no reader config set up. This can be fixed manually, but it would be good if this was configurable by the ansible script, with variables for the IDS reader account config.

Related is icatproject/icat.manual#11, as this just affects the IDS configuration of the account to use, and without the corresponding ICAT rules config the reader account won't actually have proper access.

Investigate running CI on CentOS

Currently, our CI just runs on Ubuntu via TravisCI. Most other CI services provide the exact same offering. Anvil provides CI on CentOS VMs, but currently doesn't provide admin access on VMs which ansible requires. Another alternative is to use Docker, as most CI providers have provisions to set up Docker containers, but this then requires more work setting up the Docker container.

installing authn-db doesn't add it to the icat.server configuration

After installing authn-db, I found I had to add it to the icat.server run.properties file and reinstall icat.server. Could this be added to the authn-db role?

The file at /home/glassfish/install/icat.server/run.properties needs to have:

# Desired authentication plugin mnemonics
authn.list = simple db
# Parameters for each of the four plugins
authn.db.url = https://localhost.localdomain:8181

and the icat.server then needs to be reinstalled.

The yum module is not compatible with python 3

The yum module, used in the common role is not compatible with python 3. The docs say to use the dnf module.

I assume we need to change to using the dnf module when we support Python 3, however, I have never used dnf so I am not sure I understand the consequences of installing software using dnf on a system where other software is installed using yum.

Update Payara Setup Script URL

As shown here, the setup_glassfish.py is currently fetched from icatproject.org. As a result, the playbook fails since this website is currently down. I'm aware this won't be permanent, however this seems like a good chance to update the location the script is retrieved from to a URL that's more actively updated, i.e. from https://raw.githubusercontent.com/icatproject/icat.utils/master/src/main/scripts/setup-glassfish.py.

I'm opening this issue as I'd like to use this repo to spin up an ICAT instance for a related repo (https://github.com/ral-facilities/datagateway-api) using a GitHub Actions workflow, however this is a blocking issue.

Allow for configuration files to be provided rather than generated

In order for the ansible script to be able to be used to set up production/preprod machines rather than just running through the ICAT tutorial, we need to be able to provide configuration files rather than have them be auto-generated.

Acceptance criteria:

• Config files can be provided in some way and these will be installed on the machine instead of auto-generated ones.
• If config files aren't provides, auto-generated files are made instead (aka keep current behaviour)

Ensure packages are installed as latest versions

Using state=latest rather than state=present will ensure that on subsequent runs, packages are upgraded. Additionally, in common, we should probably run apt and yum cache updates via the update_cache option.

Upgrade icat component versions

Upgrade versions:
icat.server 4.10.0
topcat 2.4.5? (soon)
ids?

Fix TRANSFORM_INVALID_GROUP_CHARS deprecation warning

[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to 
allow bad characters in group names by default, this will change, but still be 
user configurable on deprecation. This feature will be removed in version 2.10.

I think this is due to us using dashes in group names - these should be switched to use underscore instead.

Add authn.anon role

Add role that installs and sets up the anonymous authenticator.

Fix bug with Lucene config on ICAT Server

There is a bug with the Lucene config on roles/icat-server/templates/run.properties.j2. The if statement never evaluates to true and therefore all Lucene config within this file will be commented out meaning Lucene won't work without manually uncommenting these options and reinstalling ICAT Server

Update ICAT Server Version to 4.11.1

The Ansible specifies 4.10.0 as the ICAT Server version to use. 4.11.1 is available and is used by DataGateway. Since only a small amount of changes have been made between the two versions, I think it's reasonable to update the version on the Ansible repository.

Create a playbook to load some example data

Can be based on the previous icatproject-contrib/icat.ansible repository.

Create a travis configuration

Create a .travis.yml file to run the ansible scripts - preferably using the pinned ansisble version defined in #11.

Remove extra facilities from topcat.json

eg. Diamond and ISIS configurations

Need to install Oracle JDBC driver jar file in payara directory

In the same way that we install/link to the mysql connector jar file here, we need install/link to the Oracle JDBC driver to allow connections to Oracle databases.

Pin the version of ansible

Create a requirements.in file to pin the version of ansible. Hopefully, this will prevent upgrades to Ansible from breaking the scripts.

See pip-tools for an example workflow.