ome/prod-playbooks

Playbooks used by the OME team for deploying production services including OMERO

OME production services playbooks

These playbooks encapsulate the running of various production servers run by the OME team. If you are looking for examples of running your own production OMERO.server see

https://github.com/ome/omero-deployment-examples

Details

• Install Ansible and dependencies using the ome-ansible-molecule package.
• Install required roles: ansible-galaxy install -r requirements.yml
• Run the site.yml playbook.

For details of individual playbooks see the comments in site.yml.

Testing

All server playbooks have a corresponding molecule test scenario under molecule.

prod-playbooks's Issues

Review infra PRs for modifications

@manics pointed out that a number of discussions had been had regarding the prod playbooks. The ones I've found are:

• ome/infrastructure#255
• ome/infrastructure#268 (comment)
• ome/infrastructure#271
• ome/infrastructure#273
• ome/infrastructure#276

These should be reviewed for modifications. Positive outcomes would include:

• trusting -C --diff on production hosts
• trusting site.yml to run periodically
• refactoring into idempotent tasks
• reducing duplication between the playbooks here

Disable PixelData service on training servers

We have recently been faced with the import of data a that does not meet the multi-resolution criteria for large images cross multiple users. This leads to large pyramids been generated (even if the data is in-place imported) and can put the system at risk. Similarly to the approach taken on several production deployments, the PixelData service could be fully disabled on the training servers especially if none of the training workflows should rely on the pyramid generation at the moment.

The omero.server.nodedescriptors configuration property should allow to select the services that need to be enabled at server startup.

In the case of IDR, we only enable Blitz and Tables

https://github.com/IDR/deployment/blob/a5871ed5fc144158fbe8d830ae062cc6d6e6417c/ansible/group_vars/omero-hosts.yml#L110-L111

For the training server, we probably also want to keep Processor-0 (for OMERO.scripts) and Indexer-0 (for the Lucene indexing).

Upgrade nightshade-webclients to OMERO.web 5.14.0

See https://www.openmicroscopy.org/2022/03/21/omero-web-5.14.0.html and #335

On nightshade login failure warn of LSD password expiry.

Address https://trello.com/c/5W4PXbN6/6471-mark-cant-log-into-ome-resources#comment-5c9367e00d650d4c7c6ae720 using ome/openmicroscopy#6044.

Update postresql prometheus exporter

Same as IDR/deployment#244

The monitoring agent started failing following the upgrade to PostgreSQL 11.

"Download the Figure_To_Pdf.py script" fails when omero_figure_release is 4.4.0

TASK [Download the Figure_To_Pdf.py script] ********************************************************************
fatal: [206.12.95.221]: FAILED! => {"changed": false, "dest": "/opt/omero/server/OMERO.server/lib/scripts/omero/figure_scripts/Figure_To_Pdf.py", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://raw.githubusercontent.com/ome/omero-figure/v4.4.0./omero_figure/scripts/omero/figure_scripts/Figure_To_Pdf.py"}

I think this is due to the fact that on PyPi, omero-figure is released at 4.4.0, but in the Github repo, the latest release is 4.3.1.

Figure move scripts on dundeeomero

In preparation of the workshop next week, noting that the scripts mentioned in https://omero-guides.readthedocs.io/en/latest/figure/docs/omero_figure_move.html#setup are not on dundeeomero server (nightshade).

What would be the correct course of action to get them there ?

cc @sbesson @will-moore @jburel

use of pip module means "--check" always shows `changed`

use of pip module means "--check"mode always shows changed for pip tasks.

Internal OME links:
https://openmicroscopy.slack.com/archives/C0K5ME1EW/p1512749355000430
https://github.com/openmicroscopy/management_tools/pull/612

OMERO.server 5.6.5 upgrade

• ome-demoserver [@sbesson]
• learning [@sbesson]
• training server [@pwalczysko]
• ome-dundeeomero [@pwalczysko]

OME Schemas migration

Current deployment

The OME schemas, available publicly from https://www.openmicroscopy.org/Schemas/ are composed of:

• the specification files (xsd), transforms (xslt) and index pages, generated from the source files under https://github.com/ome/ome-model
• the HTML documentation generated using oXygen Editor from the live schemas

At the moment, the schema pages are hosted on www-legacy.openmicroscopy.org:

• the specification pages are statically served under https://www-legacy.openmicroscopy.org/Schemas/

• a set of Apache redirect rules allow to redirect the legacy https://www-legacy.openmicroscopy.org/XMLSchemas/ URL (for the FC schema?)

   # Legacy schema URLs
    RewriteRule ^/XMLschemas/(.*) /Schemas/$1 [P]
    RewriteRule ^/XMLschemas/ /Schemas/ [P]
    RewriteRule ^/XMLschemas$ /Schemas/ [P]
  

• the generated documentation is statically served under https://www-legacy.openmicroscopy.org/schema_doc/

• a set of Apache redirect rules allow to map the generation documentation under e.g. https://www-legacy.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06

   # Documentation redirects
   RewriteRule ^/Schemas/Documentation/Generated/OME-(.*)/(.*) /schema_doc/$1/$2 [P]
   RewriteRule ^/Schemas/Documentation/Generated/OME-(.*)/ /schema_doc/$1/ [P]
   RewriteRule ^/Schemas/Documentation/Generated/OME-(.*)$ /schema_doc/$1/ [P]
  

These locations are then proxied from www-legacy.openmicroscopy to www.openmicroscopy.org

prod-playbooks/www/www-deploy.yml

Lines 36 to 41 in 3b3d2e0

	- location: /Schemas
	server: https://www-legacy.openmicroscopy.org/Schemas
	- location: /XMLschemas
	server: https://www-legacy.openmicroscopy.org/XMLschemas
	- location: /schema_doc
	server: https://www-legacy.openmicroscopy.org

Specification migration

A first step towards the simplification of this deployment was performed in ome/www.openmicroscopy.org#420 with the inclusion of the specification, transforms and index pages as part of the static OME Jekyll website - see https://ome.github.io/www.openmicroscopy.org/Schemas/. As per the proxy configuration above, these pages are not the ones used in production yet.

Generated documentation migration

In order to get rid of the www-legacy -> www proxying, the generated oXygen documentation also needs to be migrated.

The paragraph below discusses a few deployment options. All of them assume that:

• the current generated oXygen documentation is largely unchanged and can be considered as a static artifact
• the primary URLs to maintain are https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome.html, https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2015-01/ome.html...

1. deploy the static generation under https://www.openmicroscopy.org/schema_doc (similar to the read-only phpBB forums -

   prod-playbooks/www/www-static.yml

   Lines 12 to 18 in 3b3d2e0

   	- role: ome.deploy_archive
   	become: yes
   	deploy_archive_dest_dir: /var/www
   	deploy_archive_src_url: https://downloads.openmicroscopy.org/web-archive/phpbbforum-20190718.tar.gz
   	deploy_archive_sha256: e9d7a7eefbacf42ddbdf92b201584913cb6d94ec331750f811232b2e91aa5b40
   	# This file is patched later so only unzip if it doesn't exist
   	when: not _phpbbforum_style_file_st.stat.exists

   and

   prod-playbooks/www/www-deploy.yml

   Lines 285 to 307 in 3b3d2e0

   	# Static copy of old phpBB forums: treat query params as part of filename
   	- location: "~ ^/community/style.php.*"
   	root: /var/www/phpbbforum/www.openmicroscopy.org
   	custom:
   	- try_files $request_uri $uri =404
   	- default_type text/css
   	- location: "~ ^/community/?$"
   	redirect301: /community/index.php
   	- location: /community
   	root: /var/www/phpbbforum/www.openmicroscopy.org
   	custom:
   	# Need to exclude extra query parameters in incoming external links
   	# e.g. sid=
   	# If an exact match isn't found try just these parameters:
   	# [f, t, p], [f, t], [f]
   	- >-
   	try_files
   	$request_uri
   	$uri?f=$arg_f&t=$arg_t&p=$arg_p
   	$uri?f=$arg_f&t=$arg_t
   	$uri?f=$arg_f
   	=404
   	- default_type text/html

   ), add some nginx redirects to serve these under the /Schemas/Documentation/Generated
   • pros: close to the current deployment, minimal amount of redeployment for the generated documentation
   • cons: requires some investigation for the nginx configuration
2. commit the generated documentation pages under https://github.com/ome/schemas, include these pages in the OME static website artifact and have it deployed as part of each update

• pros: minimal amount of redirection/nginx configuration
• cons: increased size for the static website, might force to manage copies more proactively

3. redirect https://www.openmicroscopy.org/Schemas/Documentation/Generated to a third-party location hosting the oXygen schema documentation

• pros: separates the official specification from the documentation
• cons: with the migration of the content away from docs.openmicroscopy.org, need to define a hosting platform

Learning maintenance: evaluate script moving users to a disabled group

The UoD SLS learning system is getting heavily used during the academic term with new students logging into the resource using their University credentials and being mapped into the default group containing the course supporting data to allow access - see

prod-playbooks/omero/learning.yml

Lines 57 to 61 in b2e0137

	omero.ldap.base: "{{ omero_server_ldap_base }}"
	omero.ldap.username: "{{ omero_server_ldap_username }}"
	omero.ldap.password: "{{ omero_server_ldap_password | default('') }}"
	omero.ldap.user_filter: "{{ omero_server_ldap_user_filter }}"
	omero.ldap.new_user_group: "{{ omero_server_ldap_new_user_group }}"

.

Eventually, many teaching semesters lead to the creation of several 100s of users in the single group. This causes performance degradation on the OMERO.web deployment as several of the queries start to run more slowly.

At the moment, the OME team runs periodically ad-hoc maintenance scripts on the system to reduce the number of users in the group. On the image.sc forum, a similar problem was discussed with a script being shared which allows to move stale users into a "graveyard" group - see https://forum.image.sc/t/omero-user-scripted-inactivation/70767/3.

For the next maintenance, we might want to evaluate whether this script could be adapted to the requirements of the SLS learning deployment and used to move students from the former year to another group to restore performance.

remove mtbc.* roles from requirements.yml

ome-demoserver.yml with current inventory/prod-hosts has two tasks that always show as changed

ome-demoserver.yml with current inventory/prod-hosts has two tasks that always show as changed:

TASK [Create a figure scripts directory] ***************************************
task path: /Users/spli/omesys/management_tools/prod/playbooks/ome-demoserver.yml:292
changed: [ome-demoserver.openmicroscopy.org] => {"changed": true, "gid": 990, "group": "omero-server", "mode": "0755", "owner": "root", "path": "/opt/omero/server/OMERO.server/lib/scripts/omero/figure_scripts", "size": 162, "state": "directory", "uid": 0}

TASK [Download the Figure_To_Pdf.py script] ************************************
task path: /Users/spli/omesys/management_tools/prod/playbooks/ome-demoserver.yml:301
changed: [ome-demoserver.openmicroscopy.org] => {"changed": true, "dest": "/opt/omero/server/OMERO.server/lib/scripts/omero/figure_scripts/Figure_To_Pdf.py", "gid": 0, "group": "root", "mode": "0644", "msg": "file already exists but file attributes changed", "owner": "root", "size": 77648, "state": "file", "uid": 0, "url": "https://raw.githubusercontent.com/ome/omero-figure/v4.0.0/omero_figure/scripts/omero/figure_scripts/Figure_To_Pdf.py"}
META: ran handlers

nightshade-webclients /etc/fstab passno

Nightshade-webclients (at least) have had root partitions resized without ome/ansible-role-lvm-partition#3 - so the partition resize will have set the passno to 2 rather than the 1 it should be.

I expect the next server upgrade PRs after ome/ansible-role-lvm-partition#3 is merged and the requirements is bumped in https://github.com/openmicroscopy/prod-playbooks/blob/master/requirements.yml to --Fixes this issue.

OMERO.web temp directory (zip downloads) hardening

We should configure our prod systems to handle an OMERO.web user trying to zip/download a large number of files and DOS'ing the system via the file creation.

Including moving whatever folder is being used for this purpose by OMERO.web out of operating system /

cf https://github.com/openmicroscopy/management_tools/issues/611

Enable websockets on demo servers

This will provide a full production server for testing websockets

Enable www/redirect tests

Enable www/redirect tests
These were disabled in #211 due to timeouts on Travis. See the issue for potential and failed workarounds

Use ome.omero_user in SLS Gallery and Virtual Microscope playbooks.

The sls-gallery.yml and learning.yml playbooks include a comment about using bin/omero db password to set the password of existing OMERO users. The ome.omero_user role can do this instead given the database settings.

Upgrade sls-gallery to OMERO.web 5.14

See https://www.openmicroscopy.org/2022/03/21/omero-web-5.14.0.html and #335

Production server playbooks: broken installation of reportlab

See https://github.com/ome/prod-playbooks/runs/3462688112 and https://github.com/ome/prod-playbooks/runs/3462688217.

The same issue as the one reported in https://forum.image.sc/t/omero-reportlab-fails-to-install-with-gcc-error/56552 now affects the Molecule testing of our current SLS OMERO playbooks. This leaves us with the same range of workarounds:

• install gcc and python3-devel on these servers
• install python-reportlab
• cap reportlab ?
• look into alternative options

/cc @will-moore @pwalczysko

Upgrade learning to OMERO.web 5.14

See https://www.openmicroscopy.org/2022/03/21/omero-web-5.14.0.html and #335

Mount data volumes on training servers

Design of an ansible feature in the training playbook which would insert a loop into the training playbook iterating over a list of directories to be mounted as a volume on the training servers governed by training playbook
The list would be by default empty inside the public training playbook playbook, but it would consume an override private variable.
This feature is mainly motivated by new addition of idr data such as ome/omero-figure#431 (comment) but can be useful for external users of the training playbook as they can replace the list of the directories accordingly for their environment.

cc @sbesson

Upgrade ome-demoserver to OMERO.web 5.14.0

See https://www.openmicroscopy.org/2022/03/21/omero-web-5.14.0.html and #335

Server Upgrade to 5.6.4

• Pub omero [@jburel ] 2022-04-28
• demo (@sbesson) - 2022-05-02
• training server [@pwalczysko & @dominikl] 2022-05-02
• learning [???] (Probably over summer)
• nightshade [@pwalczysko, @dominikl, @sbesson @jburel] 2022-05-04

OMERO.web prerequisites aren't updated on upgrades

For some reason https://github.com/openmicroscopy/openmicroscopy/blob/v5.4.9/components/tools/OmeroWeb/requirements-py27-all.txt doesn't appear to be re-evaluated on OMERO.web upgrades for at least

• ome-demoserver.yml

This may well be an issue with https://github.com/openmicroscopy/ansible-role-omero-web rather than the "prod playbook", but at least a workaround can be written in this repo, and I'd class this repo as the place to leave a top-level issue with the config of a server defined here.

SLS Gallery and Virtual Microscope playbooks don't restart web

learning.yml and sls-gallery.yml have omero_web_systemd_start: False because there was originally an issue that the playbook would not work correctly for a fresh install, perhaps because the web server was started before all configuration and certificates were yet in place.

OMERO.web app configuration style unification and fixes

While reviewed the configuration changes in #332, we realised that our Ansible playbooks use very different styles for setting the OMERO.web app configuration:

• sls-gallery and learning explicitly set the OMERO.web configuration using the primary omero_web_config_set - see

  prod-playbooks/sls-gallery.yml

  Lines 81 to 111 in d21c6f7

  	omero_web_config_set:
  	omero.web.server_list:
  	- ["localhost", 4064, "SLS Gallery"]
  	omero.web.prefix: '/ome-sls'
  	omero.web.static_url: '/ome-sls/static/'
  	omero.web.login_redirect:
  	redirect:
  	- webindex
  	viewname: "load_template"
  	query_string: "experimenter=-1"
  	args:
  	- userdata
  	omero.web.ui.top_links:
  	- ["Image Gallery", "webindex", {"title": "Image Gallery"}]
  	- ["HELP", "https://help.openmicroscopy.org/web-client.html", {"title": "Help", "target": "new"}]
  	- ["SLS Homepage", "https://www.lifesci.dundee.ac.uk/", {"title": "SLS Homepage", "target": "new"}]
  	omero.web.caches:
  	default:
  	BACKEND: django_redis.cache.RedisCache
  	LOCATION: redis://127.0.0.1:6379/0
  	omero.web.session_engine: django.contrib.sessions.backends.cache
  	omero.web.apps:
  	- "omero_iviewer"
  	omero.web.open_with:
  	- ["Image viewer", "webgateway", {"supported_objects": ["image"], "script_url": "webclient/javascript/ome.openwith_viewer.js"}]
  	- ["omero_iviewer", "omero_iviewer_index", {"supported_objects": ["images", "dataset", "well"], "script_url": "omero_iviewer/openwith.js", "label": "OMERO.iviewer"}]
  	omero.web.viewer.view: omero_iviewer.views.index
  	omero_web_apps_packages:
  	- omero-iviewer=={{ omero_web_apps_release.omero_iviewer }}
  	omero_web_python_addons:
  	- "django-redis<4.9"

  and

  prod-playbooks/learning.yml

  Lines 82 to 118 in d21c6f7

  	omero_web_config_set:
  	omero.web.server_list:
  	- ["localhost", 4064, "Virtual Microscope"]
  	omero.web.prefix: '/dundee'
  	omero.web.static_url: '/dundee/static/'
  	omero.web.login_redirect:
  	redirect:
  	- webindex
  	viewname: "webindex_custom"
  	omero.web.ui.top_links:
  	- ["Virtual Microscope", "webindex", {"title": "Virtual Microscope"}]
  	- ["HELP", "https://help.openmicroscopy.org/virtual-microscope.html", {"title": "Help", "target": "new"}]
  	omero.web.ui.right_plugins:
  	- ["Acquisition", "webclient/data/includes/right_plugin.acquisition.js.html", "metadata_tab"]
  	omero.web.caches:
  	default:
  	BACKEND: django_redis.cache.RedisCache
  	LOCATION: redis://127.0.0.1:6379/0
  	omero.web.session_engine: django.contrib.sessions.backends.cache
  	omero.web.apps:
  	- "omero_gallery"
  	- "omero_iviewer"
  	- "virtualmicroscope"
  	omero.web.open_with:
  	- ["Image viewer", "webgateway", {"supported_objects": ["image"], "script_url": "webclient/javascript/ome.openwith_viewer.js"}]
  	- ["omero_iviewer", "omero_iviewer_index", {"supported_objects": ["images", "dataset", "well"], "script_url": "omero_iviewer/openwith.js", "label": "OMERO.iviewer"}]
  	omero.web.viewer.view: omero_iviewer.views.index
  	omero.web.public.enabled: true
  	omero.web.public.password: "{{ omero_web_public_password | default('public') }}"
  	omero.web.public.url_filter: "/(webgateway|gallery)/"
  	omero.web.public.user: "{{ omero_web_public_user | default('public') }}"
  	omero_web_apps_packages:
  	- omero-gallery=={{ omero_web_apps_release.omero_gallery }}
  	- omero-iviewer=={{ omero_web_apps_release.omero_iviewer }}
  	- omero-virtual-microscope=={{ omero_web_apps_release.omero_virtual_microscope }}
  	omero_web_python_addons:
  	- "django-redis<4.9"

• ome-demoserver.yml converts three Jinja template containing a series of config append and config set subcommands into configuration files

  prod-playbooks/ome-demoserver.yml

  Lines 213 to 246 in d21c6f7

  	- name: Config for OMERO.web plugins
  	become: yes
  	template:
  	src: templates/omero-web-config-for-webapps.j2
  	dest: "{{ omero_web_basedir }}/config/omero-web-config-for-webapps.omero"
  	owner: "root"
  	group: "root"
  	mode: "u=rw,go=r"
  	notify:
  	- restart omero-web
  	- name: OMERO.web config for CORS
  	become: yes
  	template:
  	src: templates/omero-web-config-for-cors.j2
  	dest: "{{ omero_web_basedir }}/config/omero-web-config-for-cors.omero"
  	owner: "root"
  	group: "root"
  	mode: "u=rw,go=r"
  	notify:
  	- restart omero-web
  	- name: OMERO.web config for signup app
  	become: yes
  	template:
  	src: templates/omero-web-config-signup.j2
  	dest: "{{ omero_web_basedir }}/config/omero-web-config-signup.omero"
  	# Contains sensitive info
  	owner: "root"
  	group: "omero-web"
  	mode: "0640"
  	notify:
  	- restart omero-web
  	no_log: true

• nightshade-webclients.yml uses a combination of omero_web_apps_names, omero_web_apps_packages, omero_web_apps_top_links, omero_web_apps_config_append and omero_web_apps_config_set - see

  prod-playbooks/nightshade-webclients.yml

  Lines 113 to 169 in d21c6f7

  	omero_web_apps_names:
  	- omero_figure
  	- omero_fpbioimage
  	- omero_iviewer
  	- omero_parade
  	- omero_webtagging_autotag
  	- omero_webtagging_tagsearch
  	omero_web_apps_packages:
  	- "omero-figure=={{ omero_figure_release }}"
  	- "omero-fpbioimage=={{ omero_fpbioimage_release }}"
  	- "omero-iviewer=={{ omero_iviewer_release }}"
  	- "omero-parade=={{ omero_parade_release }}"
  	- "omero-webtagging-autotag=={{ omero_webtagging_autotag_release }}"
  	- "omero-webtagging-tagsearch=={{ omero_webtagging_tagsearch_release }}"
  	omero_web_apps_top_links:
  	- label: Figure
  	link: figure_index
  	attrs:
  	title: Open Figure in new tab
  	target: _blank
  	- label: Tag Search
  	link: tagsearch
  	omero_web_apps_config_append:
  	omero.web.open_with:
  	- - omero_figure
  	- new_figure
  	- supported_objects:
  	- images
  	target: _blank
  	label: OMERO.figure
  	- - omero_fpbioimage
  	- fpbioimage_index
  	- supported_objects:
  	- image
  	script_url: fpbioimage/openwith.js
  	label: FPBioimage
  	- - omero_iviewer
  	- omero_iviewer_index
  	- supported_objects:
  	- images
  	- dataset
  	- well
  	script_url: omero_iviewer/openwith.js
  	label: OMERO.iviewer
  	omero.web.ui.center_plugins:
  	- - Auto Tag
  	- omero_webtagging_autotag/auto_tag_init.js.html
  	- auto_tag_panel
  	- - Parade
  	- omero_parade/init.js.html
  	- omero_parade
  	omero_web_apps_config_set:
  	omero.web.viewer.view: omero_iviewer.views.index

• omero/training-server/playbook.yml copies a .omero file containing a series of omero config set and omero config append commands into the configuration directory - see

  prod-playbooks/omero/training-server/playbook.yml

  Lines 182 to 191 in d21c6f7

  	- name: Outreach only config for OMERO.web plugins
  	become: yes
  	template:
  	src: files/omero-web-outreach-webapps.omero
  	dest: "{{ omero_web_basedir }}/config/omero-web-outreach-webapps.omero"
  	owner: "root"
  	group: "root"
  	mode: "u=rw,go=r"
  	notify:
  	- restart omero-web

For comparison, the IDR deployment playbooks use a style most closest to nightshade-webclients i.e. the built-in omero_web_apps* variables supplemented by omero_web_apps_config_append and omero_web_apps_config_set -see https://github.com/IDR/deployment/blob/77b759ce21f2a165903aa5cbea7fcb673fcf55a8/ansible/group_vars/omero-hosts.yml#L162-L604

The mixture of styles is particularly confusing. Deciding on the preferred style(s) and applying them systematically would certainly reduce maintenance.

A related issue highlighted by the progression #332 is that extreme care should be put in the usage of the _append variables. The configuration files stored under /opt/omero/web/config are loaded as part of the omero-web service lifecycle. Internally omero config append includes some logic avoiding the duplication of a value in a configuration list. However, removing a value from a list e.g. unregistering a web app requires more thoughts

Add support for SLA

The OMERO sign-up application has been deployed in production onto the OME Demo server

As a RFE, it would be useful to display the Service Level Agreement (SLA), either as a pop-up or as a separate static page linked from the signup page.

OME demo server: fix DH key exchange errors

See https://forum.image.sc/t/omero-in-place-import-dh-key-too-small/83219/7

The server is currently hosted on CentOS7 and is affected by the DH key exchange errors when connecting from e.g. Ubuntu 22.04 as discussed in https://forum.image.sc/t/omero-login-ssl-error-dh-key/79574.

Work is ongoing to update omero-certificates (see ome/omero-certificates#33) and the OMERO documentation (ome/omero-documentation#2315) with recommendations on how to update the server configuration. Once these are finalized, these recommendations should likely be applied via the playbook.

Define use of tags

See discussion under #14 (comment)