Drupal 8 Project for implementing the CLARITY Climate Services Information System (CSIS).
This repository contains the general configuration for CSIS docker containers deployed at CSIS Development System and CSIS Production System virtual servers.
The actual drupal site configuration which is mapped to the bind-mount volume drupal-data is stored in the separate private clarity-csis-drupal repository. This repository is mainly used for synchronising configuration and content between the development and production system.
CLARTIY CSIS is implemented as dockerized Drupal 8 with PostGIs database. The initial configuration is based on Continuous Drupal. The main artefacts of the repository are
- drupal/Dockerfile: Customised Drupal 8 Docker Container for CLARITY CSIS
- postgis/Dockerfile: Customised PostGIS 10 Docker Container for CLARITY CSIS
- docker-compose.yml: for configuring deploying the customised containers
CLARTIY CSIS is deployed as docker containers csis-drupal and csis-postgis on CSIS Development System and CSIS Production System virtual servers. A separate nginx-proxy exposes the internal Drupal Apache service to csis-dev.myclimateservice.eu and csis.myclimateservice.eu.
Building the drupal and postgis docker image is straightforward:
cd /docker/100-csis
docker-compose buildContainers are managed via the customised docker-compose.yml. The actual configuration is maintained in an .env file while .env.sample can be used as blueprint of this configuration file. Variables in this file will be substituted into docker-compose.yml.
The configuration on the development and production system is identical, except for the environment variables VIRTUAL_HOST and LETSENCRYPT_HOST required by nginx-proxy. Example for development system:
VIRTUAL_HOST=csis-dev.ait.ac.at,csis-dev.clarity.cismet.de,csis-dev.myclimateservice.eu
LETSENCRYPT_HOST=csis-dev.ait.ac.at,csis-dev.clarity.cismet.de,csis-dev.myclimateservice.euThe containers can be started with:
docker-compose up -d --force-recreate --remove-orphansData of different containers is stored in the following bind-mount volumes:
- Drupal site configuration in
/docker/100-csis/drupal-data/ - postgres database in
/docker/100-csis/postgresql_data/
UI Integration Tests both for the deployed development and production system are performed with help of cypress.io and executed on Jenkins CI. The test specifications are maintained in repository csis-technical-validation in branches csis-cypress and csis-dev-cypress. If any of the test fails, the CI system will automatically post a new issue in the repository.
CSIS Services are monitored with help of statping. The monitoring services are also integrated in Jenkins CI and a new issue is posted in repository csis-technical-validation when one of the monitored services fails.
Apart from the general virtual server backups performed by AIT data center, daily full backups of the CSIS data volumes stored at /docker/100-csis are saved in /docker/100-csis-daily.tar.gz. Additional incremental backups can be performed with help of docker-duplicity.
Upgrading the system is a multi-step process that involves not only updating the container images but also the Drupal core system as such as well custom and integrated apps and modules.
Stop and remove drupal containers:
cd /docker/100-csis
docker-compose downManually trigger incremental duplicity backup:
cd /docker/999-duplicity
docker-compose upHint: You can restore the latest backup-set with docker-compose up -f restore.yml to ./restore directory.
Edit Drupal Dockerfile and change base image to docker-drupal new version. E.g. FROM drupal:8.9.1-apache for csis-drupal and FROM postgis/postgis:10-3.0 for csis-postgis.
Hint: This does not update the existing CSIS Drupal Core, since the actual Drupal system lives in a host-mounted ./drupal-data volume! Therefore a manual update inside the new running container is required. See upgrading Drupal Core and Modules.
Edit compose file and change custom images tags, e.g. to 8.9.1-apache and csis-postgis:10-3.0, in order to be able to revert to the previous image:
drupal:
image: csis-drupal:8.9.1-apache
build:
context: ./drupal
container_name: ${COMPOSE_PROJECT_NAME}-drupalAlso upgrade csis-postgis image, if necessary.
drupal-db:
image: csis-postgis:10-3.0
build:
context: ./postgis
container_name: ${COMPOSE_PROJECT_NAME}-postgisBuild our custom csis-drupal docker images:
cd /docker/100-csis
docker-compose buildRecreate containers:
docker-compose up -d --force-recreate --remove-orphansFurthermore any changes to the docker and compose files have to be committed and a new release has to be made.
The actual site configuration which includes the Drupal core system and the different modules lives in a separate bind-mount volume and has to be updated separately.
sh into csis-drupal container:
sudo su docker
docker exec -it --user 999 csis-drupal bashBackup configuration on private clarity-csis-drupal repository:
# backup configuration
drush cex
git add --all
git commit -a -m "commit message"
git push
## Username for 'https://scm.atosresearch.eu'
## Password for 'https://[email protected]':In general, Drupal Core and Modules can be updated with help of drush and composer:
composer update
drush updbHowever, some of the modules had to be patched, so that the update process is not so straightforward. For more information refer to this issue .
Currently, the following custom modules and integrated apps are deployed together with CSIS:
They have to be updated separately. Instruction for doing so are provided in the respective readme.md files in the repository root.
The synchronisation between the development and production system consists of two separate steps for synchronising content and configuration.
NOTE: This issue discusses how to migrate configuration (e.g. site configuration like caching, configuration of Views and display & form modes, available fields in content types, ...) but not actual content (node A, Tax. term B). That is adressed in a separate issue (#147).
We're using the Configuration Split module to manage, which configuration will be enabled on the Dev or Prod server only and which will be enabled on both. Configuration will always be imported/exported completely as yaml files, but into different directories (inside /app/config/), which will (depending on the split settings) be used or ignored on the two environments. That way the configuration will always be completely available in our private Git repo.
Splits can be managed at /admin/config/development/configuration/config-split. We will have one split called dev_split. On the Dev server this split needs to be active, while on the Prod server it has to be deactivated. That will be handled in the settings.php by setting
$config['config_split.config_split.dev_split']['status'] either "TRUE" or "FALSE".
- Changes will be made on Dev server (e.g. a new field or a module will be enabled/disabled)
- View all the changes at
/admin/config/development/configurationand: 2.a) If this change should not take effect on Prod, then the split needs to be configured to exclude the change (see Managing splits above) 2.b) If this change should take effect on Prod, the split doesn't require any changes. - via terminal export the configuration using
drush config-split:export(short:drush csex)
Note: When asked whether to perform a normal (including filters) export, selectYES. Confirm export by viewing pending configuration changes (as shown in step 2). This should now be empty. - Push configuration to private repository and pull on Prod server
- Run
composer installfirst if composer.json file changed, otherwise configuration import might fail - If there are any new modules added by composer, they now need to be installed in Drupal via the "Extend" page in the Backend UI
- Import configuration via
drush config-split:import(short:drush csim) - check if updates in DB are necessary with
drush updb - clear cache with
drush cr
Due to the difficult selection possibilities in the UI of this module and unpredictable relationships between individual configuration files, it is not feasible to use this module for short-term differences in the configuration of both servers (like "new field was added but we don't want to yet include it in the upcoming synchronization"). It's difficult to achieve and very error-prone. This module is meant to be used for permanent differences in the config (like always disabling the Devel module and it sub-modules on the Prod server for security reasons).
NOTE: This issue discusses how to migrate actual content (nodes like data packages, resources etc. and taxonomy terms like the EU-GL taxonomy for example). Synchronizing configuration is discussed in issue #138 and needs to be completed BEFORE synchronizing content.
For synchronizing content we use the Migrate module, which is part of the Drupal Core and some of its additional modules (Migrate Plus and Migrate Tools). The content is pulled from the Dev via REST-Views (all called "Drupal-shared ...", which means that Dev needs to be accessible via HTTP.
The general workflow will be that certain nodes (GL-step templates, data packages and all their referenced entities) and all of the taxonomies will be exported in Dev and imported in Prod. The following actions are performed:
- New elements will be created in Prod
- Existing elements will be updated
- Elements removed in Dev will also be removed in Prod
- For synchronizing only published elements will be taken into account
The target site (in our case the production system) needs for each content type the migration configuration stored in a .yml file (these .yml files are stored and synchronized with the other configuration .yml files, see #138). To add a new migration configuration:
- Create the yml file (use existing ones for reference)
- Add the file in the Drupal backend UI at .../admin/config/development/configuration/single/import (as configuration type select "Migration")
- Synchronize configuration between Dev and Prod (#138)
- If an existing migration needs to be changed, use the "Single item export" feature under same link as in (2) and export desired migration including the migration UUID, which will be needed for the re-import later after changes to file are done
- in the Docker-container of Prod run either the provided bash script to synchronize all content in the correct order
- or synchronize individual content types using drush (make sure to migrate referenced items first!):
drush mim [ID_of_Migration_file] --update
--updateargument can be omitted. In that case existing items will not be updated but instead be skipped- in case a migration gets stuck, its status can be reset with
drush mrs [ID_of_Migration_file] - should a migration fail (e.g. content was removed in Prod and can't be found for lookup process) the migration can be rolled back with
drush mr [ID_of_Migration_file] --allargument can be used withdrush mrordrush mrsto take affect on all migrations and not just onedrush mslists all available migrations and their status and other details (can also be seen in Drupal BE)- under Structure -> Migrations (.../admin/structure/migrate) all migrations can also be viewed in the BE. It's also possible to remove individual migrations there.
MIT © Austrian Institute Of Technology, cismet GmbH and Smart Cities Consulting
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent