inasafe/docker-realtime-orchestration

Orchestration scripts for running InaSAFE realtime services

InaSAFE

InaSAFE is free software that allows disaster managers to study realistic natural hazard impact scenarios for better planning, preparedness and response activities. InaSAFE is a plugin for QGIS.

For more information about InaSAFE and its documentation please visit inasafe.org.

The latest source code is available at https://github.com/inasafe/inasafe, which contains modules for risk calculations, GIS functionality and functions for impact modelling.

• Current Travis test status master:
• Current Landscape code health master:
• Current Travis test status develop:
• Current Landscape code health develop:
• PyPi Downloads for InaSAFE:
• PyPi Downloads for the old 'safe' library (currently unmaintained):
• Github download:

Quick Installation Guide

You first need to have QGIS installed. Grab your free copy from the QGIS download page.

To install the InaSAFE plugin, use the plugin manager in QGIS:

Plugins → Manage and Install Plugins

Then search for "InaSAFE", select it and click the install button. The plugin will now be added to your plugins menu.

Note: You may need to restart QGIS if upgrading from a prior version.

Participation

We work under the philosophy that stakeholders should have access to the development and source code, and be able to participate in every level of the project - we invite comments, suggestions and contributions. See our milestones list and our open issues list for known bugs and outstanding tasks. You can also chat live with our developers and community members using the link below.

System Requirements

• A standard PC with at least 4GB of RAM running Windows, Linux or Mac OS X
• The QGIS Open Source Geographic Information System (http://www.qgis.org). Although InaSAFE will probably run on any version of QGIS version above 2.0 or newer, our supported version is the latest Long Term Release version of QGIS.

Running tests

• InaSAFE is using the Unittest Python framework.
• Different ways that you can run tests:
  • On Travis using a Pull Request
  • In QGIS Desktop if you enable developer mode in Plugins -> InaSAFE -> Options -> Advanced, restart QGIS and then click the run tests button.
  • Using docker make docker-test
  • Using PyCharm if you set up the dev environment with the qgis_prefix_path

History

• In March 2011, Risk In A Box (the predecessor to InaSAFE) was built as a web application running as a GeoNode Plugin.
• InaSAFE development as a QGIS plugin started in January 2012.
• In October 2012, version 1.0 of InaSAFE was launched at the 5th Asian ministerial conference on disaster risk reduction (AMCDRR) where it was demonstrated to the then president of Indonesia, Susilo Bambang Yudhoyono who praised the project.
• On 30 Jan 2013, InaSAFE was awarded the prestigious Black Duck Open Source Rookie of the Year Award.
• Version 2.0 of InaSAFE was released in January, 2014.
• Version 3.0 was released in March 2015 which added support for using the QGIS API within impact functions.
• Version 4.0 was released in March 2017.
• Version 5.0 was released in November 2018 which add support for QGIS 3.x

Disclaimer

InaSAFE has been jointly developed by the Indonesian Government-BNPB, the Australian Government, the World Bank-GFDRR and independent contributors. These agencies and the individual software developers of InaSAFE take no responsibility for the correctness of outputs from InaSAFE or decisions derived as a consequence.

License

InaSAFE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 (GPLv3) as published by the Free Software Foundation.

The full GNU General Public License is available in LICENSE.txt or http://www.gnu.org/licenses/gpl.html

Disclaimer of Warranty (GPLv3)

There is no warranty for the program, to the extent permitted by applicable law. Except when otherwise stated in writing the copyright holders and/or other parties provide the program "as is" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The entire risk as to the quality and performance of the program is with you. Should the program prove defective, you assume the cost of all necessary servicing, repair or correction.

Limitation of Liability (GPLv3)

In no event unless required by applicable law or agreed to in writing will any copyright holder, or any other party who modifies and/or conveys the program as permitted above, be liable to you for damages, including any general, special, incidental or consequential damages arising out of the use or inability to use the program (including but not limited to loss of data or data being rendered inaccurate or losses sustained by you or third parties or a failure of the program to operate with any other programs), even if such holder or other party has been advised of the possibility of such damages.

docker-realtime-orchestration's Issues

update README for current orchestration

It will need to show full ansible-docker-rancher documentation before the first BNPB training session.

docs and orchestration

From @gubuntu on May 18, 2017 10:59

README-docker.md make deploy tries to read two different sftp_credential env files but there is just one. Please create and populate the other one with correct defaults and check that make deploy works.

Copied from original issue: inasafe/inasafe-django#162

Add rsync to sftp container

Self explanatory.
Needed for BMKG who uses rsync to copy data.

sync content from production to staging

From @gubuntu on May 16, 2017 11:0

to enable full functionality on staging with all the latest events and media...

• use btsync to sync static content and database backups from production to staging

• script daily update of staging from synced data

Copied from original issue: inasafe/inasafe-django#160

We should replace the btsync image with my generic one

Since we built this repo, I have made a generic btsync container. It is much more re-usable with launch syntax like this:

docker pull kartoza/btsync
docker run --name "myname.kartoza.com" \
    --restart=always \
    --hostname="myname.kartoza.com" \
    -e SECRET=123456 \
    -e DEVICE=somedevice.kartoza.com \
    -v /home/blah/yourshare:/btsync \
    -d -t kartoza/btsync

See https://github.com/kartoza/docker-btsync for more details. It would be good to replace our btsync container with this.

add SSL to standard orchestration

specifically, the BNPB instance needs to work over SSL.

We need a working, documented backup system for the server

From @christodewit on January 31, 2017 6:23

Can you please spend some time to come up with a plan to ensure we have a proper backup system? Perhaps we can rsync / btsync backups to a filestore on the AIFDR cluster? Also we should document how to go about restoring the running services in the case of total hardware failure / security breach. Including but not limited to:

• osm.linfiniti.com (for osm downloader) - @timlinux to do
  

• experimental.inasafe.org (for nightly packages)
  

• test documentation builds
  

• live web site
  

• download site
  

• blog
  

We can also look at using hetzner backup services though I don't think they provide a big enough storage space. Also we could look at using backblaze - its USD 5 a month with unlimited storage.

@Charlotte-Morgan
See original ticket at inasafe/inasafe#1298 for further discussion.

Copied from original issue: inasafe/inasafe-realtime#37

end-of-life this repo

Keep this going for realtime until the seamless platform has replaced realtime.

Keep it in maintenance mode at the moment - work on existing tickets only.

Realtime orchestration will eventually consist of GeoNode-GeoSAFE orchestration + Realtime module. So, migrate realtime-specific code from here into inasafe-realtime

We need a working, documented backup system for the server

From @christodewit on January 31, 2017 6:23

Can you please spend some time to come up with a plan to ensure we have a proper backup system? Perhaps we can rsync / btsync backups to a filestore on the AIFDR cluster? Also we should document how to go about restoring the running services in the case of total hardware failure / security breach. Including but not limited to:

• osm.linfiniti.com (for osm downloader) - @timlinux to do
  

• experimental.inasafe.org (for nightly packages)
  

• test documentation builds
  

• live web site
  

• download site
  

• blog
  

We can also look at using hetzner backup services though I don't think they provide a big enough storage space. Also we could look at using backblaze - its USD 5 a month with unlimited storage.

@Charlotte-Morgan
See original ticket at inasafe/inasafe#1298 for further discussion.

Copied from original issue: inasafe/inasafe-realtime#37

Even if we put credentials in a mounted ssh-config, it is possible that the remote host identification changed

This causes the client to remove their known_hosts first in order to log in again

Inasafe-django Backup Plan

From @lucernae on October 16, 2015 9:56

By using sftp-backup, we can create a hanoi-fashioned backup plan to remote sftp server.

How SFTP Backup works

SFTP-backup is a backup service in docker. To use it, one must provides remote SFTP Credentials, SFTP directory to put the backup into, the target directory to backup (as a docker volume) and a local backup folder (as a docker volume).

This service will try to make a compressed tar ball (tar.gz) of the target directory.
The compressed file will be stored into a local backup folder and then pushed to
a remote sftp directory. Each archive is given a unique filename based on the day
the archive is created, and will be put into a nicely ordered structure of Year/Month
folder. So, in the local and remote folder, there can be several backup file to be kept.

To control how many backup file to keep. We can specify 3 environment variable into
the docker container. There are yearly, monthly, and daily variable. Each variable
is used as a limit of how many latest backup to keep. Example, 7 daily backup means
the service will try to keep 7 most recent daily backup, and 12 monthly backup means
the service will try to keep 12 most recent monthly backup (created at 1st date each
month). The same applies to yearly backup.

There are also command to push and pull all backups. Pull commands mean the
service will try to pull all the backups being kept in remote SFTP directory in to local
backup folder. Push command do otherwise. The command will overwrites existing
backup if the name is conflicted.

In inasafe django, two folder needs to be backed up. The latest dump, and media folder.
The media folder is quite big, so maybe we want to reduce the backup frequency.

Copied from original issue: inasafe/inasafe-django#115

We should document the detail of the containers e.g the port used

orchestration tweaks

I completed 'make checkout' after you did

cd inasafe-src/
git clone https://github.com/lucernae/inasafe.git

your current code does a pull but it needs to make sure the initial clone has been done.

Also, push realtime-backport-cherry-pick branch upstream and update orchestration to pull those dev branches.

Prepare to migrate backup architecture to be able to use by Rancher

From @lucernae on July 26, 2017 2:40

Our current backup system relies on accessing the host machine directly and doing several commands to fetch or push backup.
We need to be able to support this from rancher, minimally, using shell login.

ref #91

Copied from original issue: inasafe/inasafe-realtime#101

Correct timezone issues in generated pdf

The pdf definitely generated at leas a minute after BMKG pushed it, but it shows that it generated 7 hours later. Perhaps this is because of timezone issues.

monitoring and alerts

Problem

When there is a problem or the server goes down, stakeholders should be notified automatically and not have to wait for someone to discover the issue.

from Charlotte, Rizky email thread 15 Jan 2018:

Actually I need your input for this one. If this happens in production like in the past, I informed Ibu Dian (BNPB) and Pak Yedi + Pak Artadi (BMKG). But since this is a staging server and mainly being used by PVMBG, is Ibu Estu the right person to inform too?

I think that it would be good if you can inform Kartoza and Geoscience Australia (me, Rikki, David) and DMI (Ivan) any time either staging or production are down. I think it would be good to contact the partners; at the moment this is BNPB, BMKG and PDC for production and PVMBG and BNPB for staging. Considering the imminent migration to production - I think we could look at expanding this list to include all agencies who will be users of the sustainable system (BNPB, BMKG, PVMBG, BNPB, PetaBencana, PDC, BPBD DKI Jakarta) and until June 2018 agencies involved in the development and maintenance (Kartoza, DMI, GA).

Proposed solution

• Sentry
• Grafana / Prometheus

Also (from realtime meeting minutes):

Set up communication protocol for reporting and discussing issues. E.g. put alert on website. Email stakeholders. E.g. if a site is down. Channel for reporting an outage. E.g. Zendesk. Build a status log. Like https://status.github.com/messages

The solution should be built into Kartoza standard orchestration

Update fig with docker compose

move inasafe rancher to rancher.kartoza.com

both staging and production

integrate as rancher catalogues linked to https://github.com/kartoza/kartoza-rancher-catalogue

(bnpb rancher to stay on bnpb servers. And use the same catalogues?)

more orchestration tweaks

add these to the right place in the README, with descriptions and in a sensible order

• make bmkg-monitor
• make bmkg-monitor-log
• make inasafe-worker
• make up

make deploy seems to just generate a deploy script - is this correct and is it still necessary? or superseded by make up?

move rancher server

Problem

Rancher server is on staging so if staging goes down, rancher goes down too.

Solution

Move rancher server to rancher.inasafe.org (78.47.48.44)