wearepal/landscapes

Landscape Modelling Tool

The prototype version can be accessed at https://landscapes.wearepal.ai/

Deployment

Requirements

You will need a server with:

• Docker, running in swarm mode
• At least 2GB of RAM
• A domain name that resolves to the server's IPv4 and/or IPv6 address
• TCP ports 80 and 443 open

Instructions

• Clone this git repository onto the server, and cd into it
• Run bin/deploy
  • The first time you deploy the app, you will be prompted for configuration details
• After a minute or two, the app should come online
• Visit the web app at its configured address, and create a user account from the menu in the top-right of the page
• To shut down the app, run docker stack rm landscapes

Customisation points

• The app uses the Caddy web server: a Caddyfile is automatically generated by the deploy script and stored in a docker config named landscapes_caddyfile
  • To reconfigure your domain, shut down the app, run docker config rm landscapes_caddyfile, and then run the deploy script again
  • Alternatively, you can provide your own Caddyfile using docker config create landscapes_caddyfile /path/to/your/Caddyfile (see Caddyfile for a template)

landscapes's Issues

Look into Playwright and see if it's worth switching to over Capybara

Implement Biodiversity component into the model view

https://records.nbnatlas.org/#tab_spatialSearch

• UK Butterfly Monitoring Scheme (UKBMS)

• Dragonfly records from the British Dragonfly Society Recording Scheme

• An 'as-complete-as-possible' Sussex Beetle Records dataset covering the period to March 2017

• UK Ladybird Survey data from iRecord

• Ladybird Survey of the UK)

Automatically build & push Docker image when a new release is created, using GitHub Actions

Add GEDI data to the app

https://trello.com/c/R27IO1fA/148-investigate-adding-gedi-data-to-the-app

Investigate differences between UKCEH component for different views

UKCEH component returns different values, I believe this is due to differences in how it is calculated (the new one considers overlap). If this is the case we should update the old view to maintain consistency.

[HOLD] Add a layer type for land-use data taken from OpenStreetMap

Or maybe a different data source for land use classification

Simplest possible implementation:

Add a new button to the top of the overlays menu in the admin tools to create an overlay taken from OSM data

The user is presented with a list of checkboxes for all the different labels OSM provides, and they select which labels they're interested in

The tool produces an overlay containing all of the polygons from OSM tagged with any of the labels the user selected, scoped to the region's extent

Implement collaborative editing for projects system

Debounce text inputs so that we don't spam other clients when we rename a project or layer

This will probably be built using Action Cable: https://guides.rubyonrails.org/action_cable_overview.html

Need to decide whether or not undo/redo history should be replicated to other users (and if so, how?) - maybe the previous changes should be part of project state and every change replicates out a new copy of the project to every listener

Users should be able to customise the available expressions in the "evaluate expression" node type

Adding custom expressions per team would be really cool, but currently we only have one expression, so this will have to wait until we have some demand for more

Move from Docker Swarm to Docker Compose

Currently we use docker in swarm mode but with a single node in the swarm. The reason for this is to use secrets. This feature is now available in vanilla docker compose. This task is to switch to docker compose so that the app is a little bit simpler.

Modelling: Use transfer API to move data between main thread & workers

Advanced - threads.js

Investigate using BSON instead of JSON for saved datasets

Did not make the release of: Saving datasets
DONE 🎉
Datasets are currently saved as a JSON format, (around 80MB per), during tests BSON format achieved around 10MB. I would like to eventually implement this but it would require finding a suitable library that can deal with float32array (seems to be a complicated data format to serialise without losing accuracy) and support backwards compability (or update all currently saved models on deploy).

add a `/legacy` url and put the old app on that

Story Map (style)

It would be good to be able to add annotations to the map. The initial version would be to be able to add points on the map and associate a str with the point.
Another possible direction would be to add this kind of annotation to an existing layer.
In the future, the annotation type may include imagery, web pages, pdfs etc.

Investigate viability of rete v2 and other libraries

Rete.js - JavaScript framework for visual programming
Rete V2 will soon be out of beta and we might want to consider upgrading in the near future.
There are other options out there for visual programming, it would be wise to consider those too before we consider switching.

Investigate weird zooming in model editor on 4k monitors

Investigate adding OS Roads data to the app

OS Data Hub

• Is it likely to be useful to us?
• What is the format of the data?
• What area and resolution does it cover?
• Can we easily get access?
• Is it encumbered by any legalities?
• What type of data layer would it be in the map view? (categorical/boolean/numeric/polygonal)
• Would we need to make changes to the map view to support this data? i.e. adding new layer settings or visualisation options etc.

Investigate adding OS Rivers data to the app

https://osdatahub.os.uk/downloads/open/OpenRivers

Model editor: CSV import node

Allow edges to be deleted if selected and user presses delete button

At the moment the only way to remove an edge is to select it at the node and then drag it away. If you have a large number of incoming edges then it seems to be be nigh on impossible to select the specific edge you want. This suggested solution would presumably work.

Give an extent a name (and save?)

Create new extents for #275

Either secure access to a mail server or set up something like mailgun

[Turbo upgrade] Handle form validation error rendering on server-side, and remove `form` Stimulus controller

Models do not recalculate correctly always - possibly if error on the page

I have a model which appears not to calculate correctly. Possibly if there is an error on the page but fails silently. Not the only answer - possibly too many inputs to a sum?

Modelling: Users can break their models into reusable "modules"

Modelling: Re-theme context menu to match main UI theme

Users can give feedback or report bugs through an interface in the web app

Add a button in the corner of the screen to a google form
Is there a trello integration to create cards from the new rows in the spreadsheet?

uncertainty in outputs (and map estimates?) heteroscedastic noise each input has it's own noise

[Segmentation] New training data interface with support for multiple labels per tile [on hold]

On hold because we're going to need image-level annotation, not tile-level, to implement weakly segmented supervision. This is a much bigger job than originally anticipated, so we will focus on deliverables for BA first

If a map layer output is given a name in model view then that should be the default in map view

At the moment, if you give a map layer a name in the model view that name is not propagated to the map view. I recommend that the model view version is saved as the map view default if the map view label is empty.

LOW PRIORITY

Modelling: Add equal and not-equal comparison nodes that work on all types of data (labellings, sets, numeric data)

Users can register/sign in by requesting a "magic link" instead of entering a password

[ON HOLD] Integrate tile conversion tool into web app

QGIS already has this functionality

• Users can upload zip files with georeferenced images in them: a new "MapTileSource" model is created to hold the metadata for these images
• Determine the bounds of the region from its sources, and enqueue map tile creation jobs appropriately
• Map tile creation jobs pull the appropriate source image(s) for the current tile and render the tile, replacing the existing tile if it exists
• Add a progress view

Modelling: "Density mapping" feature to account for both distance & density (distance map doesn't account for density and so is not super useful)

Add textures/patterns to labels so that they're easier to differentiate

Rework the invitation system

• If I invite a user who does not have an account, they receive an invitation to sign up
• If I invite a user who does have an account, they must accept the invitation before they are added to the team

Snapshot Tool: KDE

Added as an option for numerical data

links:

https://www.npmjs.com/package/fast-kde

https://observablehq.com/@d3/kernel-density-estimation

https://www.react-graph-gallery.com/density-plot

Implement "evaluate expression" node type in old modelling tool

We want a new node type that allows the user to choose from a predefined list of expressions (e.g. a^2 + 3 * b - c) from a dropdown menu.

First we should find a suitable Javascript library that can parse & evaluate expressions.

The node should automatically have 1 numeric input for each variable (a, b, c in the example above), and 1 numeric output.

When the model is run, the node should compute the value of the expression for each tile.

[Turbo upgrade] Migrate from Turbolinks to Turbo

Add a admin panel (maybe under teams?) for uploading data in the app rather than going via the command line

Nothing clever - assume that the data is well-formed and ready to use!

IDEA: Datasets aren't owned by specific teams, rather teams have a flag to indicate that they have permission to access a restricted dataset. Every team has access to generally available datasets.

[Turbo upgrade] Replace `content-loader` Stimulus controller with Turbo streaming (use Postgres ActionCable adapter)

Investigate how to implement annotation tool into new model view

This is no longer about implementing this, but working out how on earth one would go about implementing this. The deliverable is some GH issues that if completed would deliver this.

The old annotation tool (Paintbrush) https://github.com/wearepal/landscapes/blob/main/app/javascript/controllers/labelling_controller.js

This needs to be available in the new system. Currently written in Stimulus, it will require rewriting into React.

Will update this later with potential options.

There should probably be a difference between adding new layers (annotations) and modifying existing layers (what-if)

Investigate adding OS Terrain data to the app

OS Data Hub
OS Data Hub

Create identity operation to allow a preview of a data stream simply

There isn't an identity operation so there isn't a convenient way of reviewing the content of a data stream. Admittedly can be done with a numeric mask set to 1 but clunky.

LOW PRIORITY

Implement undo & redo functionality for projects view

Need to have the the projects page as the landing page temporarily for all members of the Wakehurst team

This is a suggested temporary but fast fix for dealing with the fact that the site at the moment is a mess of pages from the old and new versions etc. We definitely should not be wasting any time right now sorting that out - a suggested temporary fix is that we direct to the following landing page: https://landscapes.wearepal.ai/teams/4/projects

This can perhaps just be done manually by telling everyone to use that as the landing page for now.

Pre-defined extents. A team(?) could have pre-defined extents which they can select rather than inserting the long/lat details

Extents could be shared across teams

When using select control on project view, the node moves

When using the select control on nodes, the node moves while trying to select from the dropdown. This is mostly visual but annoying to work with.

Add land registry data layer to the app

https://trello.com/c/hQtYynZT/149-investigate-adding-land-registry-data-to-the-app

Model view is sluggish and doesn't give any indication when processing

The model view appears to take several seconds to compute any changes. Is this because it is recalculating the model? We may want to consider switching to a manual recalculation?

LOW PRIORITY