kartoza/geocontext

A django app to retrieve context for spatial feature

GeoContext

A django project to retrieve information of a point.

View a running instance at http://geocontext.kartoza.com

The latest source code is available at https://github.com/kartoza/geocontext.

Project activity

• Current test status master: | | Coverage

• Current test status develop: | | Coverage

Key features

• Stateless and easy to deploy
• Able to retrieve geo-context information of a point from several services
• Optimized for rapid information retrieval
• Currently supports WFS, WMS, ArcREST and Placename sources
• Using cache mechanism to retrieve context frequently requested information
• Three level hierarchy (service --> group --> collection) to sort services to request
• Simple API returning json data

API

After you have working geocontext instance, you can then check the available API from the API documentation links (you can find it in the main page) or see the content and the hierarchy of context service (you can also find the link in the main page)

V2

New query endpoint accepting GET keyword parameters: /api/v2/query?

Keyword	Description	Details
registry	service, group or collection	REQUIRED
key	Unique key of the registry	REQUIRED
x	Longitude in DD, DMS or DD	REQUIRED
y	Latitude in DD, DMS or DD	REQUIRED
srid	Coordinate reference	OPTIONAL (Default 4326)
tolerance	Query tolerance in m	OPTIONAL (Default 10m)
outformat	Output format (geojson / json)	OPTIONAL (Default geojson)

Quick Installation Guide

For deployment we use docker so you need to have docker running on the host. GeoContext is a django app so it will help if you have some knowledge of running a django site.

1. Clone the project

git clone git://github.com/kartoza/geocontext.git && cd /docs/

2. Ensure shared volume permissions are set

make permissions

3. Copy the sample.env file as .env and change the values inside if necessary.

4. Copy docker-compose.override.sample.yml as docker-compose.override.yml and change the values inside if necessary.

5. Setup and run the web service: available on http://localhost/

make setup-web

6. If needed, create a superuser for Django admin access on http://localhost/admin/ - here you can manually modify data

make superuser

Loading Data GeoContext use a json file to populate the context service and its group and collection. Everything is stored in geocontext.json. You can load it by running this command:

make import-data

Or if you have your own json file, you can load it with:

make import-data FILE_URI=path/to/your/json/file

Be careful, it will replace your context service data (and its hierarchy) with your new one from the json file. But no worries, when your run this import-data command, it will export your context service data to a new json file.

You can also export your context service to json file by running:

make data-export

Developers quick start with Docker and VSCode

An easy way to set up a locally development environment is with Docker and VSCode.

1. Optional: add any ENV variables needed to for uwsgi & dvweb to /deployment/.env
2. Use setup-dev to build the production and development containers, create superuser, and import default services.

make setup-dev

3. Geocontext should now be running at localhost.
4. DB can be accessed at port 'localhost:25432', user&password='docker', database=gis
5. Attach a VSCode session by right clicking on the running geocontext_devweb container in the VSCode remote - Containers extension
6. Now we can get into the running containers to run management commands, run another server at another port, debug etc.
7. In the geocontext_devweb container terminal you can run a test server for debugging as follows:

python manage.py runserver 8001

9. Pytest is used for testing. Unit & integration tests, flake8 linting, coverage and documentation can be run with:

make test-full

10. See README-legacy for PyCharm and SSH access to devweb container.

Deployment

1. Config git repo/branch and dockerhub/image:tag to use for deploy in top of Makefile
2. Run make command to build and push image

make deploy

• Note the UWSGI image must be available (setup-dev / setup-web)

Participation

We work under the philosophy that stakeholders should have access to the development and source code, and be able to participate in every level of the project - we invite comments, suggestions and contributions. See our milestones list and our open issues list for known bugs and outstanding tasks.

Credits

GeoContext was funded by JRS and developed by Kartoza.com, Freshwater Research Center and individual contributors.

License

GeoContext is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 (GPLv3) as published by the Free Software Foundation.

The full GNU General Public License is available in LICENSE.txt or http://www.gnu.org/licenses/gpl.html

Disclaimer of Warranty (GPLv3)

There is no warranty for the program, to the extent permitted by applicable law. Except when otherwise stated in writing the copyright holders and/or other parties provide the program "as is" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The entire risk as to the quality and performance of the program is with you. Should the program prove defective, you assume the cost of all necessary servicing, repair or correction.

Thank you

Thank you to the individual contributors who have helped to build GeoContext:

• Tim Sutton (@timlinux) : [email protected]
• Ismail Sunni (@ismailsunni)
• Dimas Tri Ciputra (@dimasciput) : [email protected]
• Andre Theron (@andretheronsa) : [email protected]
• Rizky Maulana Nugraha (@lucernae) : [email protected])

geocontext's Issues

We need to update the API to require an API key

Currently the API is open for anyone to use. Please update the API so that it requires and API access token.

We need to also have a form that will let users generate a new API key.

Refactor code to have each query type as its own class

Each of the query types (WFS, WMS, ARCRest etc) are better suited as their own classes for diverse implementation

We need to improve way to report errors

Problem

@dimasciput was trying to access the following URL https://geocontext.kartoza.com/api/v1/geocontext/value/group/30.77825/-24.73478/eco_geo_class/

But in fact, it needed to be https://geocontext.kartoza.com/api/v1/geocontext/value/group/30.77825/-24.73478/eco_geo_group

The request gave a 404 but no definitive error message.

Whilst interacting with the API we expect users to know the contents of the json file. It would be awesome to change our input boxes from free text to use autocomplete so that users can generate the correct URL

Populate geocontext service registry regulary

Problem

Whilst solving some urgent issues on healthy rivers we are defining groups, registry layer directly using the admin interface.

We need to keep track of what has been added to the admin site so that we can add it in the json file so that the two do not diverge.

@dimasciput please point me to any new layers you have defined in the admin page as well as group definitions so that I can populate the json regularly.

Switch the Content dropdown to match the hieararchy and remove prefix `context`

Assemble data to use future sites

Problem

As a side project can you start assembling africa-wide data for geocontext and gisbox use
more for the former particularly things that are useful in the healthsector
the idea would be to take the tech that we have built for freshwaterbiodiversity and apply it on healthsites.io so that we can assign clinics / hospitals with contextual data
and then add slice and dice capabilities like Dimas has build on healthy rivers

e.g. Geospatial World/

First high-resolution land cover map of Africa by ESA - View the images!
The map has been created using a year of data gathered from Sentinel-2A satellite. And at a resolution of 20 m per pixel, we can explore from Zimbabwe
Oct 10th, 2017

Particularly things that relate to health like urban vs rural, malaria areas, etc

Message 1-kartoza-chat

Improve code to cater for all geometry categories

Problem

If you query the geocontext it doesn't return all the data mainly because the code is hardcoded to assume the geometry column in the database is called geom.

The group definition is depicted here

The code that breaks thing up is here:
https://github.com/kartoza/geocontext/blob/develop/django_project/geocontext/utilities.py#L54

Solution

• We need a better way to identify the geometry column generated from the gml.
• We should not let users change their data to match the service, rather have the service to be dynamic

Needs container running resilio sync so backups may be moved off the server

Needs container running resilio sync so backups may be moved off the server - Specifically relating to DB backups in this instance.

Add field allowing one to decide if a GeoContextRegistry/Group needs to be printable as a graph

Currently this is brute forced in reverse detecting if a current field is present. This should rather be explicitly stated in the DB.

Populate Service

Populate the json for geocontext service. Please follow this sample:
https://github.com/ismailsunni/geocontext/blob/hierarchy_2/django_project/base/management/commands/geocontext.json

Let me know if you need my help

Add new data types to geo context and populate the service

Problem

Add the following layers to geoserver

• Altitude
• Aspect
• steepness
• Ruggedness

Solution

Populate the json file #33

cc @timlinux

Extent models to include graphable column

Problem

I am trying to import data into the geocontext service and I get the following error.

   Number of Context Group 17
   Number of Context Collection 5
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: column geocontext_contextgroup.graphable does not exist
LINE 1: ...ription", "geocontext_contextgroup"."group_type", "geocontex...

Solution

Make happy
^

Report for collection and layer groups

Problem

do we have a report that lists a full collection, its group, and their layers?

And also include metadata for layers

Placename search does use all supported genome search types

Geonames has an extended list of queries one is able to run returning either JSON or CSV. By implementing their data structure more completely all of these could be utilised.

https://www.geonames.org/export/web-services.html

Update geocontext

• Update geocontext data
• Update context layers for healhtyriver and ledet

Add group type to group context

GeoContext API should return GeoJSON

I just realised that the document we get back is not a geojson document.

It would be interesting to see if we can also / instead deliver it as a GeoJSON document so that it can be used easily in web mapping clients. I don't know if it would work well given the nested structure of the geocontext document though.

SSL Support needed

Please ask @dimasciput to show you how he sets up certbot and configure and deploy if for GeoContext.

Deployment and documentation thereof

Ensure the deployment method is functioning as intended and the developer technical documentation reflects the process needed clearly.

We need a gazetteer lookup

This will return a placename given a coordinate pair.

Have a look at geonames.org

Use a django query to find the nearest named place to the point.

For extra points return a relative location e.g. 12km SSE of Swellendam

This will need separate semantics to how the other layer types are defined.

Simple View to List All CSR, Groups, Collections and their descriptions

We need a more beautiful landing page

Problem

We can do better than this right?

Proposed solution

• Add some margins on the left and right of the page so it is not full width
• Add some white space between the menu and content
• Put a nice big icon in the jumbotron

• Add some informative text as below:

GeoContext is a service that will fetch interesting data for a geographic position. It is an API that you can use to integrate this information into your own web site. The API returns JSON documents which enumerate any information found for the point you specify.

GeoContext can connect to various different types of services such as WMS and WFS to collate the point information. You can configure more services as needed and collate a number of services into a group. A number of groups can be combined into a collection.

Currently, the service is free to use, in the future we may charge a subscription to use it.

Populate json files with missing temperature layers and boundary layers.

Problem

it would be good to have:

• Country
• Province
• District
• Municipality
• Parcel / Farm

All these layers are available from maps.kartoza.com and will be added to the json file.

Some other layers are missing for climate

The monthly mean temp and monthly mean stddev temp that are missing months

cc @timlinux

Remove Cache from the API

Reasons:

• bad for privacy
• it can make your browser crash

Requirements file has security issues

Problem

GitHub is currently reporting that there are security issues with our requirements file(s).

See https://github.com/kartoza/geocontext/network/alerts

Proposed solution

The issue is probably relating to requests which is used by this project. Please review the issue and make the needed fixes.

Add Import Tool in the UI

There is already in the management command, but it will be convenient if we have also in the UI.

Documentation for end user

Getting started and Info page allowing quick and easy implementation by the end user

Should include:

• Snippet showing the end user [Insert this JS code] get {This JSON Response}

Extend service to allow for the lookup of nearest prominent data piece (city, river, etc)

Extend service to allow for the lookup of nearest prominent data piece (city, river, etc)

Give more descriptive names to groups

gavin [5 minutes ago]
I think the names of these graphs should be changed to something more descriptive for each one e.g. ‘historic rainfall’ or ‘mean monthly rainfall’ with no reference to the word ‘raster’ in names (it means nothing to users) - pls make a ticket

Regex filter for geocontext layers do not work in some instances

Problem

When we populating the geocontext service the regex is not working consistently especially for WFS layers.

When the regex is defined as gml:name it consistently work provided the data has a column with name.

If the data has a column gml:category the service does not work.

In some instances the regex has to be kartoza:category.

Solution

We need to identify and get the service working consistently especially when populating the json.
When do we use kartoza:column_name vs gml_column_name vs gml:name

Fix geocontext not getting some data

Data like average monthly rainfall is missing, find the issue and fix it.

Update of front page broke the visuals for the drop down box on some of the other pages

When accessing the API it looks as follows:

Proposed solution: Simply need to increate the z-index in the css. As for the text being cut off - a place holder html to stand behind the "position: fixed" menu should stop content from rendering behind it.

GeoContext api not returning "graphable" value

GeoContext api not returning "graphable" value

Create GeoContext Plugin for Open Layer

Support for climate data

We have another use case where we will receive numerical data back e.g. temperature, rainfall etc. And another source type that we should support so that we can fetch data from ArcGIS Rest services. I would like @NyakudyaA to register these layers as. 'Climate' context group.

Elsenburg publishes all the climate layers as ArcGIS Rest services here:

https://gis.elsenburg.com/arcgis/rest/services/Climate/MapServer

You can easily add them to QGIS like this:

Or to OpenLayers like this:

https://openlayers.org/en/latest/examples/vector-esri.html

And we can do point queries from python like this

https://github.com/Schwanksta/python-arcgis-rest-query

How to test arc rest services

Problem

When importing data into the service the following message

Load Context Groups....
   Context Group A test of the placename type is loaded
No CSR registered for arc_rest_test_is
No CSR registered for arc_rest_test_the
No CSR registered for arc_rest_test_best

Usually indicates that the service was not successfully imported.

an example in the app

Working in the app

Add Import and Export for Model/Database

• Create a tool to export the database (excluding the cache) to a json file (or another readable format).
• Create a tool to populate the models/db based on that file.

Implement Context Service Registry Hierarchy

For example:
Country --> Province --> District --> Sub District

Add informative key names for arc gis service rest

Problem

The geocontext service is identified by keys and names. Please add informative keys and names when populating the JSON.

Solution

Make it happy

We need to support ESRI rest services

There are many ESRI servers out there - we should support using them as a layer source too.

See e.g. https://portal.environment.gov.za/server/rest/services/Corp/ProtectedAreas/MapServer/ for an example server we could be using.

Provenance, Notes, and Licensing for CSR

DockerHub container for nginx service needs to be moved from Dimas' account to kartoza's

DockerHub container for nginx service needs to be moved from Dimas' account to kartoza's

Improve rendering issues with Geocontext

Problem

Currently, we have problems with rendering issues in Geocontext and this usually results in a timeout. There are some layers that are naturally big and I have tried a couple of solutions in order to make them faster
ie

• Materialize all the tables and reindex them
• Drop uneccesary columns from the matrialized views.
• Check the bottleneck for fetching layers - whether it is the latency from fetching them from postgis.kartoza.com
  
• Created geopackage layers for the troublesome layers and use them in geoserver - because geoserver would fetch them locally in the data dir - still this does not work
• Simplify geometries

Solution

I think we could improve our code on Geocontext by using WFS filters in our code

example : http://maps.kartoza.com/geoserver/kartoza/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=kartoza:northern_cape_cba&cql_filter=INTERSECTS(geom, POINT (20.84 -29.32))&outputFormat=application/json

This should fetch only that single features.

Disadvantages: The CQL filter assumes that the layers geometry and the point geometry should be the same CRS.
Our geocontext code should handle the layer reprojection.
We could also translate the geometry when we create the views

Tweak Landing Page

Just remove unrelated stuff

We need to get the river name that is closest to a point feature in Geocontext

Problem

We would like to associate a site with the name of the river closest to it.

Solution

Not all rivers have names from the database.

We are going to create a PostgreSQL function and then from geocontext we call that function to get the name of the river

I have added a PostgreSQL function to do this

CREATE OR REPLACE  FUNCTION   finder (latitude double precision, longitude double precision) RETURNS character varying
    LANGUAGE plpgsql
    AS $$
  DECLARE 
	river_name character varying;
  BEGIN
    SELECT entity_name  INTO river_name
	FROM hydr_lines_exp
	WHERE entity_name is not null
	ORDER BY wkb_geometry <-> st_transform((st_setsrid(st_makepoint(latitude,longitude),4326)),3857)
	LIMIT 1;
	RETURN river_name;
	

  END
$$;

Then from the database client, we just call select finder(24.23,-29.79)

cc @dimasciput

Support for Privileged Access

See : kartoza/LEDET_BIMS#83

We need a way to clear the cache in geocontext periodically

GeoContext needs to be optimised in order to return the requests more quickly.

This is causing bottlenecks in the data queue which seem to be leading to other threads crashing.

Need to look at how to minimise the response and request times for these services.

We need a favicon

Please use the final icon from #43 to make a favicon for the site and make sure it is used everywhere.

API to Retrieve All Services Registry in a Collection / Groups