open-eo/openeo-grassgis-driver

OpenEO driver for GRASS GIS/actinia. Backend description at https://openeo.mundialis.de/.well-known/openeo

The OpenEO GRASS GIS driver

The OpenEO GRASS GIS driver implements the openEO Core API interface for the GRASS GIS as a Service (actinia-core; available from https://github.com/actinia-org/actinia-core; DOI: https://doi.org/10.5281/zenodo.5879231) software solution for parallel, large scale geodata processing.

It is a highly scalable REST interface to process geodata with the GRASS GIS in a distributed environment. GRASS GIS is a free and open source software package providing geospatial processing engines in a single integrated environment for raster, vector, and 3D-voxel processing as well as image processing capabilities.

It is deployed on openeo.mundialis.de and will be used for processing all openEO API calls that are send to the OpenEO GRASS GIS driver.

What is openEO?

openEO - A Common, Open Source Interface between Earth Observation Data Infrastructures and Front-End Applications is an H2020 project funded under call EO-2-2017: EO Big Data Shift, under grant number 776242. The project runs from Oct 2017 to Sept 2020.

http://openeo.org/

Purpose of this document

This document demonstrates the application of the openEO Actinia Core wrapper to solve the three use cases that were defined by the development group for the first prototype:

https://open-eo.github.io/openeo-api/examples-poc/

Installation

Please refer to https://open-eo.github.io/openeo-grassgis-driver/

Use cases

Please refer to https://open-eo.github.io/openeo-grassgis-driver/#openeo-use-case-1

openeo-grassgis-driver's Issues

Make it more easy to change config values

Currently all configuration is done in config.py. When changing a value, an already built python wheel needs to be rebuild. It would be better to outsource the values to a real config file which is read during startup/runtime.

Implement CORS

I tried to access the back-end at http://openeo.mundialis.de with the browser-based JavaScript client.

Unfortunately, I can't access it. It seems CORS is not implemented as I get:

Access to XMLHttpRequest at 'http://openeo.mundialis.de/api/v0.4/' from origin 'http://editor.openeo.org' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

It would be great to get CORS support as specified at https://open-eo.github.io/openeo-api/v/0.4.1/cors/

You can easily check it yourself with the Web Editor at http://editor.openeo.org .

Clean set up, should add gdal to requirements.txt

I'm starting out on a completely fresh conda environment, following the Installation directions in https://open-eo.github.io/openeo-grassgis-driver/#installation

I am on Windows 10, using conda environments instead of virtualenv, just because I am familiar with it and by default virtualenv did not work for me.

I am failing tests because gdal is not included under https://github.com/Open-EO/openeo-grassgis-driver/blob/master/requirements.txt

I'm working out where in the process to install those for this to work.

Log request to this driver fails within openeo-python-client and openeo-webeditor

I have tested the logs functionatliy within the openeo-python-client and the openeo-webeditor. But the logs request of a job fails with an internal server error: TypeError: Object of type EoLink is not JSON serializable.

[2021-10-02 18:55:17,249] ERROR in app: Exception on /api/v1.0/jobs/user-job-6f60f678-1dee-42af-864e-ebbfa1f3364b/logs [GET]
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib/python3.8/site-packages/flask_restful/__init__.py", line 468, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/openeo_grass_gis_driver/authentication.py", line 76, in decorated
    return f(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/flask_restful/__init__.py", line 583, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/openeo_grass_gis_driver/jobs_job_id_logs.py", line 87, in get
    return make_response(jsonify(job_logs), 200)
  File "/usr/lib/python3.8/site-packages/flask/json/__init__.py", line 370, in jsonify
    dumps(data, indent=indent, separators=separators) + "\n",
  File "/usr/lib/python3.8/site-packages/flask/json/__init__.py", line 211, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.8/site-packages/flask/json/__init__.py", line 100, in default
    return _json.JSONEncoder.default(self, o)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type EoLink is not JSON serializable

temporal intervals are left-closed

See Open-EO/openeo-processes#331 (comment)

Check if handling of temporal intervals is correctly implemented in the processes load_collection and filter_temporal

Path variables are not enclosed in curly braces

From https://openeo.mundialis.de/api/v0.3/:

{
  "endpoints": [
    {
      "methods": [
        "GET"
      ], 
      "path": "/processes/<string:process_id>"
    },
...

From the API spec:

Variables in the paths must be placed in curly braces, e.g. {process_id}

While it doesn't really matter for a human, this is important for the openEO Hub: It can only detect functionalities as being supported when the paths are correctly formatted. The Hub tolerates different parameter names, e.g. your prefix with string: is okay. But the enclosing must follow the spec. Otherwise the functionality will be incorrectly reported as non-supported.

BTW your GET //collections/<string:name> endpoint has two leading slashes, probably a typo?

Docker build fails in the testing phase (test_list_strds, test_6_graph_apply)

Running the docker build following the documentation fails with the following errors during the testing phase:

git clone https://github.com/Open-EO/openeo-grassgis-driver.git
cd openeo-grassgis-driver
docker build -f docker/Dockerfile -t openeo_grass_gis_driver .

...

=================================== FAILURES ===================================
___________________ ActiniaInterfaceTestCase.test_list_strds ___________________

self = <test_actinia_interface.ActiniaInterfaceTestCase testMethod=test_list_strds>

    def test_list_strds(self):
        iface = ActiniaInterface(self.gconf)
        status, layers = iface.list_strds(
            location="nc_spm_08", mapset="landsat")
        pprint(layers)

        self.assertEqual(status, 200)
>       self.assertEqual(len(layers), 1)
E       AssertionError: 4 != 1

tests/test_actinia_interface.py:47: AssertionError
----------------------------- Captured stdout call -----------------------------
['lsat5_1987', 'lsat5_1987_10', 'lsat5_1987_30', 'lsat5_1987_40']
__________________ GraphValidationTestCase.test_6_graph_apply __________________

self = <test_process_graph_validation.GraphValidationTestCase testMethod=test_6_graph_apply>

    def test_6_graph_apply(self):
        """Run the validation test
        """
        response = self.app.post(
            self.prefix +
            '/validation',
            data=json.dumps(APPLY_1),
            content_type="application/json",
            headers=self.auth)
>       self.assertEqual(response.status_code, 200)
E       AssertionError: 400 != 200

tests/test_process_graph_validation.py:83: AssertionError

I have also tested 2.5.3 release, it also has the same issue.

Function array_element failing in backend

Using the array_element function in a Graph fails in the backend in t.rast.bandcalc with expression=(1.2 * data[0])
Running the following code from the python client (0.9)
def my_bandmath(data):
band1 = array_element(data, index=0)

band2 = array_element(data, index=0)

return 1.2 * band1

datacube.reduce_dimension(dimension="bands",reducer=my_bandmath)
resulting in the Graph

{
"process_graph": {
"loadcollection1": {
"process_id": "load_collection",
"arguments": {
"id": "nc_spm_08.modis_lst.strds.LST_Day_monthly",
"spatial_extent": null,
"temporal_extent": [
"2015-01-01",
"2015-03-01"
]
}
},
"reducedimension1": {
"process_id": "reduce_dimension",
"arguments": {
"data": {
"from_node": "loadcollection1"
},
"dimension": "bands",
"reducer": {
"process_graph": {
"arrayelement1": {
"process_id": "array_element",
"arguments": {
"data": {
"from_parameter": "data"
},
"index": 0
}
},
"multiply1": {
"process_id": "multiply",
"arguments": {
"x": 1.2,
"y": {
"from_node": "arrayelement1"
}
},
"result": true
}
}
}
}
},
"saveresult1": {
"process_id": "save_result",
"arguments": {
"data": {
"from_node": "reducedimension1"
},
"format": "GTIFF",
"options": {}
},
"result": true
}
}
}

Example process graphs?

Is there a set of example process graphs for demonstration purposes?

Missing OPEN-EO Functions

The following OPENEO (1.0) core functions are not available in openeo-grassgis-driver:

• aggregate_spatial
• aggregate_temporal
• aggregate_temporal_period
• all
• any
• apply_dimension
• apply_kernel
• aply_neighborhood
• arccos
• arcsin
• arctan
• arctan2
• arcsinh
• arctanh
• array_contains
• array_filter
• array_find
• array_labels
• between
• ceil
• dimension_labels
• drop_dimension
• first
• floor
• gte
• is_valid
• last
• ln
• log
• order
• pi
• rename_dimension
• rename_labels
• round
• sqn
• sin
• sinh
• sort
• tan
• tanh

use official process definitions

Currently, process definitions have been added manually to the openeo-grass driver. As in the example openeo python driver, process definitions can be directly taken from the official repo https://github.com/Open-EO/openeo-processes. This would make maintenance much easier.

Error in /collections endpoint

Hi,

when accessing https://openeo.mundialis.de/api/v1.0/collections the current server response is:

[
  {
    "description": "An internal error occurred while catching strds layers!"
  }, 
  400
]

It would be nice if:

1. the HTTP status code would not be a 200 OK but some 500, so that accessing clients treat it as "gone wrong" straight away,
2. the response was formatted as a proper error from the API spec,
3. that wouldn't happen at all of course 👼

Failed tests while running openEO GRASS GIS driver test suite

I was following the installation instructions here to deploy the GRASS GIS driver locally. Everything went fine without issues, but when running the openEO GRASS GIS driver test suite using

python3.5 setup.py test > console_out.txt

I get a number of failed tests as documented in the attached txt file. I noted that only the tests associated with "tests/test_jobs.py" and "tests/test_processes.py" failed - the others passed. Could someone please point out if this is expected or is there something I am doing wrong?

Additionally, I also get the following warning while running the line above:

/mnt/h/edu/semester_4/grass/openEO/venv/lib/python3.5/site-packages/setuptools/dist.py:407: UserWarning: The version specified ('0.1-poc.post0.dev15+g0cddcc2.dirty') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version

Thanks!
console_out.txt

Fix several issues reported by the validatior

@bgoesswe deployed a validation tool for back-ends. The results for the GRASS GIS back-end can be found here:
https://www.geo.tuwien.ac.at/openeoct/backend/validate/4

There are several issues reported:

• /collections and /collections/collection_id sends a non-JSON media type
• It seems there are issues with authentication. /credentials/basic is missing and the credentials are directly asked for at each endpoint? According to the API they just accept the Bearer Token...
• /output_formats seems to still return a response compatible only to API v0.3. Basically, just move the value for formats to the root level and remove the property default.

Problems on the data endpoint

I have used the Docker deployment approach to setup the service (also created a docker-compose file for easier deployment).
Most of the exploration endpoints are working, but the /data endpoint returns an error. I assume that I haven't downloaded the data within the Dockerfile. Can you tell me where to store the data or how to configure the service?

STRDSInfoModel: missing band_reference and band_reference for image collections

At time the image collection endpoint (https://openeo.mundialis.de/collections) gives the following error:

'  "message": "The model \\"STRDSInfoModel\\" does not have an attribute \\"number_of_bands\\"", \n'

openeo-grassgis-driver test log captured:

----------------------------- Captured stdout call -----------------------------
('{\n'
 '  "accept_datetime": "2019-11-28 13:05:36.207693", \n'
 '  "accept_timestamp": 1574946336.2076917, \n'
 '  "api_info": {\n'
 '    "endpoint": "strdsmanagementresource", \n'
 '    "method": "GET", \n'
 '    "path": '
 '"/api/v1/locations/latlong_wgs84/mapsets/modis_ndvi_global/strds/ndvi_16_5600m", \n'
 '    "request_url": '
 '"http://actinia.mundialis.de/api/v1/locations/latlong_wgs84/mapsets/modis_ndvi_global/strds/ndvi_16_5600m"\n'
 '  }, \n'
 '  "datetime": "2019-11-28 13:05:36.906189", \n'
 '  "exception": {\n'
 '    "message": "The model \\"STRDSInfoModel\\" does not have an attribute '
 '\\"number_of_bands\\"", \n'
 '    "traceback": [\n'
 '      "  File '
 '\\"/usr/local/lib/python3.6/dist-packages/actinia_core-0.99.1.post0.dev22+geb8bca3-py3.6.egg/actinia_core/resources/ephemeral_processing.py\\", '
 'line 1350, in run\\n    self._execute()\\n", \n'
 '      "  File '
 '\\"/usr/local/lib/python3.6/dist-packages/actinia_core-0.99.1.post0.dev22+geb8bca3-py3.6.egg/actinia_core/resources/strds_management.py\\", '
 'line 594, in _execute\\n    self.module_results = '
 'STRDSInfoModel(**strds)\\n", \n'
 '      "  File '
 '\\"/usr/local/lib/python3.6/dist-packages/flask_restful_swagger_2/__init__.py\\", '
 'line 337, in __init__\\n    \'The model \\"{0}\\" does not have an attribute '
 '\\"{1}\\"\'.format(self.__class__.__name__, k))\\n"\n'
 '    ], \n'
 '    "type": "<class \'ValueError\'>"\n'
 '  }, \n'
 '  "http_code": 400, \n'
 '  "message": "The model \\"STRDSInfoModel\\" does not have an attribute '
 '\\"number_of_bands\\"", \n'
 '  "process_chain_list": [\n'
 '    {\n'
 '      "1": {\n'
 '        "flags": "g", \n'
 '        "inputs": {\n'
 '          "input": "ndvi_16_5600m", \n'
 '          "type": "strds"\n'
 '        }, \n'
 '        "module": "t.info"\n'
 '      }\n'
 '    }\n'
 '  ], \n'
 '  "process_log": [\n'
 '    {\n'
 '      "executable": "t.info", \n'
 '      "parameter": [\n'
 '        "type=strds", \n'
 '        "input=ndvi_16_5600m", \n'
 '        "-g"\n'
 '      ], \n'
 '      "return_code": 0, \n'
 '      "run_time": 0.6452951431274414, \n'
 '      "stderr": [\n'
 '        ""\n'
 '      ], \n'
 '      "stdout": '
 '"id=ndvi_16_5600m@modis_ndvi_global\\nname=ndvi_16_5600m\\nmapset=modis_ndvi_global\\ncreator=mundialis\\ntemporal_type=absolute\\ncreation_time=\'2017-02-16 '
 "11:31:50.189306'\\nmodification_time='2017-05-29 "
 "21:04:13.222786'\\nsemantic_type=mean\\nstart_time='2013-01-01 "
 "00:00:00'\\nend_time='2017-05-25 00:00:00'\\ngranularity='16 "
 'days\'\\nmap_time=interval\\nnorth=90.0\\nsouth=-90.0\\neast=180.0\\nwest=-180.0\\ntop=0.0\\nbottom=0.0\\naggregation_type=None\\nnumber_of_bands=None\\nnumber_of_maps=101\\nnsres_min=0.05\\nnsres_max=0.05\\newres_min=0.05\\newres_max=0.05\\nmin_min=-2000.0\\nmin_max=-2000.0\\nmax_min=9947.0\\nmax_max=9994.0\\nraster_register=raster_map_register_73eb5d056c51424288153b87a714630a\\n"\n'
 '    }\n'
 '  ], \n'
 '  "progress": {\n'
 '    "num_of_steps": 1, \n'
 '    "step": 1\n'
 '  }, \n'
 '  "resource_id": "resource_id-21b3cb05-ef05-4780-87ea-cc62ed86fb5a", \n'
 '  "status": "error", \n'
 '  "time_delta": 0.6985125541687012, \n'
 '  "timestamp": 1574946336.9061809, \n'
 '  "urls": {\n'
 '    "resources": [], \n'
 '    "status": '
 '"http://actinia.mundialis.de/api/v1/resources/openeo/resource_id-21b3cb05-ef05-4780-87ea-cc62ed86fb5a"\n'
 '  }, \n'
 '  "user_id": "openeo"\n'
 '}\n')
________________________ DataTestCase.test_collections _________________________

self = <test_collections.DataTestCase testMethod=test_collections>

    def test_collections(self):
        """Test the return of raster and vector maps in the north carolina test dataset
    
        :return:
        """
        response = self.app.get('/collections', headers=self.auth)
        data = json.loads(response.data.decode())
    
        pprint(data)
    
        dsets = ["nc_spm_08.landsat.raster.lsat5_1987_10",
                 "nc_spm_08.PERMANENT.vector.lakes",
                 "nc_spm_08.PERMANENT.raster.elevation"]
    
        data_id_list = []
    
>       for entry in data["collections"]:
E       TypeError: list indices must be integers or slices, not str

tests/test_collections.py:32: TypeError
----------------------------- Captured stdout call -----------------------------
[{'description': 'An internal error occurred while catching strds layers!'},
 400]

Discovery endpoints must not request authentication

Currently, when I open https://openeo.mundialis.de/api/v0.3/ I am prompted:

One should be able to discover a backend without the need to sign up. I.e. the endpoints

• /
• /collections
• /collections/{collection_id}
• /processes
• /output_formats
• /service_types
• /udf_runtimes (new in v0.4)

should be reachable without authentication.

For example, this is essential for the Hub -- password-protected backends can't be crawled.

Endpoint /collections is extremely slow

Trying to access https://openeo.mundialis.de/api/v1.0/collections leaves one waiting and waiting, until after a minute there is a 504: Gateway time-out error. Can this be sped up? Because at the moment, the user cannot know which collections are available...

OPEN-EO math functions "add,divide,multiply" used via apply broken in backend

the official OPEN-EO math functions "add,substract,divide,multiply" deliver wrong results.
Calling the functions "add,substract,divide,multiply" via apply for single Band Collection and with 2 as second parameter y
results in identical internal grass call to t.rast.mapcalc with expression=(2) and resulting outputs completely filled with value 2

SubGraph: for apply :
"apply1": {
"process_id": "apply",
"arguments": {
"data": {
"from_node": "loadcollection1"
},
"process": {
"process_graph": {
"add1": {
"process_id": "add",
"arguments": {
"x": {
"from_parameter": "x"
},
"y": 2
},
"result": true
}
}
}
}
}

simpleAddOpenEO-ExampleGraph.txt

invalid path for logging

At https://github.com/Open-EO/openeo-grassgis-driver/blob/master/src/openeo_grass_gis_driver/utils/logging.py#L18 the default path for the log file is "/openeo-grassgis-driver.log" which should be invalid on most systems: never ever create a file directly at /.

Support Sentinel-2 Level-2A data (with AUX bands) as collection

Currently, a Sentinel-2 collection is expected to have exactly 13 bands: https://github.com/Open-EO/openeo-grassgis-driver/blob/master/src/openeo_grass_gis_driver/collection_information.py#L209

This however only applies to Level-1C data, as Level-2A does not have a B10 anymore.
Instead, Level-2A data contains additional auxiliary bands (AOT, SCL, TCI, WVP). It would be required to

• allow for Sentinel-2 scenes to contain only 12 bands
• decide on whether to support the auxiliary bands including band metadata

Missing bbox coordinate breaks fetching of collection details

I just tried latlong_wgs84.PERMANENT.strds.ndvi_16_5600m once more (see last paragraph in #49 (comment)), this time receiving a response after 1856ms, which was the 500 error (live example).

The error page read:

ValueError: could not convert string to float: 'None'

And in the (very long) traceback the last entry was:

File "/usr/local/lib/python3.6/dist-packages/openeo_grass_gis_driver-0.0.1-py3.6.egg/openeo_grass_gis_driver/collection_information.py", line 132, in get
extent = CollectionExtent(spatial=(float(layer_data["west"]), float(layer_data["south"]),

So I guess it's simply a missing coordinate in the bbox?

Windows installation: KeyError os.environ["HOME"]

I'm failing several of the tests on a Windows install, because Windows uses "USERPROFILE" instead of "HOME".

I'm not sure if this is designed to run with Windows or not, for now I am going to replace the code on my end with this:

import os.path
home_folder = os.path.expanduser('~')

(src: https://stackoverflow.com/questions/14742064/python-os-environhome-works-on-idle-but-not-in-a-script/33935163)

Sync GH release version to backend_version

The deployed grassgis-backend returns

{
  "api_version": "1.0.1",
  "backend_version": "0.2.0",
  "description": "GRASS GIS Driver",
  "endpoints": [
...

while the current github release is 1.0.2. This should be in sync.

Endpoint /collections completely broken and returning error in wrong format

Currently, https://openeo.mundialis.de/api/v1.0/collections replies with:

[
  {
    "description": "An internal error occurred while catching mapset from location nc_spm_08!"
  }, 
  400
]

Aside from that it would be nice if this error wouldn't be there at all (:wink:), there are a couple of issues with that error response:

1. This is not a JSON error object as described in the Error Handling paragraph of the spec. That paragraph is only indicated as "SHOULD", but following it would be nice nonetheless :)
2. When I assume the number there (400) is meant to be a HTTP status code, it is a wrong one because "internal error" should result in something in the 5xx range.
3. The actual HTTP status code returned is 200 OK.