ouranosinc/raven

WPS services related to hydrological modeling

Raven : Hydrological modeling and analytics

Raven (the bird)

Raven offers processes related to hydrological modeling, and in particular, the Raven hydrological modelling framework.

Raven is an open source server project offering data collection and preparation, as well as geoprocessing and catchment delineation through the Web Processing Service (WPS) standard. Raven processes can be embedded in a graphical user interface or accessed directly from a programming environment. From Python, birdy WPSClient provides a user-friendly python interface to Raven's WPS processes for geospatial processing.

The properties of custom watersheds can be extracted from a Digital Elevation Model and a land-use database.

Raven can be compiled and installed, or simply deployed using docker. A hosted version is available at https://pavics.ouranos.ca/twitcher/ows/proxy/raven.

Documentation

Learn more about Raven in its official documentation at https://pavics-raven.readthedocs.io.

Submit bug reports, questions and feature requests at https://github.com/Ouranosinc/raven/issues

Contributing

You can find information about contributing in our Developer Guide.

Please use bumpversion to release a new version.

License

Free software: MIT license

Credits

This project was funded by the CANARIE research software program.

Hydrological models are based on the Raven modeling framework.

This package was created with Cookiecutter and the bird-house/cookiecutter-birdhouse project template.

raven's Issues

WPS for goodness of fit metrics (2)

Spotpy provides a series of metrics:

• Bias
• PBias
• Nash-Sutcliff (NSE)
• logarithmic Nash-Sutcliff (logNSE)
• logarithmic probability (logp)
• Correlation Coefficient (r)
• Coefficient of Determination (r^2)
• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
• Mean Absolute Error (MAE)
• Relative Root Mean Squared Error (RRMSE)
• Agreement Index (AI)
• Covariance, Decomposed MSE (dMSE)
• Kling-Gupta Efficiency (KGE).

Add support for parallel multi-watershed analysis (8)

How do we use parallelism to do this ?

Optimize travis build

Description

I think the Travis build would go faster with only one call to conda update. We could add a full_install to the Makefile and use that in the travis config. In this repo, another build of Python is downloaded due to the tests and docs dependencies.

Emulate MOHYSE in Raven (4)

Configure shared filesystem with ComputeCanada

WPS for frequential analysis (4)

Estimate distribution parameters
Return value for given return period.

Watershed delineation WPS (8)

This issue will probably need to be split in smaller chunks:

• Select delineation algorithm (Richard could provide more info on this)
• Add algo to PAVICS services (CRIM)
• Host a reference set of polygons to use as benchmark (ETS please provide reference dataset)
• Find a way to test the service automatically (return performance metric, eg ref)
• Support for WFS input (multiple inputs)

Gather uses of Compute Canada at ETS and U Waterloo

While we are engaging with Compute Canada, it would be a good idea to get experience from ETS and U Waterloo on how they use or expect to use tools and services on CC.

Prepare short interviews on major uses of the teams, covering major aspects to be delivered in the CANARIE project.

Emulate HMETS in Raven (16)

WPS for model parameter uncertainty (4)

WPS service/frontend widget for hydrological series visualization (8)

Processes such as raven, raven-hmets, etc return an hydrograph output reference to a netCDF file. Clicking "visualize" on this file should open the time series widget and display the time series generated by the model. For basic simulations, there is only one time series. For more complex simulations there will be multiple simulations along time across the params or basins dimensions. Ideally we would be able to display those as well, but it can be done during the maintenance year.

Add builds based off of publicly available RAVEN source code and executables

It would be great to test via Travis CI the builds and source code available at http://raven.uwaterloo.ca/Downloads.html. This would automatically inform us if any versions online are broken. Not a major priority but this could be easily done in strict Travis or with Travis-Tox.

Automate creation of thin Python client for WPS server

WPS client generator

From Statement of Work. Script automatically generates client library from WPS description, Code to automatically generate client libraries (eg python) from WPS description.

Allow users to select the hardware a process will run on, including CC HPCs

Wrap existing model calibration algorithms as WPS (8)

Challenge is to connect calibration algorithms to model description through WPS to avoid the calibration algorithm having to call WPS services to get model outputs (the overhead would kill us).

WPS interface for the HBV-EC emulator (2)

WPS for graphical representation of model fit (2)

Select land-use raster database (2)

Automatic testing for Raven

Description

Set-up travis-ci for this repo.

WPS to extract watershed properties from DEM and land use layer (8)

We absolutely need:

• Area
• Lat, lon coordinates of watershed centroid
• Mean elevation

Return as LiteralOutput ? A JSON dictionary ?

Implement WPS to analyze the DEM and return multiple properties (area, slope, etc).

Also hard-code land-use file and extract additional information (open other ticket).

Add support for DDS calibration algorithm (4)

Add to spotpy or add as an independent service.

Add support for parallel multi-model analysis (8)

This would probably be a process calling other processes to benefit from parallelism and scalability. Not clear to me how that would work.

JupyterLab access to scalable computing ressources

Allow users to launch parallel computations.

Make start fails on new install

Description

After make install (successful), make start gives the following error:

Starting application ...
Traceback (most recent call last):
File "/home/ets/anaconda3/envs/raven/bin/raven", line 11, in
load_entry_point('raven', 'console_scripts', 'raven')()
File "/home/ets/anaconda3/envs/raven/lib/python3.6/site-packages/pkg_resources/init.py", line 487, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/ets/anaconda3/envs/raven/lib/python3.6/site-packages/pkg_resources/init.py", line 2728, in load_entry_point
return ep.load()
File "/home/ets/anaconda3/envs/raven/lib/python3.6/site-packages/pkg_resources/init.py", line 2346, in load
return self.resolve()
File "/home/ets/anaconda3/envs/raven/lib/python3.6/site-packages/pkg_resources/init.py", line 2352, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
ModuleNotFoundError: No module named 'raven.cli'; 'raven' is not a package
Makefile:105: recipe for target 'start' failed
make: [start] Error 1 (ignored)

Environment

• Raven version used, if any:
• Python version, if any: Python 3.6
• Operating System: Linux Mint 18.X

Steps to Reproduce

git init (in new folder)
git clone [...]
make install
make start

Paste the command(s) you ran and the output.

See above

Additional Information

Links to other issues or sources.

CF-Compliance of hydrographs.nc output

Description

The netCDF output created by Raven has a dimension called nbasins but no matching coordinates.
From the netCDF docs:

A coordinate variable is a one-dimensional variable with the same name as a dimension, which names the coordinate values of the dimension. It must not have any missing data (for example, no _FillValue or missing_value attributes) and must be strictly monotonic (values increasing or decreasing). A two-dimensional variable of type char is a string-valued coordinate variable if it has the same name as its first dimension, e.g.: char time( time, time_len); all of its strings must be unique. A variable's coordinate system is the set of coordinate variables used by the variable. Coordinates that refer to physical space are called spatial coordinates, ones that refer to physical time are called time coordinates, ones that refer to either physical space or time are called spatio-temporal coordinates.

• Make coordinate variables for every dimension possible (except for string length dimensions).
• Give each coordinate variable at least unit and long_name attributes to document its meaning.
• Use an existing netCDF Convention for your coordinate variables, especially to identify spatio-temporal coordinates.
• Use shared dimensions to indicate that two variables use the same coordinates along that dimension. If two variables' dimensions are not related, create separate dimensions for them, even if they happen to have the same length.

So for example there could be a variable called nbasins storing the Raven basin index. It would probably make sense to rename nbasins to just basin or basin_id, but I guess that would break backward compatibility for users and it's not a big deal.

Here are other CF-compliance issues that were raised by CF-checker:

CHECKING NetCDF FILE: /group_workspaces/jasmin4/ceda_wps/production/cache/uploads/Hydrographs.nc
=====================
Using CF Checker Version 3.0.5
Checking against CF Version CF-1.6
Using Standard Name Table Version 61 (2018-11-13T11:55:45Z)
Using Area Type Table Version 9 (04 July 2018)

WARN: (2.6.1): No 'Conventions' attribute present

------------------
Checking variable: time
------------------
WARN: (3): No standard_name or long_name attribute specified

------------------
Checking variable: precip
------------------

------------------
Checking variable: basin_name
------------------
INFO: attribute cf_role is being used in a non-standard way
WARN: (2.2): Could not get typecode of variable.  Variable types supported are: char, byte, short, int, float, real, double
ERROR: (9.5): Invalid value for cf_role attribute

------------------
Checking variable: q_sim
------------------

------------------
Checking variable: q_obs
------------------

------------------
Checking variable: q_in
------------------

ERRORS detected: 1
WARNINGS given: 3
INFORMATION messages: 1

User-friendly WPS using selected watershed (4)

Description

Create a WPS with minimal user inputs:

• Name of observation product (Livneh, NRCan, DEH data)
• Click on basin outlet to identify watershed (largest possible)
• If params are given, fine, if not, automatic calibration
• Assume Canopex or USGS for streamflow obs
• Fetch watershed properties from other WPS_watershed_meta
• Set initial conditions (explicitly or defaults)
• Set start and end date explicitly

Configure load balancing and scheduling to account for user-defined external resources

Not clear what that entails exactly.

WPS for hydrological indicators (4)

EG EFlowStats
https://github.com/USGS-R/EflowStats
R package

Support for MATLAB geometries?

Description

I hadn't considered this before but will the shape formats that RAVEN needs to support include geometry files encoded in the .mat format? I'm not at all familiar with MATLAB data files and I can't seem to find any documented support for handling .mat inputs. Is this a requisite feature?

Déploiement sur serveur test

Description

Deploy raven on Ouranos test server

Ideally before December 14th.

WPS interface for the HMETS emulator (2)

Add support for simulation in ungauged catchments (8)

This can be done by running models for nearby calibrated catchments and averaging the outputs.
TODO: Clarify method and split in smaller issues.

Upload HydroSheds high-res DEM to GeoServer (1)

Confirm with Richard which DEM is used for basin delineation.

Add support for multi-model uncertainty analysis (8)

Is this a process or a workflow ?

Connect ComputeCanada accounts to PAVICS

JupyterLab instance

Install and configure JupyterLab to work with PAVICS user accounts and get access to stored data.

• General info: https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906
• Installation instructions: http://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html
• Security: https://jupyter-notebook.readthedocs.io/en/stable/security.html
• Docker container: https://github.com/mikebirdgeneau/jupyterlab-docker

Authentication wish-list:

• JupyterLab and PAVICS share the same credential system, so that logging in one allows user to switch to other without reentering credentials.
• Access to JupyterLab is granted to users through Magpie interface (as another service).

Dockerize Raven (2)

Create Dockerfile describing how to build a docker image for Raven and Ostrich as this will be critical for deployment of this bird.

RAVEN code (Version 2.8.1) is found here: http://raven.uwaterloo.ca/Downloads.html
OSTRICH code can be found here: http://www.eng.buffalo.edu/~lsmatott/Ostrich/OstrichMain.html

There is a subversion repo as well.

Virtual environment for hydrological modeling experimentation

Provide JupyterLab environment (including libraries and data) for hydrological modeling.

WPS for generic Raven framework (8)

Launch Raven using as input the config file describing a user-defined model structure.
Think about how the service will handle data inputs, since they depend on the model structure.

WPS interface for the GR4J emulator (2)

Workflow example for model calibration and simulation (2)

For past and future conditions.

Create gauged catchments data set (8)

Description

Input data to #18

Upload land-use raster database to GeoServer (2)

Check metadata is clean and CF conformant.

WPS interface for the MOHYSE emulator (2)

Configuring Raven GDAL and Rasterio

I hoping to finalize the Raven GIS class methods in the coming weeks but I'm currently blocked on an issue.

In order to more easily perform raster masking using vectors (extracting points within the contour and setting outside pixels to NaN's) most of the literature suggests that Rasterio would be a robust library to perform this. Unfortunately, Rasterio and native GDAL don't work well together; they can't easily be installed alongside each other even with Anaconda.

My question here would be what seems like a more worthwhile goal: adding some spaghetti code methods cobbled together that seem to do an okay job for now, or configuring Rasterio to to work in its own environment (hopefully its own bird) to remove a lot of boilerplate code and leverage their functionalities. Questions of time, resources, and priorities come to mind.

In the mean time, I'll be looking around for other options (system calls to bash libraries that can more or less do this). Hopefully I can find something that isn't too resource-heavy.

Thoughts?

User interface for watershed delineation service (4)

User story

User wants to run an hydrological model on a catchment. To do that, she needs information about that catchment. She goes to the PAVICS UI and in Maps Control, selects "Watershed delineation". She then clicks on the map, which calls the shape-selection process from raven, passing a longitude, latitude tuple. With a double-click, option collect-upstream is set to True. The process returns a geojson object with the watershed outline, which is then displayed on the map. Hovering on the selected region displays a panel with watershed properties. The name property can be edited by the user. These properties are provided by the watershed-properties process in raven as a json file. These properties, as well as the lats and lons, can be used as inputs to raven-gr4j-cemaneige and regionalization processes. They can also be saved in the user workspace.

• Add Watershed delineation to Map Controls widget
• Call shape-selection on click, passing lat, lon from click event. Store geojson output and watershed id.
• - [ ] Call shape-selection on double-click, passing lat, lon from click event and collect-upstream=True. Store geojson output and list of watershed ids.
• Call watershed-properties on click, passing ids of watersheds. Store json dictionary of properties.
• On hover over the selected region, display panel showing the json property dictionary, with the name property being editable.
• Mechanism to pass the properties and lat, long to the process launcher.
• Add DEM from Geoserver to the list of basemaps.
• Store the geojson and properties in the user workspace.

Should self.name refer to self._name?

raven/raven/models/basic.py

Line 109 in f04c32f

self.name = name

@julemai Reading over and linting the model code a bit. Wondering if this variable should actually be the intitialized self._name.

Raven progress file

Description

An option in Raven (and Ostrich) to create an output file storing the progress % from 0 to 100 and the estimated number of seconds until the process completion. If these could be saved in a json file, e.g. something like:

{
	"% progress": 65,
	"seconds remaining": 123
}

It doesn't have to be super accurate, especially for short jobs, just give a rough idea.

On our end, we'd be able to read this file once in a while during execution to tell the user how far along is the simulation.

Pass model arguments directly in Raven.run

Description

At the moment we need to specify which RV file each parameter belongs to. Since parameter names are unique, this is not necessary.