ropensci/waterinfo

R package to download time series data from waterinfo.be

wateRinfo

wateRinfo facilitates access to waterinfo.be, a website managed by the Flanders Environment Agency (VMM) and Flanders Hydraulics Research. The website provides access to real-time water and weather related environmental variables for Flanders (Belgium), such as rainfall, air pressure, discharge, and water level. The package provides functions to search for stations and variables, and download time series.

To get started, see:

• Get started: an introduction to the package’s main functionalities.
• Function reference: overview of all functions.
• Articles: tutorials on how to use the package.

Installation

You can install the development version of wateRinfo from GitHub with:

# install.packages("devtools")
devtools::install_github("ropensci/wateRinfo")

# Or rOpenSci R-universe
install.packages("wateRinfo", repos = "https://ropensci.r-universe.dev")

Example

For a number of supported variables (documented by VMM), the stations providing time series data for a given variable can be listed with the command get_stations().

If you want to know the supported variables, ask for the supported variables:

library(wateRinfo)

supported_variables("en")
#>              variable_en
#> 1              discharge
#> 6        soil_saturation
#> 7          soil_moisture
#> 8  dew_point_temperature
#> 9     ground_temperature
#> 10           ground_heat
#> 11            irradiance
#> 12          air_pressure
#> 13 air_temperature_175cm
#> 14              rainfall
#> 20     relative_humidity
#> 21  evaporation_monteith
#> 25    evaporation_penman
#> 29        water_velocity
#> 34           water_level
#> 39     water_temperature
#> 40        wind_direction
#> 41            wind_speed

Listing the available air pressure stations:

get_stations("air_pressure")
#>      ts_id station_latitude station_longitude station_id station_no
#> 1 78124042         51.20300          5.439589      12213   ME11_002
#> 2 78005042         51.02263          2.970584      12206   ME01_003
#> 3 78039042         51.24379          4.266912      12208   ME04_001
#> 4 78073042         50.88663          4.094898      12210   ME07_006
#> 5 78107042         51.16224          4.845708      12212   ME10_011
#> 6 78022042         51.27226          3.728299      12207   ME03_017
#> 7 78090042         50.73795          5.141976      12211   ME09_012
#> 8 78056042         50.86149          3.411318      12209   ME05_019
#>              station_name stationparameter_name parametertype_name
#> 1             Overpelt_ME                    Pa                 Pa
#> 2               Zarren_ME                    Pa                 Pa
#> 3              Melsele_ME                    Pa                 Pa
#> 4           Liedekerke_ME                    Pa                 Pa
#> 5            Herentals_ME                    Pa                 Pa
#> 6            Boekhoute_ME                    Pa                 Pa
#> 7 Niel-bij-St.-Truiden_ME                    Pa                 Pa
#> 8              Waregem_ME                    Pa                 Pa
#>   ts_unitsymbol dataprovider
#> 1           hPa          VMM
#> 2           hPa          VMM
#> 3           hPa          VMM
#> 4           hPa          VMM
#> 5           hPa          VMM
#> 6           hPa          VMM
#> 7           hPa          VMM
#> 8           hPa          VMM

Each of the stations in the list for a given variable, are represented by a ts_id. These can be used to download the data of a given period with the command get_timeseries_tsid(), for example Overpelt (ts_id = 78124042):

overpelt_pressure <- get_timeseries_tsid("78124042", 
                                         from = "2017-04-01", 
                                         to = "2017-04-02")
head(overpelt_pressure)
#>             Timestamp  Value Quality Code
#> 1 2017-04-01 00:00:00 1008.8          130
#> 2 2017-04-01 00:15:00 1008.7          130
#> 3 2017-04-01 00:30:00 1008.7          130
#> 4 2017-04-01 00:45:00 1008.6          130
#> 5 2017-04-01 01:00:00 1008.5          130
#> 6 2017-04-01 01:15:00 1008.4          130

Making a plot of the data with ggplot2:

library(ggplot2)
ggplot(overpelt_pressure, aes(x = Timestamp, y = Value)) + 
    geom_line() + 
    xlab("") + ylab("hPa") + 
    scale_x_datetime(date_labels = "%H:%M\n%Y-%m-%d", date_breaks = "6 hours")

Another option is to check the available variables for a given station, with the function get_variables(). Let’s consider again Overpelt (ME11_002) and check the first ten available variables at the Overpelt measurement station:

vars_overpelt <- get_variables("ME11_002")
head(vars_overpelt, 10)
#>    station_name station_no    ts_id    ts_name parametertype_name
#> 1   Overpelt_ME   ME11_002 78522042 HydJaarMax                 Ts
#> 2   Overpelt_ME   ME11_002 78523042 HydJaarMin                 Ts
#> 3   Overpelt_ME   ME11_002 78693042       P.15                 Ud
#> 4   Overpelt_ME   ME11_002 94682042   MaandMin                 Ta
#> 5   Overpelt_ME   ME11_002 78531042       P.10                 Ts
#> 6   Overpelt_ME   ME11_002 78518042     DagGem                 Ts
#> 7   Overpelt_ME   ME11_002 78521042 HydJaarGem                 Ts
#> 8   Overpelt_ME   ME11_002 78524042 KalJaarGem                 Ts
#> 9   Overpelt_ME   ME11_002 78533042       P.60                 Ts
#> 10  Overpelt_ME   ME11_002 78694042      Pv.15                 Ud
#>    stationparameter_name
#> 1                  SoilT
#> 2                  SoilT
#> 3                   WDir
#> 4                     Ta
#> 5                  SoilT
#> 6                  SoilT
#> 7                  SoilT
#> 8                  SoilT
#> 9                  SoilT
#> 10                  WDir

Different pre-calculated variables are already available and a ts_id value is available for each of them to download the corresponding data. For example, DagGem (= daily mean values) of RH (= relative humidity), i.e. ts_id = 78382042:

overpelt_rh_daily <- get_timeseries_tsid("78382042", 
                                         from = "2017-04-01", 
                                         to = "2017-04-30")
head(overpelt_rh_daily)
#>             Timestamp Value Quality Code
#> 1 2017-04-01 23:00:00 80.19          130
#> 2 2017-04-02 23:00:00 89.58          130
#> 3 2017-04-03 23:00:00 79.56          130
#> 4 2017-04-04 23:00:00 84.13          130
#> 5 2017-04-05 23:00:00 84.19          130
#> 6 2017-04-06 23:00:00 82.71          130

ggplot(overpelt_rh_daily, aes(x = Timestamp, y = Value)) + 
    geom_line() + 
    xlab("") + ylab(" RH (%)") + 
    scale_x_datetime(date_labels = "%b-%d\n%Y", date_breaks = "5 days")

Unfortunately, not all variables are documented, for which the check for the appropriate variable is not (yet) fully supported by the package.

More detailed tutorials are available in the package vignettes!

Note on restrictions of the downloads

The amount of data downloaded from waterinfo.be is limited via a credit system. You do not need to get a token right away to download data. For limited and irregular downloads, a token will not be required.

When you require more extended data requests, please request a download token from the waterinfo.be site administrators via the e-mail address [email protected] with a statement of which data and how frequently you would like to download data. You will then receive a client-credit code that can be used to obtain a token that is valid for 24 hours, after which the token can be refreshed with the same client-credit code.

Get token with client-credit code: (limited client-credit code for testing purposes)

client <- paste0("MzJkY2VlY2UtODI2Yy00Yjk4LTljMmQtYjE2OTc4ZjBjYTZhOjRhZGE4",
                 "NzFhLTk1MjgtNGI0ZC1iZmQ1LWI1NzBjZThmNGQyZA==")
my_token <- get_token(client = client)
print(my_token)
#> Token:
#> eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiJiNTFlM2U5MC1iY2FjLTQ4Y2ItYmNmOS0wM2NmMDIwODA5MDIiLCJpYXQiOjE2NTcyMTI0NTAsImlzcyI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MC9LaVdlYlBvcnRhbC9hdXRoIiwiYXVkIjoiMzJkY2VlY2UtODI2Yy00Yjk4LTljMmQtYjE2OTc4ZjBjYTZhIiwiZXhwIjoxNjU3Mjk4ODUwfQ.PQv_g2cfrOCdfcpmINZncTAbZgwchzxXwmSjlJU3WIk
#> 
#> Attributes:
#>  url: http://download.waterinfo.be/kiwis-auth/token
#>  type: Bearer
#>  expires: 2022-07-08 18:47:30 CEST

Receive information on the validity of the token:

is.expired(my_token)
#> [1] FALSE

Check when the token expires:

expires.in(my_token)
#> Time difference of 23.99999 hours

Use token when retrieving data:

get_stations(variable_name = "verdamping_monteith", token = my_token)
#>      ts_id station_latitude station_longitude station_id station_no
#> 1 94310042         51.02263          2.970584      12206   ME01_003
#> 2 94516042         50.73795          5.141976      12211   ME09_012
#> 3 94530042         51.16224          4.845708      12212   ME10_011
#> 4 94544042         51.20300          5.439589      12213   ME11_002
#> 5 94474042         51.24379          4.266912      12208   ME04_001
#> 6 94502042         50.88663          4.094898      12210   ME07_006
#> 7 94460042         51.27226          3.728299      12207   ME03_017
#> 8 94488042         50.86149          3.411318      12209   ME05_019
#>              station_name stationparameter_name parametertype_name
#> 1               Zarren_ME                   pET                PET
#> 2 Niel-bij-St.-Truiden_ME                   pET                PET
#> 3            Herentals_ME                   pET                PET
#> 4             Overpelt_ME                   pET                PET
#> 5              Melsele_ME                   pET                PET
#> 6           Liedekerke_ME                   pET                PET
#> 7            Boekhoute_ME                   pET                PET
#> 8              Waregem_ME                   pET                PET
#>   ts_unitsymbol dataprovider
#> 1            mm          VMM
#> 2            mm          VMM
#> 3            mm          VMM
#> 4            mm          VMM
#> 5            mm          VMM
#> 6            mm          VMM
#> 7            mm          VMM
#> 8            mm          VMM

Other clients

Besides this wateRinfo R client to gather data from waterinfo.be, there is also a Python client available. The pywaterinfo package contains similar functionalities.

The Flanders Hydraulics Research center also distributes clients for R, Python and Matlab upon request to download the data they share on waterinfo.be. For more information, contact them directly via [email protected].

Acknowledgements

This package is just a small wrapper around waterinfo.be to facilitate researchers and other stakeholders in downloading the data from waterinfo.be. The availability of this data is made possible by de Vlaamse Milieumaatschappij, Waterbouwkundig Laboratorium, Maritieme Dienstverlening & Kust, Waterwegen en Zeekanaal NV en De Scheepvaart NV.

Meta

• We welcome contributions including bug reports.
• License: MIT
• Get citation information for wateRinfo in R doing citation("wateRinfo").
• Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

waterinfo's Issues

HIC/WL changed the quality code

When downloading new data from waterinfo.be with the origin from HIC, the disclaimer provides new validation codes:

HIC Quality Code Interpretation
6-8 External validated data (6=good, 7=estimated, 8=suspect)
10-19 Good measurements
20-29 Good calculations
30-39 Estimate measurements
40-49 Estimate calculations
60-69 Suspect measurements
70-79 Suspect calculations

110-179 Unchecked

221-223 Unknown (import)
255 of -1 Missing

Need to have a check and do an update of the documentation.

Suggestion: Add INBO as copyrightholder

E.g. by adding person("Research Institute for Nature and Forest", role = c("cph", "fnd"), email = "[email protected]") to the Authors@R field in the DESCRIPTION. See https://github.com/inbo/inbotheme/blob/master/DESCRIPTION

new installation instructions

Currently states Stijnvanhoey/wateRinfo. It should be INBO/

Figure in README not included in website

The figure at https://github.com/inbo/wateRinfo/blob/master/figures/README-plot_pressure-1.png which is referenced in the README is not rendered on the website:

https://inbo.github.io/wateRinfo/

That is probably because figures is not included in the site_build process. I'm not sure what the convention is, but it might be better to move figures to man/figures, as is done here for example: https://github.com/r-lib/pkgdown/tree/master/man/figures

Contributor code of conduct

Nice package! A mini remark: the link to your "contributor code of conduct" broke in the readme

Provide info about the tabel output format

cfr. https://github.com/ropensci/tidyhydat/blob/master/R/hy_stations.R#L30

Setup Travis for pkgdown build

@stijnvanhoey I have setup Travis to build the pkgdown website on successful package check for a number of repositories now, see e.g.

• https://inbo.github.io/watina/
• https://inbo.github.io/inborutils/
• https://trias-project.github.io/trias/

Do you foresee mayor problems in setting this up for wateRinfo (like credentials for accessing the API)?

API access to predicted tidal water levels.

Another useful -related- feature for fieldwork would be to have access to the short-term tidal water level prediction (https://www.waterinfo.be/default.aspx?path=NL/Thema/Getij_KT), e.g. to know when a field site at a certain elevation is accessible (no longer inundated).

The prediction graphs on the website are sometimes hard to read accurately. However it seems that these data are not available for download from the waterinfo website.

Maybe we can ask if they can make these data available?
(Question from @LennertSchepers )

Permission to use and change the CONTRIBUTING.md file

Hi,

I have attended a talk from ROpenSci on how to make an R package easier for others to contribute.

In the webpage, wateRinfo was used as guide.

Could I ask for permission if I can apply , use and make changes to the CONTRIBUTING.md of this package for my own use ? If so may I ask how to I give attribute/credit to this ?

Kindly advise.

Meaning of Quality Code

Quality Code is an output of get_timeseries_tsid() but there seems to be no function to get the meaning of these codes.
When downloading data straight from the portal (www.waterinfo.be) the meaning of the codes is added in the zip-file download:

VMM Quality Code Interpretation

#10/110 - Excellent
#30/100/130 - Good
#50/150 - Moderate
#70/170 - Poor
#80/180 - Estimated
#90/190 - Suspect
#220 - Default
#-1 - Missing

##HIC Quality Code Interpretation
##High-resolution timeseries
#40 - Good
#80 - Estimated
#120 - Suspect
#200 - Unchecked
#60 - Complete
#160 - Incomplete
#-1 - Missing

##Aggregated timeseries
#40 - Good
#100 - Estimated
#120 - Suspect
#200 - Unchecked
#-1 - Missing

The quality label seems to be dubious by the way: check this series of air pressure data with several measurements < 900 hpa: far below lowest air pressure limits but rated with quality code 130 - good

waregem_pressure <- get_timeseries_tsid("78056042", from = "2017-01-01", to = "2019-06-12") waregem_pressure %>% filter(Value< 900)

Custom_attributes in query deprecated

According to API documentation, custom_attributes is deprecated and ca_sta should be used instead (relevant for get_stations).

error while installing

Following error message occurred when installing the wateRinfo package:

Installing 1 package: openssl
Installing package into ‘C:/Users/meiredk/Documents/R/win-library/3.4’
(as ‘lib’ is unspecified)
trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.4/openssl_1.0.zip'
Content type 'application/zip' length 3572638 bytes (3.4 MB)
downloaded 3.4 MB

package ‘openssl’ successfully unpacked and MD5 sums checked
Warning: cannot remove prior installation of package ‘openssl’

The downloaded binary packages are in
	C:\Users\meiredk\AppData\Local\Temp\RtmpOOOOhf\downloaded_packages
Installing 1 package: rlang
Installing package into ‘C:/Users/meiredk/Documents/R/win-library/3.4’
(as ‘lib’ is unspecified)
trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.4/rlang_0.2.0.zip'
Content type 'application/zip' length 818670 bytes (799 KB)
downloaded 799 KB

package ‘rlang’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
	C:\Users\meiredk\AppData\Local\Temp\RtmpOOOOhf\downloaded_packages
"C:/PROGRA~1/R/R-34~1.3/bin/x64/R" --no-site-file --no-environ --no-save --no-restore  \
  --quiet CMD INSTALL  \
  "C:/Users/meiredk/AppData/Local/Temp/RtmpOOOOhf/devtools35c85d432df/inbo-wateRinfo-02fd611"  \
  --library="C:/Users/meiredk/Documents/R/win-library/3.4" --install-tests 

ERROR: dependency 'openssl' is not available for package 'wateRinfo'
* removing 'C:/Users/meiredk/Documents/R/win-library/3.4/wateRinfo'
In R CMD INSTALL
Installation failed: Command failed (1)

after a manual installation of openssl:

install.packages('openssl')

the installation of the package was successfull.

session info:

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=Dutch_Belgium.1252  LC_CTYPE=Dutch_Belgium.1252   
[3] LC_MONETARY=Dutch_Belgium.1252 LC_NUMERIC=C                  
[5] LC_TIME=Dutch_Belgium.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base 

datasource error

Trying to run code for creating chart for logo:

library(tidyverse)
library(wateRinfo)

tide <- get_variables("zes21a-1066") %>%
  filter(grepl(".*10min.base", ts_name)) %>%
  pull(ts_id) %>%
  get_timeseries_tsid(,
                      from = "2017-11-21 08:00:00",
                      to = "2017-11-23 23:00:00",
                      datasource = 4)

I get:

Error: Waterinfo API request failed [500]
Waterinfo error:
                InvalidParameterValue
Waterinfo return message: Datasource parameter not found in config.

I'm using wateRinfo 0.2.1.

This error is probably related to #38 and #39, breaking old code. There's probably not much that can be done to protect users from this, but I think the error message could be clearer: should I pass a new value to datasource or drop the parameter all together or something else?

Fetch error when knitting README.Rmd

Knitting README.Rmd to README.md, I get:

pandoc: Could not fetch https://ropensci.org/public_images/ropensci_footer.png
TlsExceptionHostPort (HandshakeFailed (Error_Misc "user error (unexpected type received. expecting handshake and got: Alert [(AlertLevel_Fatal,HandshakeFailure)])")) "ropensci.org" 443
Error: pandoc document conversion failed with error 67
Execution halted

Order and titles of articles

We currently have two titles per article:

1. their article title (defined in the vignette):
2. their introduction title on the hidden articles page (defined in _pkgdown.yml)

And their order is also different in the navigation and articles page:

file	nav order	article title	articles page order	introduction title
define_date_periods.Rmd	1	Definition of the date period to download	3	How to define the period?
download_timeseries_batch.Rmd	2	Downloading time series in batch	2	Downloading the data of multiple stations/variables
download_timeseries.Rmd	3	Download time series from waterinfo.be	1	Tutorial on the main downloading functionalities

I don't find that very intuitive. Here is what I suggest:

download_timeseries.Rmd

This is clearly the introduction article, going deeper than the example in the README. I would make this one the get started article, cf. bioRad (can be done by naming it wateRinfo.Rmd), with the title Introduction to wateRinfo or Introduction to downloading data from waterinfo.be

define_date_periods.Rmd

This vignette is referenced twice in the above one (although without a link) and it's an important one, so I would list it first in articles. I suggest to use the title Define the date period to download

download_timeseries_batch.Rmd

This vignette can be second. I would keep the title Download time series from multiple stations/variables

(I'm use active tense titles Define, Download, as those are also used for the subtitles in the vignettes)

Revise/Create examples or export functions

These examples are currently lacking or exposing a too extended output

top-level docs

The package should contain top-level documentation for ?foobar, (or ?foobar-package if there is a naming conflict). Optionally, you can use both ?foobar and ?foobar-package for the package level manual file, using @Aliases roxygen tag.

improve testing with off/online support

see https://github.com/nealrichardson/httptest

Date parse error when on devtools::check

When doing a devtools::check(), I get:

* checking examples ... ERROR
Running examples in ‘wateRinfo-Ex.R’ failed
The error most likely occurred in:

> base::assign(".ptime", proc.time(), pos = "CheckExEnv")
> ### Name: get_timeseries_tsid
> ### Title: Download timeseries data from waterinfo.be
> ### Aliases: get_timeseries_tsid
> 
> ### ** Examples
> 
> get_timeseries_tsid("35055042", from = "2017-01-01", to = "2017-01-02")
Error in check_date_format(from) : 
  The date string can not be properly parsed in any of the
             following formats: ymd_hms, ymd, ym, y)
Calls: get_timeseries_tsid -> parse_period -> check_date_format
Execution halted

Line causing the error:

https://github.com/inbo/wateRinfo/blob/199e426b18b4957584cd54d935607b7d1dff25f2/R/get_timeseries.R#L35

Function called:

https://github.com/inbo/wateRinfo/blob/a31a5c231f7ee59056ee3183103553cb5ee83142/R/parse_period.R#L49-L60

I have tried to look into this, but I am stuck. 😄

suggestion: apply INBO styling

See https://inbo.github.io/INBOtheme/. I'm happy to send a PR.

wateRinfo to CRAN

Need to check which steps are still required to do so, but a CRAN submission would simplify the installation by users.

wateRinfo API request failed

When I tried to download tidal data from multiple stations in the Scheldt Estuary through the following code, I got an error. Note that this occurs in the tidal-eel example from the R-blog.

tidal_data <-
  tidal_zone_ts_ids %>%
  group_by(ts_id) %>%
  
  # Download tidal data for each ts_id (time series id)
  do(get_timeseries_tsid(
      .$ts_id,
      from = min(eel$date), # Start of eel tracking data
      to = max(eel$date),   # End of eel tracking data
      datasource = 4
  ))

The error:

Error: Waterinfo API request failed [500]
Waterinfo error:
                DatasourceError
Waterinfo return message: Could not create WDP connection in given time frame, returning lock.

More informative error message when credit system limits are exceeded

I have a small suggestion that might improve the experience of new users: Currently the error message when credit system limits are exceeded is not particularly helpful. When I encountered it first, I had the impression something was going wrong with the xml parsing of the returned data, not really with the credit limits. In the meantime I do have access to a token, but it would be very helpful if these errors could be caught, and a message could be provided pointing the user to the token and credit system information in the help files or website.

Below a reprex, of which I think it represents this exceedence of credit system limits:

library(wateRinfo)
lapply(
  get_stations("water_level", "day")$ts_id,
  get_timeseries_tsid,
  from = "1990-01-01",
  to = "2020-03-01"
)
#> No encoding supplied: defaulting to UTF-8.
#> Error: lexical error: invalid char in json text.
#>                                        <?xml version="1.0" ?><Exceptio
#>                      (right here) ------^

^{Created on 2020-03-13 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.6.3 (2020-02-29)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  Dutch_Belgium.1252          
#>  ctype    Dutch_Belgium.1252          
#>  tz       Europe/Paris                
#>  date     2020-03-13                  
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version    date       lib source                             
#>  askpass       1.1        2019-01-13 [1] CRAN (R 3.6.1)                     
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 3.6.1)                     
#>  backports     1.1.5      2019-10-02 [1] CRAN (R 3.6.1)                     
#>  callr         3.4.2      2020-02-12 [1] CRAN (R 3.6.2)                     
#>  cli           2.0.2      2020-02-28 [1] CRAN (R 3.6.3)                     
#>  crayon        1.3.4      2017-09-16 [1] CRAN (R 3.6.1)                     
#>  curl          4.3        2019-12-02 [1] CRAN (R 3.6.1)                     
#>  desc          1.2.0      2018-05-01 [1] CRAN (R 3.6.1)                     
#>  devtools      2.2.1      2019-09-24 [1] CRAN (R 3.6.1)                     
#>  digest        0.6.25     2020-02-23 [1] CRAN (R 3.6.3)                     
#>  dplyr         0.8.5      2020-03-07 [1] CRAN (R 3.6.3)                     
#>  ellipsis      0.3.0      2019-09-20 [1] CRAN (R 3.6.1)                     
#>  evaluate      0.14       2019-05-28 [1] CRAN (R 3.6.1)                     
#>  fansi         0.4.1      2020-01-08 [1] CRAN (R 3.6.2)                     
#>  fs            1.3.1      2019-05-06 [1] CRAN (R 3.6.1)                     
#>  glue          1.3.2      2020-03-12 [1] CRAN (R 3.6.3)                     
#>  highr         0.8        2019-03-20 [1] CRAN (R 3.6.1)                     
#>  htmltools     0.4.0      2019-10-04 [1] CRAN (R 3.6.1)                     
#>  httr          1.4.1      2019-08-05 [1] CRAN (R 3.6.1)                     
#>  jsonlite      1.6.1      2020-02-02 [1] CRAN (R 3.6.2)                     
#>  knitr         1.28       2020-02-06 [1] CRAN (R 3.6.2)                     
#>  lubridate     1.7.4      2018-04-11 [1] CRAN (R 3.6.1)                     
#>  magrittr      1.5        2014-11-22 [1] CRAN (R 3.6.1)                     
#>  memoise       1.1.0      2017-04-21 [1] CRAN (R 3.6.1)                     
#>  openssl       1.4.1      2019-07-18 [1] CRAN (R 3.6.1)                     
#>  pillar        1.4.3      2019-12-20 [1] CRAN (R 3.6.2)                     
#>  pkgbuild      1.0.6      2019-10-09 [1] CRAN (R 3.6.1)                     
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 3.6.1)                     
#>  pkgload       1.0.2      2018-10-29 [1] CRAN (R 3.6.1)                     
#>  prettyunits   1.1.1      2020-01-24 [1] CRAN (R 3.6.2)                     
#>  processx      3.4.2      2020-02-09 [1] CRAN (R 3.6.2)                     
#>  ps            1.3.2      2020-02-13 [1] CRAN (R 3.6.3)                     
#>  purrr         0.3.3      2019-10-18 [1] CRAN (R 3.6.1)                     
#>  R6            2.4.1      2019-11-12 [1] CRAN (R 3.6.1)                     
#>  Rcpp          1.0.3      2019-11-08 [1] CRAN (R 3.6.1)                     
#>  remotes       2.1.0      2019-06-24 [1] CRAN (R 3.6.1)                     
#>  rlang         0.4.5      2020-03-01 [1] CRAN (R 3.6.3)                     
#>  rmarkdown     2.1        2020-01-20 [1] CRAN (R 3.6.2)                     
#>  rprojroot     1.3-2      2018-01-03 [1] CRAN (R 3.6.1)                     
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.6.1)                     
#>  stringi       1.4.6      2020-02-17 [1] CRAN (R 3.6.2)                     
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 3.6.1)                     
#>  testthat      2.3.2      2020-03-02 [1] CRAN (R 3.6.3)                     
#>  tibble        2.1.3      2019-06-06 [1] CRAN (R 3.6.1)                     
#>  tidyselect    1.0.0      2020-01-27 [1] CRAN (R 3.6.2)                     
#>  usethis       1.5.1      2019-07-04 [1] CRAN (R 3.6.2)                     
#>  vctrs         0.2.4      2020-03-10 [1] CRAN (R 3.6.3)                     
#>  wateRinfo   * 0.3.0.9065 2020-03-13 [1] Github (ropensci/wateRinfo@c03e4be)
#>  withr         2.1.2      2018-03-15 [1] CRAN (R 3.6.1)                     
#>  xfun          0.12       2020-01-13 [1] CRAN (R 3.6.2)                     
#>  yaml          2.2.1      2020-02-01 [1] CRAN (R 3.6.2)                     
#> 
#> [1] C:/Users/brogiers/Documents/R/win-library/3.6
#> [2] C:/Program Files/R/R-3.6.3/library

use a NEWS or NEWS.md file in the root of your package

and provide the developers with sufficient information on release tags, http://r-pkgs.had.co.nz/description.html#version

Use Markdown for function documentation

I didn't know this, but you can use Markdown in addition to Roxygen syntax for function document: http://klutometis.github.io/roxygen/articles/markdown.html, by adding to DESCRIPTION:

Roxygen: list(markdown = TRUE)

It basically gets rid of all the something{text} syntax, making it easier to write and read.

Create hex logo for wateRinfo

@stijnvanhoey, fun challenge for you. 😄

Can be included cf. https://github.com/CornellLabofOrnithology/auk/blob/master/README.Rmd

Collection of HEX logos: http://hexb.in/

https://docs.ropensci.org/wateRinfo

The website referenced at the top of the repo is now https://docs.ropensci.org/wateRinfo. Has this recently changed from https://ropensci.github.io/wateRinfo?

Which one is hosted from where? I don't think it makes sense to maintain (the CI for) both.

Extend package to request precipitation as raster map waterinfo.be

According to more recent documentation , precipitation radar images can be requested as well using the waterinfo.be API using a number of new calls:

• GetRasterTimeseriesValue (analogue to existing getTimeseriesvalues API call), returning HDF5 Opera files and metadata using the html format

e.g. https://hydro.vmm.be/grid/kiwis/KiWIS?datasource=10&service=kisters&type=queryServices&request=getrastertimeseriesvalues&ts_path=COMP_VMM/Vlaanderen_VMM/N/5m.Cmd.Raster.O.PAC_1h_1km_cappi_adj&period=PT1H&from=2018-10-09T15:00:00.000+01:00&format=hdf5

• GetRasterFile to get a single geotiff or AsciiGrid file
• GetRasterToPointValues request a timeseries for a single point of the raster image using raster_x and raster_y as query parameters

Note that the related reliability can also be requested using the existing GetTimeseriesValues request. Datasource for these raster datasets is 10.

Credit limit: object 'error_message' not found

We are now experiencing some issues with the newly introduced credit limit error message. Without token, the get_stations() call below runs fine. With, it fails. I unfortunately cannot provide a full reprex without exposing my client id ...

my_token <- wateRinfo::get_token("***")
wateRinfo::get_stations("discharge", token = my_token) 
#> Error in grep(pattern = "Credit limit exceeded", error_message): object 'error_message' not found
wateRinfo::is.expired(my_token)
#> [1] FALSE
wateRinfo::expires.in(my_token)
#> Time difference of 23.9999 hours

^{Created on 2020-06-02 by the reprex package (v0.3.0)}

support for token-users

VMM service (temporary) not operational

If anyone happens to have issues on requesting waterinfo data from VMM, with the following error message:

 Error: Waterinfo API request failed [500]
Waterinfo error:
                DatasourceError
Waterinfo return message: Error logging into WDPConnector with given username and password or data was empty. 

The base URL we use to request data - http://download.waterinfo.be/tsmdownload - seems to be out of operation. We requested the VMM on an update or alternative URL to use.

add #' @noRd to internal functions.

cfr recommendations

MOW-HIC

The wateRinfo package is very usefull and transparant in downloading environmental data from the waterinfo.be database. However, it would be an added value if the data from MOW-HIC would be downloadable with the package as well. Now we can only get a part (i.e. the VMM part) of the available data via wateRinfo.

Unit tests with wrong `ts_id`failing

Tests are failing (probably) due to changes in the API responses received from waterinfo.be. I can reproduce them locally:

── 1. Failure: datasource not included in the API call (@test-waterinfo_call.R#35)  ──────────────────────────────────────────────────────────────
`call_waterinfo(query)` threw an error with unexpected message.
Expected match: "API did not return json - The server encountered an .*"
Actual message: "Waterinfo API request failed [500]\nWaterinfo error:\n                InvalidParameterValue\nWaterinfo return message: ts_id must be a number."

── 2. Failure: tomcat error message content (@test-waterinfo_call.R#51)  ─────────────────────────────────────────────────────────────────────────
content(res, "text", encoding = "UTF-8") not identical to `tomcat_html_response`.
1/1 mismatches
x[1]: "{\"type\": \"error\", \"code\": \"InvalidParameterValue\", \"message\": \"ts_id must be a number.\"}"
y[1]: "<!doctype html><html lang=\"en\"><head><title>HTTP Status 500 – Internal Server Error</title><style type=\"text/css\">H1 {font-family:Tahom
y[1]: a,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-colo
y[1]: r:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:T
y[1]: ahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P 
y[1]: {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height:...

I assume the response on wrong ts_id as an input of the API is now checked differently. Need to be fixed by adjusting the tests...

Setup GitHub Actions workflows

Suggested in #86 (comment), PR: #88

• Add R CMD Check
• Add pkgdown
• Remove Travis settings files
• Remove wateRinfo from Travis
• Remove Appveyor setting files
• Remove wateRinfo from Appveyor: https://ci.appveyor.com/project/stijnvanhoey/waterinfo-agry3/

Improved handling `return_fields`

Currently, the return-fields for each of the calls are hard-coded in the package

in get_variables:

    return_fields <- c("station_name", "station_no", "ts_id",
                       "ts_name", "parametertype_name",
                       "stationparameter_name")

in get_timeseries_tsid:

    # define the date fields we require
    return_fields <- c("Timestamp", "Value", "Quality Code")

and in get_stations:

    custom_attributes <- c("dataprovider")
    return_fields <- c("custom_attributes", "station_id", "station_no",
                       "station_name", "stationparameter_name",
                       "ts_unitsymbol", "parametertype_name", "ts_id")

together with the defaults station_latitude and station_longitude.

The decision to keep these has been a (deliberated) decision during the implementation based on the available information.

Still, more flexibility for the user (while keeping proper defaults) could improve the workflow. As an example, the API supports the md_returnfields: station_cartnorthing and station_carteasting as input as well, returning the coordinates in Lambert 72 instead of the default WGS84. Altough conversion can be done in R as well (see for example this function), the argument usage would simplify the workflow.

However, I do not have information or access to the possible arguments (either custom_attributes or md_returnfields) and the waterinfo documentation does not provide an overview as well. @WillemMaetens, is there such an overview available somewhere?

Vignette failure

Seen via https://github.com/r-universe/ropensci/runs/4511974492?check_suite_focus=true

--- re-building ‘wateRinfo.Rmd’ using rmarkdown
   Quitting from lines 126-128 (wateRinfo.Rmd) 
   Error: processing vignette 'wateRinfo.Rmd' failed with diagnostics:
   No station with the name K06_OM225 could be identified by waterinfo API.
   --- failed re-building ‘wateRinfo.Rmd’
   

appveyor and coveralls integration not working

I have troubles with the coveralls and appveyor integrations:

• the coveralls integration is not working, as the link is not known. The old link is still working, but not receiving any updates.
• I have a failing build in appveyor, but no access rights to see the error logs.

@sckott or @karthik Can you provide some help on this?

Tidal data from MOW-HIC stations?

On the waterinfo.be website, also the data from the navigable waterways is available, for example the tidal data of the Scheldt river (data from MOW-HIC). These stations are not accessible with the wateRinfo package. Is this a matter of adding some more functions, or are these data not (yet?) available through the API?

Handle ropensci package tips

https://github.com/ropensci/onboarding/blob/master/packaging_guide.md#-miscellaneous

hardcoded datasource in get_stations call

https://github.com/ropensci/wateRinfo/blob/master/R/get_stations.R#L70
Should be able to include the datasource, either 1 or 4

New website: effect on package?

I just read that https://waterinfo.be got upgraded (December 2020):

@stijnvanhoey Should we run the tests to see if all functions in the package still operate as intended?

Binaries via inbo drat ter beschikking stellen

binaries kan je ter beschikking stellen via https://inbo.github.io/drat/

de gebruikers hebben hiervoor een aangepaste Rprofile.site nodig.
zie https://inbo-tutorials.netlify.com/installation/admin/r/

Add package to R Hydrology task view

As suggested by @boshek, the package can be enlisted in the R Hydrology task view after CRAN acceptance.

CONTRIBUTING now part of side menu on pkgdown website

The new website uses pkgdown 1.3.0.9100, which automatically includes CONTRIBUTING and CODE OF CONDUCT in the side menu:

That means we don't have to explicitly link to it CONTRIBUTING in the navbar:

wateRinfo/_pkgdown.yml

Lines 33 to 34 in 3918e6b

	- text: Contributing
	href: CONTRIBUTING.html

We don't have to define a navbar at all (but use the default) if we are fine with:

• Not having contributing in the navbar
• Not having the twitter icon in the navbar (fine with me, especially since website is now included)
• Using "articles" rather than "tutorials" (see #53 (comment))
• Having the home icon back

See https://inbo.github.io/inborutils/ for an example

@stijnvanhoey, your call. 😄

adjust unit testing

We could have a check on how we could improve the unit tests with https://github.com/ropensci/webmockr