emodnet/emodnetwfs

Access EMODnet Web Feature Service data through R

EMODnetWFS: Access EMODnet Web Feature Service data through R

The goal of EMODnetWFS is to allow interrogation of and access to EMODnet geographic vector data in R though the EMODnet Web Feature Services. Web Feature services (WFS) represent a change in the way geographic information is created, modified and exchanged on the Internet and offer direct fine-grained access to geographic information at the feature and feature property level. EMODnetWFS aims at offering an user-friendly interface to this rich data.

Installation and setup

You can install the development version of EMODnetWFS from GitHub with:

# install.packages("pak")
pak::pak("EMODnet/EMODnetWFS")

If you want to avoid reading messages from EMODnetWFS such as “WFS client created successfully”, set the "EMODnetWFS.quiet" option to TRUE.

options("EMODnetWFS.quiet" = TRUE)

Available services

All available services are contained in the tibble returned by emodnet_wfs().

service_name	service_url
bathymetry	https://ows.emodnet-bathymetry.eu/wfs
biology	https://geo.vliz.be/geoserver/Emodnetbio/wfs
biology_occurrence_data	https://geo.vliz.be/geoserver/Dataportal/wfs
chemistry_cdi_data_discovery_and_access_service	https://geo-service.maris.nl/emodnet_chemistry/wfs
chemistry_cdi_distribution_observations_per_category_and_region	https://geo-service.maris.nl/emodnet_chemistry_p36/wfs
chemistry_marine_litter	https://www.ifremer.fr/services/wfs/emodnet_chemistry2
geology_coastal_behavior	https://drive.emodnet-geology.eu/geoserver/tno/wfs
geology_events_and_probabilities	https://drive.emodnet-geology.eu/geoserver/ispra/wfs
geology_marine_minerals	https://drive.emodnet-geology.eu/geoserver/gsi/wfs
geology_sea_floor_bedrock	https://drive.emodnet-geology.eu/geoserver/bgr/wfs
geology_seabed_substrate_maps	https://drive.emodnet-geology.eu/geoserver/gtk/wfs
geology_submerged_landscapes	https://drive.emodnet-geology.eu/geoserver/bgs/wfs
human_activities	https://ows.emodnet-humanactivities.eu/wfs
physics	https://prod-geoserver.emodnet-physics.eu/geoserver/ows
seabed_habitats_general_datasets_and_products	https://ows.emodnet-seabedhabitats.eu/geoserver/emodnet_open/wfs
seabed_habitats_individual_habitat_map_and_model_datasets	https://ows.emodnet-seabedhabitats.eu/geoserver/emodnet_open_maplibrary/wfs

To explore available services you can use:

View(emodnet_wfs())

Create Service Client

Specify the service using the service argument.

wfs_bio <- emodnet_init_wfs_client(service = "biology")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Emodnetbio/wfs"
#> ℹ Version: "2.0.0"

wfs_bio
#> <WFSClient>
#> ....|-- url: https://geo.vliz.be/geoserver/Emodnetbio/wfs
#> ....|-- version: 2.0.0
#> ....|-- capabilities <WFSCapabilities>

Get WFS Layer info

You can get metadata about the layers available from a service.

emodnet_get_wfs_info(service = "biology")
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Emodnetbio/wfs"
#> ℹ Version: "2.0.0"
#> # A tibble: 35 × 9
#> # Rowwise: 
#>    data_source service_name service_url   layer_name title abstract class format
#>    <chr>       <chr>        <chr>         <chr>      <chr> <chr>    <chr> <chr> 
#>  1 emodnet_wfs biology      https://geo.… mediseh_c… EMOD… "Coral … WFSF… sf    
#>  2 emodnet_wfs biology      https://geo.… mediseh_c… EMOD… "Coral … WFSF… sf    
#>  3 emodnet_wfs biology      https://geo.… mediseh_c… EMOD… "Cymodo… WFSF… sf    
#>  4 emodnet_wfs biology      https://geo.… Species_g… EMOD… "This d… WFSF… sf    
#>  5 emodnet_wfs biology      https://geo.… Species_g… EMOD… "This d… WFSF… sf    
#>  6 emodnet_wfs biology      https://geo.… Species_g… EMOD… "This d… WFSF… sf    
#>  7 emodnet_wfs biology      https://geo.… mediseh_h… EMOD… "Haloph… WFSF… sf    
#>  8 emodnet_wfs biology      https://geo.… mediseh_m… EMOD… "Maërl … WFSF… sf    
#>  9 emodnet_wfs biology      https://geo.… mediseh_m… EMOD… "Maërl … WFSF… sf    
#> 10 emodnet_wfs biology      https://geo.… mediseh_p… EMOD… "This d… WFSF… sf    
#> # ℹ 25 more rows
#> # ℹ 1 more variable: layer_namespace <chr>

or you can pass a wfs client object.

emodnet_get_wfs_info(wfs_bio)
#> # A tibble: 35 × 9
#> # Rowwise: 
#>    data_source service_name service_url   layer_name title abstract class format
#>    <chr>       <chr>        <chr>         <chr>      <chr> <chr>    <chr> <chr> 
#>  1 emodnet_wfs biology      https://geo.… mediseh_c… EMOD… "Coral … WFSF… sf    
#>  2 emodnet_wfs biology      https://geo.… mediseh_c… EMOD… "Coral … WFSF… sf    
#>  3 emodnet_wfs biology      https://geo.… mediseh_c… EMOD… "Cymodo… WFSF… sf    
#>  4 emodnet_wfs biology      https://geo.… Species_g… EMOD… "This d… WFSF… sf    
#>  5 emodnet_wfs biology      https://geo.… Species_g… EMOD… "This d… WFSF… sf    
#>  6 emodnet_wfs biology      https://geo.… Species_g… EMOD… "This d… WFSF… sf    
#>  7 emodnet_wfs biology      https://geo.… mediseh_h… EMOD… "Haloph… WFSF… sf    
#>  8 emodnet_wfs biology      https://geo.… mediseh_m… EMOD… "Maërl … WFSF… sf    
#>  9 emodnet_wfs biology      https://geo.… mediseh_m… EMOD… "Maërl … WFSF… sf    
#> 10 emodnet_wfs biology      https://geo.… mediseh_p… EMOD… "This d… WFSF… sf    
#> # ℹ 25 more rows
#> # ℹ 1 more variable: layer_namespace <chr>

You can also get info for specific layers from wfs object:

layers <- c("mediseh_zostera_m_pnt", "mediseh_posidonia_nodata")

emodnet_get_layer_info(wfs = wfs_bio, layers = layers)
#> # A tibble: 2 × 9
#> # Rowwise: 
#>   data_source service_name    service_url layer_name title abstract class format
#>   <chr>       <chr>           <chr>       <chr>      <chr> <chr>    <chr> <chr> 
#> 1 emodnet_wfs https://geo.vl… biology     mediseh_p… EMOD… "Coastl… WFSF… sf    
#> 2 emodnet_wfs https://geo.vl… biology     mediseh_z… EMOD… "Zoster… WFSF… sf    
#> # ℹ 1 more variable: layer_namespace <chr>

Finally, you can get details on all available services and layers from the server

emodnet_get_all_wfs_info()

Get WFS layers

You can extract layers directly from a wfs object using layer names. All layers are downloaded as sf objects and output as a list with a named element for each layer requested.

emodnet_get_layers(wfs = wfs_bio, layers = layers)
#> $mediseh_zostera_m_pnt
#> Simple feature collection with 54 features and 3 fields
#> Geometry type: POINT
#> Dimension:     XY
#> Bounding box:  xmin: -4.167154 ymin: 33.07783 xmax: 15.35766 ymax: 45.72451
#> Geodetic CRS:  WGS 84
#> First 10 features:
#>                      gml_id id country                   the_geom
#> 1   mediseh_zostera_m_pnt.1  0  Spagna  POINT (-2.61314 36.71681)
#> 2   mediseh_zostera_m_pnt.2  0  Spagna POINT (-3.846598 36.75127)
#> 3   mediseh_zostera_m_pnt.3  0  Spagna POINT (-3.957785 36.72266)
#> 4   mediseh_zostera_m_pnt.4  0  Spagna POINT (-4.039712 36.74217)
#> 5   mediseh_zostera_m_pnt.5  0  Spagna POINT (-4.100182 36.72331)
#> 6   mediseh_zostera_m_pnt.6  0  Spagna POINT (-4.167154 36.71226)
#> 7   mediseh_zostera_m_pnt.7  0  Spagna POINT (-1.268366 37.55796)
#> 8   mediseh_zostera_m_pnt.8  0 Francia   POINT (4.84864 43.37637)
#> 9   mediseh_zostera_m_pnt.9  0  Italia  POINT (13.71831 45.70017)
#> 10 mediseh_zostera_m_pnt.10  0  Italia  POINT (13.16378 45.72451)
#> 
#> $mediseh_posidonia_nodata
#> Simple feature collection with 465 features and 3 fields
#> Geometry type: MULTICURVE
#> Dimension:     XY
#> Bounding box:  xmin: -2.1798 ymin: 30.26623 xmax: 34.60767 ymax: 45.47668
#> Geodetic CRS:  WGS 84
#> First 10 features:
#>                         gml_id id         km                       the_geom
#> 1   mediseh_posidonia_nodata.1  0 291.503233 MULTICURVE (LINESTRING (27....
#> 2   mediseh_posidonia_nodata.2  0  75.379502 MULTICURVE (LINESTRING (23....
#> 3   mediseh_posidonia_nodata.3  0  38.627764 MULTICURVE (LINESTRING (22....
#> 4   mediseh_posidonia_nodata.4  0 110.344802 MULTICURVE (LINESTRING (19....
#> 5  mediseh_posidonia_nodata.13  0  66.997461 MULTICURVE (LINESTRING (9.1...
#> 6  mediseh_posidonia_nodata.14  0  18.090640 MULTICURVE (LINESTRING (9.7...
#> 7  mediseh_posidonia_nodata.15  0  16.618978 MULTICURVE (LINESTRING (9.8...
#> 8  mediseh_posidonia_nodata.16  0   1.913773 MULTICURVE (LINESTRING (10....
#> 9  mediseh_posidonia_nodata.83  0   2.173447 MULTICURVE (LINESTRING (15....
#> 10 mediseh_posidonia_nodata.84  0   2.817453 MULTICURVE (LINESTRING (15....

You can change the output crs through the argument crs.

emodnet_get_layers(wfs = wfs_bio, layers = layers, crs = 3857)
#> ℹ crs transformed to 3857.
#> ℹ crs transformed to 3857.
#> $mediseh_zostera_m_pnt
#> Simple feature collection with 54 features and 3 fields
#> Geometry type: POINT
#> Dimension:     XY
#> Bounding box:  xmin: -463885.4 ymin: 3905639 xmax: 1709607 ymax: 5736311
#> Projected CRS: WGS 84 / Pseudo-Mercator
#> First 10 features:
#>                      gml_id id country                  the_geom
#> 1   mediseh_zostera_m_pnt.1  0  Spagna POINT (-290893.4 4399707)
#> 2   mediseh_zostera_m_pnt.2  0  Spagna POINT (-428201.3 4404494)
#> 3   mediseh_zostera_m_pnt.3  0  Spagna POINT (-440578.6 4400520)
#> 4   mediseh_zostera_m_pnt.4  0  Spagna POINT (-449698.6 4403229)
#> 5   mediseh_zostera_m_pnt.5  0  Spagna POINT (-456430.1 4400610)
#> 6   mediseh_zostera_m_pnt.6  0  Spagna POINT (-463885.4 4399075)
#> 7   mediseh_zostera_m_pnt.7  0  Spagna POINT (-141193.9 4517168)
#> 8   mediseh_zostera_m_pnt.8  0 Francia  POINT (539748.1 5369436)
#> 9   mediseh_zostera_m_pnt.9  0  Italia   POINT (1527115 5732431)
#> 10 mediseh_zostera_m_pnt.10  0  Italia   POINT (1465385 5736311)
#> 
#> $mediseh_posidonia_nodata
#> Simple feature collection with 465 features and 3 fields
#> Geometry type: MULTICURVE
#> Dimension:     XY
#> Bounding box:  xmin: -242654.3 ymin: 3537818 xmax: 3852508 ymax: 5696879
#> Projected CRS: WGS 84 / Pseudo-Mercator
#> First 10 features:
#>                         gml_id id         km                       the_geom
#> 1   mediseh_posidonia_nodata.1  0 291.503233 MULTICURVE (LINESTRING (302...
#> 2   mediseh_posidonia_nodata.2  0  75.379502 MULTICURVE (LINESTRING (257...
#> 3   mediseh_posidonia_nodata.3  0  38.627764 MULTICURVE (LINESTRING (246...
#> 4   mediseh_posidonia_nodata.4  0 110.344802 MULTICURVE (LINESTRING (221...
#> 5  mediseh_posidonia_nodata.13  0  66.997461 MULTICURVE (LINESTRING (101...
#> 6  mediseh_posidonia_nodata.14  0  18.090640 MULTICURVE (LINESTRING (108...
#> 7  mediseh_posidonia_nodata.15  0  16.618978 MULTICURVE (LINESTRING (110...
#> 8  mediseh_posidonia_nodata.16  0   1.913773 MULTICURVE (LINESTRING (121...
#> 9  mediseh_posidonia_nodata.83  0   2.173447 MULTICURVE (LINESTRING (169...
#> 10 mediseh_posidonia_nodata.84  0   2.817453 MULTICURVE (LINESTRING (169...

You can also extract layers using a WFS service name.

emodnet_get_layers(
  service = "biology",
  layers = c("mediseh_zostera_m_pnt", "mediseh_posidonia_nodata")
)
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Emodnetbio/wfs"
#> ℹ Version: "2.0.0"
#> $mediseh_zostera_m_pnt
#> Simple feature collection with 54 features and 3 fields
#> Geometry type: POINT
#> Dimension:     XY
#> Bounding box:  xmin: -4.167154 ymin: 33.07783 xmax: 15.35766 ymax: 45.72451
#> Geodetic CRS:  WGS 84
#> First 10 features:
#>                      gml_id id country                   the_geom
#> 1   mediseh_zostera_m_pnt.1  0  Spagna  POINT (-2.61314 36.71681)
#> 2   mediseh_zostera_m_pnt.2  0  Spagna POINT (-3.846598 36.75127)
#> 3   mediseh_zostera_m_pnt.3  0  Spagna POINT (-3.957785 36.72266)
#> 4   mediseh_zostera_m_pnt.4  0  Spagna POINT (-4.039712 36.74217)
#> 5   mediseh_zostera_m_pnt.5  0  Spagna POINT (-4.100182 36.72331)
#> 6   mediseh_zostera_m_pnt.6  0  Spagna POINT (-4.167154 36.71226)
#> 7   mediseh_zostera_m_pnt.7  0  Spagna POINT (-1.268366 37.55796)
#> 8   mediseh_zostera_m_pnt.8  0 Francia   POINT (4.84864 43.37637)
#> 9   mediseh_zostera_m_pnt.9  0  Italia  POINT (13.71831 45.70017)
#> 10 mediseh_zostera_m_pnt.10  0  Italia  POINT (13.16378 45.72451)
#> 
#> $mediseh_posidonia_nodata
#> Simple feature collection with 465 features and 3 fields
#> Geometry type: MULTICURVE
#> Dimension:     XY
#> Bounding box:  xmin: -2.1798 ymin: 30.26623 xmax: 34.60767 ymax: 45.47668
#> Geodetic CRS:  WGS 84
#> First 10 features:
#>                         gml_id id         km                       the_geom
#> 1   mediseh_posidonia_nodata.1  0 291.503233 MULTICURVE (LINESTRING (27....
#> 2   mediseh_posidonia_nodata.2  0  75.379502 MULTICURVE (LINESTRING (23....
#> 3   mediseh_posidonia_nodata.3  0  38.627764 MULTICURVE (LINESTRING (22....
#> 4   mediseh_posidonia_nodata.4  0 110.344802 MULTICURVE (LINESTRING (19....
#> 5  mediseh_posidonia_nodata.13  0  66.997461 MULTICURVE (LINESTRING (9.1...
#> 6  mediseh_posidonia_nodata.14  0  18.090640 MULTICURVE (LINESTRING (9.7...
#> 7  mediseh_posidonia_nodata.15  0  16.618978 MULTICURVE (LINESTRING (9.8...
#> 8  mediseh_posidonia_nodata.16  0   1.913773 MULTICURVE (LINESTRING (10....
#> 9  mediseh_posidonia_nodata.83  0   2.173447 MULTICURVE (LINESTRING (15....
#> 10 mediseh_posidonia_nodata.84  0   2.817453 MULTICURVE (LINESTRING (15....

Layers can also be returned to a single sf object through argument reduce_layers.
If TRUE the function will try to reduce all layers into a single sf.

If attempting to reduce fails, it will error:

emodnet_get_layers(
  wfs = wfs_bio,
  layers = layers,
  reduce_layers = TRUE
)
#> Error in `value[[3L]]()`:
#> ! Cannot reduce layers.
#> ℹ Try again with `reduce_layers = FALSE`

Using reduce_layers = TRUE is also useful for returning an sf object rather than a list in single layer request.

emodnet_get_layers(
  service = "biology",
  layers = c("mediseh_posidonia_nodata"),
  reduce_layers = TRUE
)
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Emodnetbio/wfs"
#> ℹ Version: "2.0.0"
#> Simple feature collection with 465 features and 3 fields
#> Geometry type: MULTICURVE
#> Dimension:     XY
#> Bounding box:  xmin: -2.1798 ymin: 30.26623 xmax: 34.60767 ymax: 45.47668
#> Geodetic CRS:  WGS 84
#> First 10 features:
#>                         gml_id id         km                       the_geom
#> 1   mediseh_posidonia_nodata.1  0 291.503233 MULTICURVE (LINESTRING (27....
#> 2   mediseh_posidonia_nodata.2  0  75.379502 MULTICURVE (LINESTRING (23....
#> 3   mediseh_posidonia_nodata.3  0  38.627764 MULTICURVE (LINESTRING (22....
#> 4   mediseh_posidonia_nodata.4  0 110.344802 MULTICURVE (LINESTRING (19....
#> 5  mediseh_posidonia_nodata.13  0  66.997461 MULTICURVE (LINESTRING (9.1...
#> 6  mediseh_posidonia_nodata.14  0  18.090640 MULTICURVE (LINESTRING (9.7...
#> 7  mediseh_posidonia_nodata.15  0  16.618978 MULTICURVE (LINESTRING (9.8...
#> 8  mediseh_posidonia_nodata.16  0   1.913773 MULTICURVE (LINESTRING (10....
#> 9  mediseh_posidonia_nodata.83  0   2.173447 MULTICURVE (LINESTRING (15....
#> 10 mediseh_posidonia_nodata.84  0   2.817453 MULTICURVE (LINESTRING (15....

Help needed?

If you get an unexpected error,

• Look up the EMODnet monitor;
• Open an issue in this repository.

Other web services

There are three ways to access EMODnet data at the moment, that complement each other.

EMODnet ERDDAP server

Some EMODnet data are also published in an ERDDAP server. You can access these data in R using the rerddap R package:

# install.packages("rerrdap")
library(rerddap)
#> Registered S3 method overwritten by 'hoardr':
#>   method           from
#>   print.cache_info httr

erddap_url <- "https://erddap.emodnet.eu/erddap/"

rerddap::ed_datasets(url = erddap_url)
#> # A tibble: 8 × 16
#>   griddap Subset tabledap Make.A.Graph wms   files Title Summary FGDC  ISO.19115
#>   <chr>   <chr>  <chr>    <chr>        <chr> <chr> <chr> <chr>   <chr> <chr>    
#> 1 ""      "http… https:/… https://erd… ""    ""    * Th… "This … ""    ""       
#> 2 ""      ""     https:/… https://erd… ""    "htt… EMOD… "The d… ""    ""       
#> 3 ""      ""     https:/… https://erd… ""    "htt… EMOD… "The d… ""    ""       
#> 4 ""      "http… https:/… https://erd… ""    "htt… EMOD… "The d… "htt… "https:/…
#> 5 ""      ""     https:/… https://erd… ""    "htt… Pres… "The p… "htt… "https:/…
#> 6 ""      ""     https:/… https://erd… ""    ""    PSMS… "Perma… ""    ""       
#> 7 ""      ""     https:/… https://erd… ""    "htt… PSMS… "Perma… ""    ""       
#> 8 ""      "http… https:/… https://erd… ""    "htt… TAO/… "This … "htt… "https:/…
#> # ℹ 6 more variables: Info <chr>, Background.Info <chr>, RSS <chr>,
#> #   Email <chr>, Institution <chr>, Dataset.ID <chr>

rerddap::ed_search(query = "vessel density", url = erddap_url)
#> # A tibble: 16 × 2
#>    title                                                     dataset_id         
#>    <chr>                                                     <chr>              
#>  1 Vessel Density                                            humanactivities_9f…
#>  2 Vessel Density                                            humanactivities_e9…
#>  3 Vessel traffic density, 2019, All                         EMODPACE_VD_All    
#>  4 Vessel traffic density, 2019, Cargo                       EMODPACE_VD_09_Car…
#>  5 Vessel traffic density, 2019, Dredging or underwater ops  EMODPACE_VD_03_Dre…
#>  6 Vessel traffic density, 2019, Fishing                     EMODPACE_VD_01_Fis…
#>  7 Vessel traffic density, 2019, High Speed craft            EMODPACE_VD_06_High
#>  8 Vessel traffic density, 2019, Miliary and law enforcement EMODPACE_VD_11_Mil…
#>  9 Vessel traffic density, 2019, Other                       EMODPACE_VD_00_Oth…
#> 10 Vessel traffic density, 2019, Passenger                   EMODPACE_VD_08_Pas…
#> 11 Vessel traffic density, 2019, Pleasure craft              EMODPACE_VD_05_Ple…
#> 12 Vessel traffic density, 2019, Sailing                     EMODPACE_VD_04_Sai…
#> 13 Vessel traffic density, 2019, Service                     EMODPACE_VD_02_Ser…
#> 14 Vessel traffic density, 2019, Tanker                      EMODPACE_VD_10_Tan…
#> 15 Vessel traffic density, 2019, Tug and Towing              EMODPACE_VD_07_Tug 
#> 16 Vessel traffic density, 2019, Unknown                     EMODPACE_VD_12_Unk…

human_activities_data_info <- rerddap::info(datasetid = "humanactivities_9f8a_3389_f08a", url = erddap_url)
human_activities_data_info
#> <ERDDAP info> humanactivities_9f8a_3389_f08a 
#>  Base URL: https://erddap.emodnet.eu/erddap 
#>  Dataset Type: griddap 
#>  Dimensions (range):  
#>      time: (2017-01-01T00:00:00Z, 2021-12-01T00:00:00Z) 
#>      y: (604500.0, 7034500.0) 
#>      x: (-622500.0, 6884500.0) 
#>  Variables:  
#>      vd: 
#>          Units: seconds

year_2020_gridded_data <- griddap(datasetx = human_activities_data_info, time = c("2020-03-18", "2020-03-19"))
#> info() output passed to x; setting base url to: https://erddap.emodnet.eu/erddap
head(year_2020_gridded_data$data)
#>         x       y                 time vd
#> 1 -622500 7034500 2020-04-01T00:00:00Z NA
#> 2 -621500 7034500 2020-04-01T00:00:00Z NA
#> 3 -620500 7034500 2020-04-01T00:00:00Z NA
#> 4 -619500 7034500 2020-04-01T00:00:00Z NA
#> 5 -618500 7034500 2020-04-01T00:00:00Z NA
#> 6 -617500 7034500 2020-04-01T00:00:00Z NA

EMODnetWCS: Access EMODnet Web Coverage Service data

This package EMODnetWFS uses Web Feature Services, hence it is limited to getting vector data. EMODnet also hosts raster data that can be accessed via Web Coverage Services (WCS). The R package EMODnetWCS makes these data available in R.

Citation

To cite EMODnetWFS, please use the output from citation(package = "EMODnetWFS").

citation(package = "EMODnetWFS")
#> 
#> To cite package 'EMODnetWFS' in publications use:
#> 
#>   Krystalli A, Fernández-Bejarano S, Salmon M (????). _EMODnetWFS:
#>   Access EMODnet Web Feature Service data through R_. R package version
#>   2.0.1.9001. Integrated data products created under the European
#>   Marine Observation Data Network (EMODnet) Biology project
#>   (EASME/EMFF/2017/1.3.1.2/02/SI2.789013), funded by the by the
#>   European Union under Regulation (EU) No 508/2014 of the European
#>   Parliament and of the Council of 15 May 2014 on the European Maritime
#>   and Fisheries Fund, <https://github.com/EMODnet/EMODnetWFS>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {{EMODnetWFS}: Access EMODnet Web Feature Service data through R},
#>     author = {Anna Krystalli and Salvador Fernández-Bejarano and Maëlle Salmon},
#>     note = {R package version 2.0.1.9001. Integrated data products created under the European Marine Observation Data Network (EMODnet) Biology project (EASME/EMFF/2017/1.3.1.2/02/SI2.789013), funded by the by the European Union under Regulation (EU) No 508/2014 of the European Parliament and of the Council of 15 May 2014 on the European Maritime and Fisheries Fund},
#>     url = {https://github.com/EMODnet/EMODnetWFS},
#>   }

Acknowledgements

This package was started by the Sheffield University during the EMODnet Biology WP4 data products workshop in June 2020. You can read the product story on the EMODnet-Biology portal

emodnetwfs's Issues

Layer that is a data.frame

library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'http://geo.vliz.be/geoserver/Dataportal/wfs'
#> ℹ Version: '2.0.0'
simple_filter_sf <- emodnet_get_layers(
    wfs = wfs,
    layers = "taxa",
    cql_filter = "scientificname='Abra'",
    reduce_layers = TRUE
)
#> Warning: Download of layer 'taxa' failed: Error in UseMethod("st_crs<-"): pas de méthode pour 'st_crs<-' applicable pour un objet de classe "data.frame"

^{Created on 2022-03-02 by the reprex package (v2.0.1)}

Handling large layers

# Start client
View(emodnet_wfs)
emodnet_client <- emodnet_init_wfs_client("seabed_habitats_general_datasets_and_products")

# See list of layers available and get Sandbanks of Europe
View(emodnet_get_wfs_info(emodnet_client))
subs <- emodnet_get_layers(emodnet_client, layers = "eusm2019_subs_full", reduce_layers = TRUE)

returns error

Warning: Download of layer 'eusm2019_subs_full' failed: Error in readBin(content, character()): R character strings are limited to 2^31-1 bytes

Is there a way to import such large layers through this package?

Confusion on how to use spatial ECQL predicates

I've been trying to include more detailed demos of using spatial ECQL predicates, following the following geoserver documentation: https://docs.geoserver.org/latest/en/user/filter/ecql_reference.html#ecql-expr

However I am hitting a lot of problems with getting filters to work and am also getting confusing results when they do work. I've isolated a couple of examples below to illustrate some of my confusion:

library(EMODnetWFS)
wfs_gs <- emodnet_init_wfs_client(service = "geology_seabed_substrate_maps")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/gtk/wfs'
#> ℹ Version: '2.0.0'

# start with a reference geometry. NOte call returns EWKT geometry MULTISURFACE
# which is not part of OCG standard
ref_sf <- emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
                       cql_filter ="id=10848", reduce_layers = TRUE) 
ref_sf
#> Simple feature collection with 1 feature and 30 fields
#> Geometry type: MULTISURFACE
#> Dimension:     XY
#> Bounding box:  xmin: 3183730 ymin: 2544099 xmax: 3184146 ymax: 2546943
#> CRS:           +init=epsg:3034
#>                      gml_id    id objectid  code country data_holder
#> 1 seabed_substrate_1m.10848 10848    10848 FR001  France        SHOM
#>       contact   scale original_scale original_grain_size
#> 1 [email protected] 1000000              0                SHOM
#>                                                                                                           references
#> 1 Garlan T. et Marches E. (2010) - DCSMM, Carte de nature des fonds de la sous-région Manche-Mer du Nord à 1/500 000
#>                                                                            comments
#> 1 Based on a compilation of maps from differents institutions between 1970 and 2010
#>   reclassification method sample_number original_substrate relation folk_16cl
#> 1                2      4             0                103        4         6
#>              folk_16cl_txt folk_7cl folk_7cl_txt folk_5cl folk_5cl_txt
#> 1 6. No data at this level        2      2. Sand        2     2. Sand 
#>   surface_feature conf_rs conf_s conf_d conf_tot shape_length   shape_area
#> 1                      NA     NA     NA       NA   0.05998287 8.306917e-05
#>                             geom
#> 1 MULTISURFACE (POLYGON ((318...

# create reference map
ref_map <- ref_sf %>% 
    sf::st_cast(to = "GEOMETRYCOLLECTION") %>% # mapview doesn't seem to work with 
    # EWKT so MULTIURFACE needs casting to a geometry collection
mapview::mapview(zcol = "folk_7cl_txt", burst = TRUE)

ref_map

#extract geometry as WKT
ref_geom_multisurface <- ref_sf %>% 
  sf::st_geometry() %>% sf::st_as_text()
ref_geom_multisurface
#> [1] "MULTISURFACE (POLYGON ((3184146 2544100, 3184141 2544099, 3184108 2544110, 3184055 2544148, 3184014 2544192, 3183974 2544250, 3183940 2544317, 3183914 2544387, 3183899 2544453, 3183890 2544510, 3183864 2544808, 3183730 2546928, 3183967 2546943, 3184058 2545528, 3184057 2545520, 3184054 2545456, 3184060 2545400, 3184057 2545336, 3184054 2545271, 3184058 2545205, 3184058 2545118, 3184090 2545026, 3184103 2544826, 3184102 2544806, 3184107 2544751, 3184098 2544557, 3184103 2544360, 3184103 2544317, 3184139 2544137, 3184146 2544100)))"

# Use WKT to construct DWITHIN query - FAILS with EWKT geometry MULTISURFACE event though that's the geometry type the data is returned as.
emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
                       cql_filter = paste0("DWITHIN(geom,", ref_geom_multisurface, ", 0.5, kilometers)"),
                       reduce_layers = TRUE ) 
#> Warning: Download of layer 'seabed_substrate_1m' failed: Error: Cannot open "/private/var/folders/8p/87cqdx2s34vfvcgh04l6z72w0000gn/T/Rtmpmzke5V/file1e416b4c119.gml"; The source could be corrupt or not supported. See `st_drivers()` for a list of supported formats.
#> NULL



# Transform to GEOMETRYCOLLECTION. 
ref_geom_colllection <- ref_sf %>% 
    sf::st_cast(to = "GEOMETRYCOLLECTION") %>% 
  sf::st_geometry() %>% sf::st_as_text()

# Use WKT to construct DWITHIN query - 
geom_colllection_res <- emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
                       cql_filter = paste0("DWITHIN(geom,", ref_geom_colllection, ", 0.5, kilometers)"),
                       reduce_layers = TRUE ) 
# WORKS
geom_colllection_res
#> Simple feature collection with 16 features and 30 fields
#> Geometry type: MULTISURFACE
#> Dimension:     XY
#> Bounding box:  xmin: 2333204 ymin: 2842851 xmax: 2751690 ymax: 3525843
#> CRS:           +init=epsg:3034
#> First 10 features:
#>                       gml_id    id objectid   code country data_holder
#> 1  seabed_substrate_1m.17209 17209    17209 IE-016 Ireland         GSI
#> 2  seabed_substrate_1m.17210 17210    17210 IE-016 Ireland         GSI
#> 3  seabed_substrate_1m.17231 17231    17231 IE-016 Ireland         GSI
#> 4  seabed_substrate_1m.17232 17232    17232 IE-016 Ireland         GSI
#> 5  seabed_substrate_1m.17225 17225    17225 IE-016 Ireland         GSI
#> 6  seabed_substrate_1m.17226 17226    17226 IE-016 Ireland         GSI
#> 7  seabed_substrate_1m.17227 17227    17227 IE-016 Ireland         GSI
#> 8  seabed_substrate_1m.17229 17229    17229 IE-016 Ireland         GSI
#> 9  seabed_substrate_1m.17230 17230    17230 IE-016 Ireland         GSI
#> 10 seabed_substrate_1m.17233 17233    17233 IE-016 Ireland         GSI
#>                  contact   scale original_scale original_grain_size references
#> 1  [email protected] 1000000              0                Folk           
#> 2  [email protected] 1000000              0                Folk           
#> 3  [email protected] 1000000              0                Folk           
#> 4  [email protected] 1000000              0                Folk           
#> 5  [email protected] 1000000              0                Folk           
#> 6  [email protected] 1000000              0                Folk           
#> 7  [email protected] 1000000              0                Folk           
#> 8  [email protected] 1000000              0                Folk           
#> 9  [email protected] 1000000              0                Folk           
#> 10 [email protected] 1000000              0                Folk           
#>                                                                comments
#> 1  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 2  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 3  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 4  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 5  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 6  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 7  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 8  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 9  Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 10 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#>    reclassification method sample_number original_substrate relation folk_16cl
#> 1                 2      2             0                           0         6
#> 2                 2      2             0                           0         6
#> 3                 2      2             0                           0         6
#> 4                 2      2             0                           0         6
#> 5                 2      2             0                           0         6
#> 6                 2      2             0                           0         6
#> 7                 2      2             0                           0         6
#> 8                 2      2             0                           0         6
#> 9                 2      2             0                           0         6
#> 10                2      2             0                           0         6
#>               folk_16cl_txt folk_7cl                folk_7cl_txt folk_5cl
#> 1  6. No data at this level        3 3. Coarse-grained sediment         3
#> 2  6. No data at this level        3 3. Coarse-grained sediment         3
#> 3  6. No data at this level        4           4. Mixed sediment        4
#> 4  6. No data at this level        4           4. Mixed sediment        4
#> 5  6. No data at this level        4           4. Mixed sediment        4
#> 6  6. No data at this level        4           4. Mixed sediment        4
#> 7  6. No data at this level        4           4. Mixed sediment        4
#> 8  6. No data at this level        4           4. Mixed sediment        4
#> 9  6. No data at this level        4           4. Mixed sediment        4
#> 10 6. No data at this level        4           4. Mixed sediment        4
#>                   folk_5cl_txt surface_feature conf_rs conf_s conf_d conf_tot
#> 1  3. Coarse-grained sediment                       NA     NA     NA       NA
#> 2  3. Coarse-grained sediment                       NA     NA     NA       NA
#> 3            4. Mixed sediment                      NA     NA     NA       NA
#> 4            4. Mixed sediment                      NA     NA     NA       NA
#> 5            4. Mixed sediment                      NA     NA     NA       NA
#> 6            4. Mixed sediment                      NA     NA     NA       NA
#> 7            4. Mixed sediment                      NA     NA     NA       NA
#> 8            4. Mixed sediment                      NA     NA     NA       NA
#> 9            4. Mixed sediment                      NA     NA     NA       NA
#> 10           4. Mixed sediment                      NA     NA     NA       NA
#>    shape_length   shape_area                           geom
#> 1    0.01020992 5.222959e-06 MULTISURFACE (POLYGON ((252...
#> 2    1.99591150 1.361911e-01 MULTISURFACE (POLYGON ((252...
#> 3    8.59975782 2.658686e-01 MULTISURFACE (POLYGON ((256...
#> 4    0.14909377 8.219173e-04 MULTISURFACE (POLYGON ((254...
#> 5    0.12131420 9.190709e-04 MULTISURFACE (POLYGON ((251...
#> 6    0.06035866 2.032417e-04 MULTISURFACE (POLYGON ((251...
#> 7    0.21341242 2.291320e-03 MULTISURFACE (POLYGON ((251...
#> 8    0.18912688 1.402354e-03 MULTISURFACE (POLYGON ((253...
#> 9    0.46555359 7.035125e-03 MULTISURFACE (POLYGON ((252...
#> 10   0.19845728 1.973919e-03 MULTISURFACE (POLYGON ((255...

# However returns very strange results!
ref_map + geom_colllection_res %>% 
    sf::st_cast(to = "GEOMETRYCOLLECTION") %>%
mapview::mapview(zcol = "folk_7cl_txt", burst = TRUE)

I also tried to test out predicate TOUCHES. Seems to fail with some geometry types:

# try to use TOUCHES predicate with GEOMETRYCOLLECTION - FAILS!
emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
                       cql_filter = paste0("TOUCHES(geom,", ref_geom_colllection,")"),
                       reduce_layers = TRUE ) 
#> Warning: Download of layer 'seabed_substrate_1m' failed: Error: Cannot open "/private/var/folders/8p/87cqdx2s34vfvcgh04l6z72w0000gn/T/Rtmpmzke5V/file1e413fb143f5.gml"; The source could be corrupt or not supported. See `st_drivers()` for a list of supported formats.
#> NULL


# Transform to single POLYGON and try again
ref_geom <- ref_sf %>% 
    sf::st_cast(to = "GEOMETRYCOLLECTION") %>% 
    sf::st_collection_extract(type = "POLYGON") %>% 
  sf::st_geometry() %>% sf::st_as_text()


# WORKS - although no features match the predicate. I guess polygons boundary do
# not share a single point?
emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
                       cql_filter = paste0("TOUCHES(geom,", ref_geom,")"),
                       reduce_layers = TRUE ) 
#> Simple feature collection with 0 features and 30 fields
#> Bounding box:  xmin: NA ymin: NA xmax: NA ymax: NA
#> CRS:           +init=epsg:3034
#>  [1] gml_id              id                  objectid           
#>  [4] code                country             data_holder        
#>  [7] contact             scale               original_scale     
#> [10] original_grain_size references          comments           
#> [13] reclassification    method              sample_number      
#> [16] original_substrate  relation            folk_16cl          
#> [19] folk_16cl_txt       folk_7cl            folk_7cl_txt       
#> [22] folk_5cl            folk_5cl_txt        surface_feature    
#> [25] conf_rs             conf_s              conf_d             
#> [28] conf_tot            shape_length        shape_area         
#> [31] geom               
#> <0 rows> (or 0-length row.names)

^{Created on 2021-11-26 by the reprex package (v2.0.1)}

So it's really not clear from the ECQL literal docs docs what types of geometries should work for different predicates. The docs relating to a geometry expression state: All standard geometry types are supported: POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, GEOMETRYCOLLECTION.
So I'm finding it hard to understand why TOUCHES doesn't work with a GEOMETRYCOLLECTION (when it does for DWITHIN yet POLYGON works for TOUCHES

It's taken a lot of time digging and I'm still left in a lot of confusion as to how to implement spatial filtering in particular and digging through docs on these topics online felt like a massive rabbit hole.

Given how much effort it took and how much confusion it generated in me, it might be safe to assume that users will have similar issues. I feel it would be good for firstly myself to understand better and inverst in more detailed documentation so that are users can successfully make use of filtering.

So my questions are:

• Do we know someone that has a solid overview of these topics that could at least point to the correct documentation or even better, explain a few thing?
• Perhaps we should better understand the stack behind the service which I imagine also affects the filtering capabilities available?

For now, the only example I've got to work correctly is the BBOX predicate so it is the only one I've included as an example in the filtering vignette so far (#29 ).

I've also been going back to basics and trying to collate useful information to help me understand what's going on here: https://annakrystalli.me/spatial-notes/ (in this repo: https://github.com/annakrystalli/spatial-notes)

But any ideas on how we can get a better understanding on these topics would be greatly appreciated @salvafern

Separate service testing from package testing and only report package testing result on README.

It feels like a lot of the failing tests recently relate more to failing services. While I'm glad we are testing them (and look forward to implementing during runtime too as discussed in #24 ), I feel it's not appropriate for our build to be showing as failing when it's services that are the issue.

The best way to implement it might well be separating service testing into a different GitHub Action. What do you think @salvafern ?

Add CQL and OGC filters

ows4r allows to pass CQL and OGC filters to the WFS requests. We should add this option as some layers are too large to be queried. For example, the EMODnet Biology occurrences (+30M at the moment!).

The CQL and OGC filters are passed to getFeatures() with the arguments cql_filter and filter. The filter has to be URL encoded. This has been fixed in the development version but it is not yet on CRAN.

These filters can be added in ews_get_layer (@annakrystalli please correct me if I'm wrong or if you have a better idea!):

EMODnetWFS/R/layers.R

Line 79 in 5ac0e2f

ews_get_layer <- function(x, wfs, suppress_warnings = FALSE){

This test works:

version
#>                _                           
#> platform       x86_64-w64-mingw32          
#> arch           x86_64                      
#> os             mingw32                     
#> system         x86_64, mingw32             
#> status                                     
#> major          4                           
#> minor          0.2                         
#> year           2020                        
#> month          06                          
#> day            22                          
#> svn rev        78730                       
#> language       R                           
#> version.string R version 4.0.2 (2020-06-22)
#> nickname       Taking Off Again

library(EMODnetWFS)
library(ows4R)
library(leaflet)
library(tibble)

# Start client
wfs_bio <- emodnet_init_wfs_client("biology_occurrence_data")
#> v WFS client created succesfully
#> i Service: 'http://geo.vliz.be/geoserver/Dataportal/wfs'
#> i Version: '2.0.0'

# Get capabilities
wfs_bio_caps <- wfs_bio$getCapabilities()

# Find basic occurrence data
wfs_bio_caps_feat <- wfs_bio_caps$findFeatureTypeByName("Dataportal:eurobis-obisenv_basic")

# Get data
cql_filter = "scientificnameaccepted='Posidonia oceanica'"

ly_qry <- wfs_bio_caps_feat$getFeatures(cql_filter = URLencode(cql_filter))

ly_qry
# A tibble: 509 x 10
   fid           id datasetid     datecollected       decimallongitude decimallatitude scientificname aphiaid   scientificnamea~ coordinateuncer~
   <chr>      <int> <chr>         <dttm>                         <dbl>           <dbl> <chr>          <chr>     <chr>                       <dbl>
 1 eurobis~ 5805417 http://www.e~ 2006-09-05 00:00:00            13.7             45.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
 2 eurobis~ 5805426 http://www.e~ 2006-09-05 00:00:00            13.7             45.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
 3 eurobis~ 5805435 http://www.e~ 2006-09-05 00:00:00            13.7             45.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
 4 eurobis~ 5805444 http://www.e~ 2006-09-05 00:00:00            13.7             45.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
 5 eurobis~ 5805453 http://www.e~ 2006-09-05 00:00:00            13.7             45.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
 6 eurobis~ 5805462 http://www.e~ 2006-09-05 00:00:00            13.7             45.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
 7 eurobis~ 5805471 http://www.e~ 2006-09-05 00:00:00            13.7             45.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
 8 eurobis~ 5869917 http://www.e~ NA                             38.6             41.6 Posidonia oce~ http://m~ Posidonia ocean~           849349
 9 eurobis~ 5869918 http://www.e~ NA                             38.6             41.6 Posidonia oce~ http://m~ Posidonia ocean~           849349
10 eurobis~ 5869919 http://www.e~ NA                              6.78            42.5 Posidonia oce~ http://m~ Posidonia ocean~               NA
# ... with 499 more rows

^{Created on 2020-12-02 by the reprex package (v0.3.0)}

Add EMODnet Chemistry contaminants WFS

From Britt:

As far as I can tell, there is only one (relatively recent) web service missing, namely that of the EMODnet Chemistry contaminants (URL: https://nodc.inogs.it/geoserver/Contaminants/wfs). It would be nice if you could also include this in the package J

emodnet_get_all_wfs_info() timeouts

Might affect this test:

EMODnetWFS/tests/testthat/test-info.R

Line 12 in ef53a44

test_that("wfs all info works", {

library(EMODnetWFS)

emodnet_get_all_wfs_info()
#> Warning in CPL_crs_from_input(x): GDAL Message 1: +init=epsg:XXXX syntax is
#> deprecated. It might return a CRS with a non-EPSG compliant axis order.
#> Error in curl::curl_fetch_memory(url, handle = handle): Timeout was reached: [emodnet02.cineca.it] Connection timed out after 10002 milliseconds

sessionInfo()
#> R version 4.0.2 (2020-06-22)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 18363)
#> 
#> Matrix products: default
#> 
#> locale:
#> [1] LC_COLLATE=English_United States.1252 
#> [2] LC_CTYPE=English_United States.1252   
#> [3] LC_MONETARY=English_United States.1252
#> [4] LC_NUMERIC=C                          
#> [5] LC_TIME=English_United States.1252    
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] EMODnetWFS_0.0.2
#> 
#> loaded via a namespace (and not attached):
#>  [1] Rcpp_1.0.5         pillar_1.4.6       ows4R_0.1-5        compiler_4.0.2    
#>  [5] highr_0.8          class_7.3-17       tools_4.0.2        digest_0.6.25     
#>  [9] checkmate_2.0.0    jsonlite_1.7.0     evaluate_0.14      lifecycle_0.2.0   
#> [13] tibble_3.0.3       lattice_0.20-41    pkgconfig_2.0.3    rlang_0.4.7       
#> [17] cli_2.0.2          DBI_1.1.0          curl_4.3           yaml_2.2.1        
#> [21] rgdal_1.5-16       xfun_0.16          e1071_1.7-3        geometa_0.6-3     
#> [25] dplyr_1.0.2        stringr_1.4.0      httr_1.4.2         knitr_1.29        
#> [29] fs_1.5.0           generics_0.0.2     vctrs_0.3.2        askpass_1.1       
#> [33] classInt_0.4-3     grid_4.0.2         tidyselect_1.1.0   glue_1.4.1        
#> [37] sf_0.9-6           R6_2.4.1           fansi_0.4.1        XML_3.99-0.5      
#> [41] rmarkdown_2.5      sp_1.4-4           tidyr_1.1.1        purrr_0.3.4       
#> [45] magrittr_1.5       usethis_1.6.3      backports_1.1.7    units_0.6-7       
#> [49] ellipsis_0.3.1     htmltools_0.5.0    assertthat_0.2.1   KernSmooth_2.23-17
#> [53] stringi_1.4.6      openssl_1.4.2      crayon_1.3.4

^{Created on 2021-01-20 by the reprex package (v0.3.0)}

Rename package to all lower-case?

Submit to CRAN

To be checked:

• R-CMD failing: https://github.com/EMODnet/EMODnetWFS/actions/runs/497401361
• Issue #9

example missing in readme

Hi all,
I noticed that there is a section for an example in the readme but the example itself is missing https://github.com/EMODnet/EMODnetWFS#example

switch to https wherever possible

Add codemeta.json file :-)

Improve communication of service status to users

I feel we need good communication channels set up with service maintainers to be able to quickly figure out whether problems are from bugs in the package or service issues.

We should discuss at the next meeting.

I think at the very least a list of contact details for each service is necessary but would be interesting to explore whether there are better options.

Bug to explore

emodnet_get_wfs_info(wfs_cml)

layers <- c("mediseh_zostera_m_pnt","mediseh_posidonia_nodata")

emodnet_get_layer_info(wfs = wfs_cml, layers = layers)

Error downloading layer

Hey Anna, I'm trying to access some of the big seabed map layers, am hitting an error when running:

wfs_seabedgeneral <- emodnet_init_wfs_client(service = "seabed_habitats_general_datasets_and_products")
hab_dat <- emodnet_get_layers(wfs = wfs_seabedgeneral, layers = "eusm2019_subs_full")

Error message is:

Warning: Download of layer 'eusm2019_subs_full' failed: Error in readBin(content, character()): R character strings are limited to 2^31-1 bytes

Seems like some problem parsing the layer?

Product story for EMODnet as a vignette

Hi @annakrystalli !

I have written a vignette on how to use this package with some more examples. The idea is to use it as a product story, but I figured it would be good to include it in the package itself.

Could you please add a short introduction paragraph?

EMODnetWFS/vignettes/emodnetwfs.Rmd

Line 17 in f9774fa

*write short introduction here: aim of the package, why it was created*

And of course let me know if there is anything you want to change/improve/suggestions etc 😊

Add contributing guide

As per https://devguide.ropensci.org/collaboration.html#contributing-guide

Here for instance there's usage of testthis, and of httptest, for which one sentence or two could be added.

Check using crs during query

Use fledge?

https://fledge.cynkra.com/dev/

Wrong bathymetry results following tutorial

Following the tutorial, the "getbathymetry" function results in a wrong raster, which provides the altitudinal values, instead of the bathymethrycal. In fact, it seems that the bathymetry is masked with values of 0s. I think that the error is given by the "name" parameter. If it is changed to "emodnet:mean_2016", the resulting raster is correct, but then I cannot plot it.

Ignore inconsequential build fails from failing workflow

https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idcontinue-on-error

assumption on CRS when downloading data from geology_seabed_substrate_maps

When downloading seabed substrate data from geology_seabed_substrate_maps , the CRS is not provided by the server. The CRS is then assumed to be 4326. This is incorrect.

wfs <- emodnet_init_wfs_client(service="geology_seabed_substrate_maps")
test <- emodnet_get_layers(wfs = wfs, layers = "seabed_substrate_1m", reduce_layers = T)

gives: Warning: crs missing. Set to default 4326

Working from the assumption that it is 3857 also yields incorrect maps, so I am a bit at a loss at what the correct CRS is.

version 1.0.0 in R3.6.1

Map all requests somewhere

Related to #64

It'd be good to have, for developers, some sort of table of how requests are made when a function makes requests. I.e. is it using httr or something else.

Edge case for emodnet_get_layer_info() with biology_occurrence_data

When I try to get info about the layer "biotic_occurrences" I also get the ones about "abiotic_occurrences" as if there were some sort of grepl() rather than == somewhere.

library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'http://geo.vliz.be/geoserver/Dataportal/wfs'
#> ℹ Version: '2.0.0'
emodnet_get_wfs_info(wfs)
#> # A tibble: 27 × 9
#>    data_source service_name         service_url layer_namespace layer_name title
#>    <chr>       <chr>                <chr>       <chr>           <chr>      <chr>
#>  1 emodnet_wfs biology_occurrence_… http://geo… Dataportal      abiotic_o… Data…
#>  2 emodnet_wfs biology_occurrence_… http://geo… Dataportal      biotic_ob… Data…
#>  3 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis-o… EMOD…
#>  4 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis-o… EMOD…
#>  5 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis-o… EMOD…
#>  6 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis-o… EMOD…
#>  7 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis_g… EMOD…
#>  8 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis_g… EMOD…
#>  9 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis_g… EMOD…
#> 10 emodnet_wfs biology_occurrence_… http://geo… Dataportal      eurobis_g… EMOD…
#> # … with 17 more rows, and 3 more variables: abstract <chr>, class <chr>,
#> #   format <chr>
emodnet_get_layer_info(wfs, c("biotic_observations", "abiotic_observations"))
#> Error in .f(.x[[i]], ...): tentative d'appliquer un objet qui n'est pas une fonction

^{Created on 2022-03-02 by the reprex package (v2.0.1)}

The solution that I'll implement in #46 is to unlist what I get from capabilities$findFeatureTypeByName and return unique rows (I checked that I also get this problem on the default branch).

Failing GitHub Actions

All our GAs are failing atm so I'm taking a closer look at them today.

1. update-pkgdown & http-tests:

• macOS-latest (devel): Looks like this emanates from package sf. Likely an issue with using binaries vs source.

  ERROR: dependencies ‘ows4R’, ‘sf’ are not available for package ‘EMODnetWFS’
  * removing 
  

• macOS-latest (release): change in epsg code (no longer returning +init). fixed in af9717b

• windows-latest:

  Error: Error: processing vignette 'emodnetwfs.Rmd' failed with diagnostics:
  package or namespace load failed for 'mapview' in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]):
   namespace 'terra' 1.4-22 is being loaded, but >= 1.5.12 is required
  

  This appears to be related to the fact that The update for {terra} to 1.5.12 from 1.4-22 was released on January 13 and it has been slow to appear as compiled binaries, which means that if the tools to compile R packages (e.g., the Rtools4 software for Windows) is not installed, it will not be possible to install 1.5.12 yet. We can either

  • just wait it out or
  • modify the GA to ensure Rtools4 is available for windows jobs, which will likely avoid other such future failures due to lagging binaries

• ubuntu-16.04 (release): Annoyingly it just says the job has failed with no further info. oldrel and 3.6 R pass though so it might be one of the issues described above.

---

2. Tests-Services:

This uses ubuntu-20.04 and seems to be installing package dependencies twice, once here with remotes::install_deps:

EMODnetWFS/.github/workflows/Test-Services.yaml

Lines 20 to 23 in b4f2f9b

	- name: Install dependencies
	run: |
	install.packages(c("remotes", "testthat"))
	remotes::install_deps(dependencies = TRUE)

and then again here with install.packages(".", repos = NULL, type="source"):

EMODnetWFS/.github/workflows/Test-Services.yaml

Lines 27 to 32 in b4f2f9b

	- name: Test Services
	run: |
	options(crayon.enabled = TRUE)
	install.packages(".", repos = NULL, type="source")
	test_service <- lapply(EMODnetWFS::emodnet_wfs$service_name, EMODnetWFS::emodnet_init_wfs_client);
	testthat::test_that("All services work", {testthat::expect_equal(class(test_service), "list")})

. I'll do some more investigations on this.

Overall I think most problems here are an issue with using binaries vs source. I found this discussion on the topic r-lib/actions#37 and while I generally prefer binaries, not least for their speed, at least when testing r devel we should likely be using source. I am happy to hear thoughts on whether we should also do this for windows @salvafern / @maelle.

In any case, I wills set it up for r-devel and dig a little deeper into the Test-Services GA issue and report back.

Geology web services on Linux

Geology servers cannot be supported on Linux yet.

• There's a curl hack but we can't use it yet. eblondel/ows4R#64
• The web services themselves might get updated.

In the meantime the solutions are

• to leave a comment here to let us know you've been trying to access these web services from Linux.
• to try and run code on another OS.
• to do the web service calls "yourself" using curl directly, with the aforementioned hack.

We don't recommend changing the security levels system wide.

Thanks for your patience.

Example of using BBOX() in cql_filter argument in emodnet_get_layers

Hi,

I wonder if it is possible to crop data when requesting a layer from a service.

Looking at the documentation there is a bbox(expr, num, num, num, num) that can be used in cql_filter, but I wonder what expr to provide if I want to crop the entire layer. Is this even possible? If so I think it would be helpful to add to the vignette about cql_filter().

Thanks once more for your work,

Jan Jaap

Shorten function names?

E.g. could emodnet_get_layer_info be emod_layer_info() or so?

Dependency on usethis - gert causes installation to fail on old ubuntu distributions

I tried to install the package in a old ubuntu distribution and failed due to a downstream dependency on the gert package, through the usethis package. Note usethis it is only used at the moment for stylish UI messages, if the problem persisted we could remove the dependency on usethis.

devtools::install_github("r-lib/gert")
# Downloading GitHub repo r-lib/gert@HEAD
# ✓  checking for file ‘/tmp/RtmpFV0XvN/remotes69bf6e4495fb/r-lib-gert-6585a65/DESCRIPTION’ (342ms)
# ─  preparing ‘gert’:
#   ✓  checking DESCRIPTION meta-information ... 
# ─  cleaning src
# ─  running ‘cleanup’
# ─  checking for LF line-endings in source and make files and shell scripts
# ─  checking for empty or unneeded directories
# ─  building ‘gert_1.1.0.tar.gz’
# 
# Installing package into ‘/data/home/salvadorf/R/x86_64-pc-linux-gnu-library/3.6’
# (as ‘lib’ is unspecified)
# * installing *source* package ‘gert’ ...
# ** using staged installation
# > curl::curl_download("http://r-lib.github.io/gert/get-libgit2-linux.sh","get-libgit2-linux.sh")
# > 
#   > 
#   > curl::curl_download('https://r-lib.github.io/gert/libgit2-1.1.0.x86_64_linux.tar.gz','bundle.tar.gz')
# > 
#   > 
#   Using static libgit2-1.1.0 for Linux x86_64
# Using PKG_CFLAGS=-DSTATIC_LIBGIT2 -I/tmp/Rtmp8ir5fb/R.INSTALL6a474e0e2c00/gert/libgit2/include
# Using PKG_LIBS=-L/tmp/Rtmp8ir5fb/R.INSTALL6a474e0e2c00/gert/libgit2/lib -lgit2 -lrt -lpthread -lssh2 -lssl -lcrypto -ldl -lpcre -lz
# Configuration OK!
#   ** libs
# [...]
# /usr/bin/ld: /tmp/Rtmp8ir5fb/R.INSTALL6a474e0e2c00/gert/libgit2/lib/libgit2.a(annotated_commit.c.o): unrecognized relocation (0x2a) in section `.text'
# /usr/bin/ld: final link failed: Bad value
# collect2: error: ld returned 1 exit status
# make: *** [gert.so] Error 1
# ERROR: compilation failed for package ‘gert’
# * removing ‘/data/home/salvadorf/R/x86_64-pc-linux-gnu-library/3.6/gert’
# Error: Failed to install 'gert' from GitHub:
#   (converted from warning) installation of package ‘/tmp/RtmpFV0XvN/file69bf33029d44/gert_1.1.0.tar.gz’ had non-zero exit status

sessionInfo()
#> R version 3.6.0 (2019-04-26)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 14.04.6 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/libblas/libblas.so.3.0
#> LAPACK: /usr/lib/lapack/liblapack.so.3.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] Rcpp_1.0.5        fansi_0.4.2       digest_0.6.25     crayon_1.3.4     
#>  [5] assertthat_0.2.1  magrittr_2.0.1    reprex_1.0.0      evaluate_0.14    
#>  [9] highr_0.8         stringi_1.5.3     rlang_0.4.10      cli_2.2.0        
#> [13] rstudioapi_0.13   fs_1.3.1          rmarkdown_2.6     tools_3.6.0      
#> [17] stringr_1.4.0     glue_1.4.2        xfun_0.20         yaml_2.2.1       
#> [21] compiler_3.6.0    htmltools_0.5.1.1 knitr_1.30

^{Created on 2021-01-28 by the reprex package (v1.0.0)}

emodnet_init_wfs_client should not return NULL when no service is created

It should error

Filtering using `PROPERTYNAME` in attribute functions is failing

Example in the ECQL filtering vignette are failing:

library(EMODnetWFS)
wfs_ha <- emodnet_init_wfs_client(service = "human_activities")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://ows.emodnet-humanactivities.eu/wfs'
#> ℹ Version: '2.0.0'
layer_attributes_summarise(wfs_ha, layer = "maritimebnds")
#> Error in sf::st_drop_geometry(.): st_drop_geometry only works with objects of class sf
layer_attributes_tbl(wfs_ha, layer = "maritimebnds")
#> Error in sf::st_drop_geometry(.): st_drop_geometry only works with objects of class sf
layer_attribute_inspect(wfs_ha, layer = "maritimebnds", attribute = "sitename")

^{Created on 2022-03-11 by the reprex package (v2.0.1)}

Running functions with debugging shows that it is the filtering with PROPERTIES filter that seems to not be working:

EMODnetWFS/R/layer_attributes.R

Lines 143 to 144 in 4afc00b

	wfs$getFeatures(layer, PROPERTYNAME=paste(attributes, collapse = ",")) %>%
	sf::st_drop_geometry() %>% tibble::as_tibble()

EMODnetWFS/R/layer_attributes.R

Line 99 in 4afc00b

attribute_vector <- wfs$getFeatures(layer, PROPERTYNAME=attribute)[[attribute]]

Bug with another server

library("EMODnetWFS")
emodnet_init_wfs_client(service = "chemistry_marine_litter")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> x WFS client creation failed.
#> ℹ Service: 'https://www.ifremer.fr/services/wfs/emodnet_chemistry2'
#> Error in curl::curl_fetch_memory(url, handle = handle): OpenSSL SSL_connect: Connexion ré-initialisée par le correspondant in connection to www.ifremer.fr:443

^{Created on 2022-03-02 by the reprex package (v2.0.1)}

CRS representations across services appear to still use PROJ4 strings

@salvafern I have noticed that sf objects returned for the servers often encode CRSs in the deprecated PROJ4 string format (eg +init=epsg:3034 returned by emodnet_get_layers(service="geology_seabed_substrate_maps", layers = "seabed_substrate_1m", reduce_layers = T)).

I know there's been big changes in the geospatial software landscape in terms of expressing CRSs as a result of changes in versions of the PROJ library > 5 & GDAL > 3 (see for example discussions of handling changes in package rgdal and here in sf. So my questions are:

• Can we check which versions of PROJ & GDAL the servers are using?
• Is there are reason CRSs are being expressed using deprecated PROJ4 strings?
• Do we need to assess:
  • potential risks of reduced accuracy as a result of using deprecated PROJ strings?
  • upcoming problems from using deprecated representations?

Remove EMODnetWFS:: when referencing emodnet_wfs

Could the function names clarify which ones perform an HTTP request?

getting seabed_substrate_1m layer from geology_seabed_substrate_maps fails in EMODnetWFS 2.0.0

Just installed EMODnetWFS and getting the seabed_substrate_1m layer from geology_seabed_substrate_maps now fails. I think that it also failed for 1.0.0, but I am not sure. This is on windows, R3.6.1.

wfs <- emodnet_init_wfs_client(service="geology_seabed_substrate_maps")
Loading ISO 19139 XML schemas...
Loading ISO 19115 codelists...
√ WFS client created succesfully
i Service: 'https://drive.emodnet-geology.eu/geoserver/gtk/wfs'
i Version: '2.0.0'
test <- emodnet_get_layers(wfs = wfs, layers = "seabed_substrate_1m", reduce_layers = T)
Space required after the Public Identifier
SystemLiteral " or ' expected
SYSTEM or PUBLIC, the URI is missing
Entity 'nbsp' not defined
Warning: Download of layer 'seabed_substrate_1m' failed: Error: 1: Space required after the Public Identifier
2: SystemLiteral " or ' expected
3: SYSTEM or PUBLIC, the URI is missing
4: Entity 'nbsp' not defined

The error is thus "Space required after the Public Identifier". Some digging in emodnet_get_layers() reveals that this comes from a call to purrr::map2

debug: out <- purrr::map2(.x = layers, .y = cql_filter, ~ews_get_layer(.x,
wfs, cql_filter = .y), wfs, suppress_warnings) %>% stats::setNames(layers)

Space required after the Public Identifier
SystemLiteral " or ' expected
SYSTEM or PUBLIC, the URI is missing
Entity 'nbsp' not defined
Warning: Download of layer 'seabed_substrate_1m' failed: Error: 1: Space required after the Public Identifier
2: SystemLiteral " or ' expected
3: SYSTEM or PUBLIC, the URI is missing
4: Entity 'nbsp' not defined

at this time layers is a single string "seabed_substrate_1m" and cql_filter is NA. The version of purrr is:

packageDescription("purrr")
Package: purrr
Title: Functional Programming Tools
Version: 0.3.4

Any pointers?

To add: getting layers from chemistry_marine_litter as per the example on the github main page works fine.

Change example in vignette as long as server is failing?

I.e. as a temporary workaround to #39

add message for download?

use VLIZ-hosted web services in tests and vignettes?

Not in all obviously, but this might make things easier for communication given that @salvafern is an author of this package?

Feature attribute metadata?

Hey @salvafern ,

I was just wondering, are metadata on the attributes of each layer available through the server? For example, when I download windfarm data:

library(EMODnetWFS)
emodnet_get_layers(service = "human_activities", layers = "windfarms")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://ows.emodnet-humanactivities.eu/wfs'
#> ℹ Version: '2.0.0'
#> Warning: crs missing. Set to default 4326
#> $windfarms
#> Simple feature collection with 219 features and 9 fields
#> Geometry type: POINT
#> Dimension:     XY
#> Bounding box:  xmin: -16.4 ymin: 27.757 xmax: 25.97 ymax: 65.74
#> Geodetic CRS:  WGS 84
#> First 10 features:
#>           gml_id country n_turbines power_mw     status start  dist_coast notes
#> 1    windfarms.1   Spain          1      5.0 Production  2018  3211.17407  <NA>
#> 2   windfarms.24  France         NA      8.0 Production  2015 10400.68077  <NA>
#> 3  windfarms.190  Greece         NA     24.0    Planned  <NA>  1711.23910  <NA>
#> 4    windfarms.4   Spain          1      4.5 Production  2013    22.36593  <NA>
#> 5   windfarms.58 Denmark         NA    240.0    Planned  <NA>  6518.04614  <NA>
#> 6    windfarms.6   Spain          5     50.0 Production  2017  3211.17407  <NA>
#> 7   windfarms.59 Denmark         NA    160.0    Planned  <NA>  8396.57314  <NA>
#> 8   windfarms.60 Denmark         NA    120.0    Planned  <NA>  7725.66890  <NA>
#> 9   windfarms.63 Denmark         NA    320.0    Planned  <NA>  9328.48812  <NA>
#> 10  windfarms.75  Norway         NA     10.0    Planned  <NA>  9225.51976  <NA>
#>      ljstatus             the_geom
#> 1  production POINT (-15.36 28.04)
#> 2  production POINT (-2.778 47.24)
#> 3     planned  POINT (25.85 40.83)
#> 4  production  POINT (-15.4 27.85)
#> 5     planned  POINT (11.01 55.59)
#> 6  production POINT (-15.36 28.04)
#> 7     planned   POINT (9.82 55.15)
#> 8     planned  POINT (10.41 56.03)
#> 9     planned  POINT (11.11 55.06)
#> 10    planned   POINT (5.01 59.15)

^{Created on 2021-08-20 by the reprex package (v2.0.1)}

How can I get information on for example, what the units dist_coast is in?

Layer selection

Now with #56 layers (features) can be selected by exact name again. It's good as it avoids getting "abiotic" stuff when querying "biotic" for instance.

Now, exact means "with the namespace" e.g. ms:sl_fishing.

How to we expect users to feed layer names? I'd say

• full layer name with namespace
• or part of that, with an argument exact that we'd pass to caps$findFeatureTypeByName().

Should exact be TRUE or FALSE by default? I'd say TRUE?

Revisit default wfs service

Currently, the default service if no service is specified in many of the functions is seabed_habitats_individual_habitat_map_and_model_datasets. This is somewhat arbitrary and the result of that being the service of interest when I first started working on the package.

• Should we remove the default value for the service argument? I feel it makes more sense for users to have to be explicit in all cases about the service they want to use and not fall back on any defaults.
• If we do want a default, is another service more sensible?

Let me know your thoughts @salvafern & @LennertSchepers !

Services not available?

It seems that some of the services mentioned in the documentation are not available anymore. I cannot get any data from
seabed_habitats_general_datasets_and_products or seabed_habitats_individual_habitat_map_and_model_datasets. This means that the default in emodnet_init_wfs_client() also does not work.

Using v1.0.0 in R3.6.1.

use httptest in tests

from #34

Add user agent to queries

And if not possible open a FR with ows4R

Get a DOI for package

Get a DOI form Zenodo (as ows4r does). Include funding details

need to investigate

library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "biology")
layer_attribute_descriptions(wfs, layer = "Species_gridded_abundances_2year")

How to use cql

• If I remember correctly it was not supported on all services?
• I think #49 is due to my using a wrong cql, should there be some try/catch somewhere?
• How to know what variables a layer has and what values they take? It does not seem emodnet_get_layer_info() has that info so what kind of call could help with that?

Bug human_activities + majorcatches

library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "human_activities")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://ows.emodnet-humanactivities.eu/wfs'
#> ℹ Version: '2.0.0'

   or_filter_sf <- emodnet_get_layers(
        wfs = wfs,
        layers = "majorcatches",
        cql_filter = "country='Baltic Sea' OR country='Bulgaria'",
        reduce_layers = TRUE
    )
#> Warning: Download of layer 'majorcatches' failed: Error: Cannot open "/tmp/RtmpZxxHz3/file630649c69027.gml"; The source could be corrupt or not supported. See `st_drivers()` for a list of supported formats.

^{Created on 2022-03-02 by the reprex package (v2.0.1)}

Create an overview of all the data?

I.e. a searchable table of all services & layers, e.g. as a regularly updated article in the pkgdown website.