ropensci/rdefra

rdefra: Interact with the UK AIR Pollution Database from DEFRA

rdefra: Interact with the UK AIR Pollution Database from DEFRA

The package rdefra allows to retrieve air pollution data from the Air Information Resource UK-AIR of the Department for Environment, Food and Rural Affairs in the United Kingdom. UK-AIR does not provide a public API for programmatic access to data, therefore this package scrapes the HTML pages to get relevant information.

This package follows a logic similar to other packages such as waterData and rnrfa: sites are first identified through a catalogue, data are imported via the station identification number, then data are visualised and/or used in analyses. The metadata related to the monitoring stations are accessible through the function ukair_catalogue(), missing stations’ coordinates can be obtained using the function ukair_get_coordinates(), and time series data related to different pollutants can be obtained using the function ukair_get_hourly_data().

DEFRA’s servers can handle multiple data requests, therefore concurrent calls can be sent simultaneously using the parallel package. Although the limit rate depends on the maximum number of concurrent calls, traffic and available infrastructure, data retrieval is very efficient. Multiple years of data for hundreds of sites can be downloaded in only few minutes.

For similar functionalities see also the openair package, which relies on a local copy of the data on servers at King’s College (UK), and the ropenaq which provides UK-AIR latest measured levels (see https://uk-air.defra.gov.uk/latest/currentlevels) as well as data from other countries.

Installation

Get the released version from CRAN:

install.packages("rdefra")

Or the development version from GitHub using the package remotes:

install.packages("remotes")
remotes::install_github("ropensci/rdefra")

Load the rdefra package:

library(rdefra)

Functions

The package logic assumes that users access the UK-AIR database in the following steps:

1. Browse the catalogue of available stations and selects some stations of interest (see function ukair_catalogue()).
2. Get missing coordinates (see function ukair_get_coordinates()).
3. Retrieves data for the selected stations (see functions ukair_get_site_id() and ukair_get_hourly_data()).

For an in-depth description of the various functionalities and example applications, please refer to the package vignette.

Meta

• This package and functions herein are part of an experimental open-source project. They are provided as is, without any guarantee.
• Please report any issues or bugs.
• License: GPL-3
• This package was reviewed by Maëlle Salmon and Hao Zhu for submission to ROpenSci (see review here) and the Journal of Open Source Software (see review here).
• Cite rdefra: citation(package = "rdefra")

rdefra's Issues

Migrate CI

Migrate CI from travis-ci.org to GitHub Actions

Question about a new study

http://www.bbc.com/news/uk-37213786 -> Do you think they used DEFRA data as accessible via your package?

Merge README_files folder to assets?

Hi, @cvitolo,
The README_files folder and the assets folder seem to be the same thing. Maybe you want to merge them? ;)

New maintainer

The rdefra package needs a new maintainer. @cherylisabella kindly accepted to take on this role, thank you!

Let's document here the steps we take :)

Wrong permissions? I see ALL the ropenscilabs repositories in my travis!

@sckott somebody must have given me some sort of permission on ropenscilabs, I see ALL the ropenscilabs repositories in my travis dashboard. Would you be able to fix this? Many thanks!

pkgdown build failure in rdefra_vignette.Rmd

See https://dev.ropensci.org/job/rdefra/lastBuild/console

── Building articles ───────────────────────────────────────────────────────────
Writing 'articles/index.html'
Reading 'vignettes/rdefra_vignette.Rmd'
Error : callr subprocess failed: invalid multibyte string at '<a0>\<c8>�<c9>�'
Error: callr subprocess failed: callr subprocess failed: invalid multibyte string at '<a0>\<c8>�<c9>�'
Execution halted

dependency on rgdal

👋 @cvitolo! this package uses rgdal that has been retired from CRAN.

Are you still able to maintain rdefra? I have a vague memory of our trying to find new maintainers?

Install instructions for dev version

The install instructions for the development version don't work for me:

devtools::install_github('cvitolo/rdefra')

Downloading GitHub repo cvitolo/rdefra@master
from URL https://api.github.com/repos/cvitolo/rdefra/zipball/master
Error in stop(github_error(request)) : Not Found (404)

Should README.md refer to ropenscilabs repository instead? This does work for me:

devtools::install_github('ropenscilabs/rdefra')

Vignette failure

👋 @cvitolo!

We see a vignette failure https://github.com/r-universe/ropensci/runs/5447146461?check_suite_focus=true

Locally I can build the vignette, and I re-launched the workflow above but the timeout appeared again.

If the web service rdefra is wrapping is known to be slow/not always working, maybe you could pre-compute the vignette? https://ropensci.org/blog/2019/12/08/precompute-vignettes/

New maintainer wanted :-)

Note #17

SiteID = NA causes hanging errors

Hi - the package is great. Thanks for making it available.

I get errors with most functions when I include sites where SiteID = NA, eg:

library(rdefra)
stations_raw <- ukair_catalogue()
stations <- ukair_get_coordinates(stations_raw)

Gives the following error:
Error in curl::curlfetchmemory(url, handle = handle) : Failure when receiving data from the peer

Though the following works instantly:

library(rdefra)
stations_raw <- ukair_catalogue()
stations <- ukair_get_coordinates(stations_raw[!is.na(stations_raw$EMEP.Site.ID),])

Thanks

PROJ 6 failures in rdefra

I'm testing rgdal and its reverse dependencies with PROJ 6 and -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H. The CRS offered by PROJ >= 6 are changing, so that even though rgdal is still using the proj_api.h interface (for a few more months), the underlying CRS parameters are changing. Since few are using PROJ 6 yet, you have some time to fix this as rgdal will be built with < PROJ 6 metadata files in most cases. However, PROJ 6 was released 1 March, so you'll need to adapt. I attach my check files:

testthat.Rout.zip
00check.log

archived on CRAN

https://cran.rstudio.com/web/packages/rdefra/ hey @cvitolo it's been archived on CRAN. can we help get it updated?

units in README

currently it's "attributes(my1)$units

> NULL"

maybe you forgot to re-knit the README?

Test failure

👋 @cvitolo!

As seen in https://github.com/r-universe/ropensci/runs/5208774810?check_suite_focus=true via https://ropensci.r-universe.dev/ui#builds

Running the tests in 'tests/testthat.R' failed.
Last 13 lines of output:
  
  == Skipped tests ===============================================================
  * Call to ukair_get_hourly_data for 1 year failed! (1)
  * Call to ukair_get_hourly_data for 2 years failed! (1)
  
  == Failed tests ================================================================
  -- Failure (test-metadata.R:23:5): Find site identification number from the UK AIR ID string. --
  x == "ABD" is not TRUE
  
  `actual`:   <NA>
  `expected`: TRUE
  
  [ FAIL 1 | WARN 0 | SKIP 2 | PASS 11 ]
``

Different variables for ukair_get_coordinates() when inputs are fed in differently

Hi there,

Depending on how I feed the Site ID into the function ukair_get_coordinates() I get a data.frame of different width. Two methods are as follows:

Set up

library('rdefra')
stations_raw <- ukair_catalogue()

Method 1

stations <- data.frame()
for (i in stations_raw$UK.AIR.ID[!is.na(stations_raw$EMEP.Site.ID)]){
        stations <- rbind(stations, ukair_get_coordinates(i))
}

Method 2

stations_raw <- ukair_catalogue()
stations_all <- ukair_get_coordinates(stations_raw[!is.na(stations_raw$EMEP.Site.ID),])

Method 1 returns a data.frame with 5 variables, and Method 2 returns a data.frame with 16 variables (and Eastings and Northings are generally incomplete).

Thank you