ukhsa-internal/coronavirus-dashboard-api-python-sdk

Coronavirus (COVID-19) in the UK - API Service SDK for Python

Coronavirus (COVID-19) in the UK - API Service

Software Development Kit (SDK) for Python

This is a Python SDK for the COVID-19 API, as published by Public Health England on Coronavirus (COVID-19) in the UK.

Documentation

Comprehensive documentations and examples for this library is available on the documentations website.

SDK for other languages

Similar libraries are also available for JavaScript, R, .Net, and Elixir.

The API

The API supplies the latest data for the COVID-19 outbreak in the United Kingdom. The endpoint for the data provided using this SDK is:

https://api.coronavirus.data.gov.uk/v1/data

See the Developers Guide for additional information on the API and see a list of latest metrics.

Installation

Python 3.7+ is required to install and use this library.

To install, please run:

pip install uk-covid19

You may install the library for a specific version of Python as follows:

python -m pip install uk-covid19

You can simply replace python with a specific version for which you wish to install the library - e.g. python3 or python3.8.

---

Developed and maintained by Public Health England.

Copyright (c) 2020, Public Health England.

coronavirus-dashboard-api-python-sdk's Issues

latest_by="newCasesByPublishDate" not returning any data

I've had a script running for a while now, pulling down the latest data. However in the last couple of days the script has been hanging. I have tracked it down to the latest_by="newCasesByPublishDate" portion. If I remove this, it returns all the data, but I am no longer able to just pull in the latest data. Any idea what is going on?

The api call looks like

api = Cov19API(
        filters=filter, 
        structure=structure,
        latest_by="newCasesByPublishDate")

'Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'

Not able to get data from API. This code was working fine till yesterday. Today getting following error - "ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))"

from uk_covid19 import Cov19API
import pandas as pd

ltla_filter = ['areaType=ltla']
cases_and_deaths = {
                    "areaType":"areaType"
                    ,"areaName":"areaName"
                    ,"areaCode":"areaCode"
                    ,"specimenDate":"date"
                    ,"dailyLabConfirmedCases":"newCasesBySpecimenDate"
                    ,"totalLabConfirmedCases":"cumCasesBySpecimenDate"
                    }
api = Cov19API(filters=ltla_filter, structure=cases_and_deaths)
data = api.get_json()  # Returns a dictionary      

Additional undocumented filter comparisons

By playing with the API, I find that comparisons other than = are supported in filters. For example, date>=2020-07-01 and date<2020-07-15 . To obtain results for e.g. all nations you can specify areaType=nation and areaCode!=Q (Q here is arbitrary; I find that omitting any filter for areaName or areaCode results in an error).

Assuming these comparisons are intended to be supported (perhaps they are experimental?), it would be good if they were documented.

Scotland LTLA historical information is missing

I am not getting historical data for Scotland.
Following code shows what is expected:
England:

ltla_filter = [
    'areaType=ltla',
    'areaName=Coventry'
]
columns = {
    "areaType":"areaType"
,"areaName":"areaName"
,"areaCode":"areaCode"
,"date" : "date"
,"dailyLabConfirmedCases":"newCasesBySpecimenDate"
}
api = Cov19API(filters=ltla_filter, structure=columns)
data = api.get_json()  # Returns a dictionary
ltlasDf = pd.DataFrame(data['data'])
ltlasDf.head(5)

Expected Output:

areaType	areaName	areaCode	date	dailyLabConfirmedCases
0	ltla	Coventry	E08000026	2020-08-05	0
1	ltla	Coventry	E08000026	2020-08-04	4
2	ltla	Coventry	E08000026	2020-08-03	7
3	ltla	Coventry	E08000026	2020-08-02	4
4	ltla	Coventry	E08000026	2020-08-01	7

However same information is not present for Scotland areas. e.g.

ltla_filter = [
    'areaType=ltla',
    'areaName=Aberdeen City'
]
columns = {
    "areaType":"areaType"
,"areaName":"areaName"
,"areaCode":"areaCode"
,"date" : "date"
,"dailyLabConfirmedCases":"newCasesBySpecimenDate"
}
api = Cov19API(filters=ltla_filter, structure=columns)
data = api.get_json()  # Returns a dictionary
ltlasDf = pd.DataFrame(data['data'])
ltlasDf.head(5)

Returns no result.

If we change specimen date to publish date then we are getting historical dates but with 0 values.

ltla_filter = [
    'areaType=ltla',
    'areaName=Aberdeen City'
]
columns = {
    "areaType":"areaType"
,"areaName":"areaName"
,"areaCode":"areaCode"
,"date" : "date"
,"dailyLabConfirmedCases":"newCasesByPublishDate"
}
api = Cov19API(filters=ltla_filter, structure=columns)
data = api.get_json()  # Returns a dictionary
ltlasDf = pd.DataFrame(data['data'])
ltlasDf.head(5)

Output:

areaType	areaName	areaCode	date	dailyLabConfirmedCases
0	ltla	Aberdeen City	S12000033	2020-08-06	25
1	ltla	Aberdeen City	S12000033	2020-08-05	0
2	ltla	Aberdeen City	S12000033	2020-08-04	0
3	ltla	Aberdeen City	S12000033	2020-08-03	0
4	ltla	Aberdeen City	S12000033	2020-08-02	0

Historica data for all Scotland information should be present either for specimen date or publish date.

500 Internal Server Error occurs occasionally

Hi! I have been using the api for a few weeks and it works great for most of the time, but I'm occasionally having the FailedRequestError (500 - Internal Server Error), and it seems that it still works after I retried for multiple times (like 10 times, it's still fine in a jupyter notebook, but just won't be much pleasant when integrated into the project with multiple py files). Here is my code:

filters_uk = ['areaType=overview']

structure_total = {
"date": "date",
"newcases": "newCasesByPublishDate",
"cumcases": "cumCasesByPublishDate",
"newdeaths": "newDeaths28DaysByPublishDate",
"cumdeaths": "cumDeaths28DaysByPublishDate"
}

api_uk = Cov19API(filters_uk, structure_total)

It is a very basic data request and it works fine till here.

The problem only occurs when I try to get time or the data.

api_uk_timestamp = api_uk.last_update

and

df_uk = api_uk.get_dataframe()

And as I descriped, both of the operations will still work anyway if I retry many times, and the problem only occurs very occasionally(I only had twice in three weeks, last time it was sovled after I restart the kernel/laptop, but this time it didn't work that way), so I am not sure whether it is only something to do with myself or it can be improved, but I'll just report it as it suggested in the error message.

Many thanks!

Elixir SDK

Hiya,

I wasn't sure where to create this issue so have followed @jchannon's lead and created an issue here, apologize.

I have created an Elixir implementation of the SDK here, I'm happy to transfer ownership if you so wish.
I've tried to follow the same programming style as the JS and Python SDK's and have implemented the test's in the same way.

Here is a link to the repo if you wanted to check it out: https://github.com/CallumVass/coronavirus-dashboard-api-elixir-sdk

Thanks

Archive data through the API

Hello 👋

The API documentation says that data previously published in the dashboard (currently on the archive: https://coronavirus.data.gov.uk/archive ) is available for download through the API as well. But I haven't been able to figure out how to do structure the query to get this. Is there an example of what such a query looks like?

For example, I want to know the number of cases in Leicester for the date 2020-06-01 as reported on 2020-06-02, 2020-06-03, .....

Any help with this would be much appreciated, thank you!

Is this API binding obsolete?

I raised a question with the service support group as some of the responses to this API were becoming slow/incomplete and unpredictable. I was told that I ought to be using v2 of the API and directed to a UI url.

Should this language binding be moving to the v2 'API'?

[nb, in the original terminology of technical architectures, an API is bound to a specific programming language. An interface that's a url is really a protocol, not an API]

Structure Query

Hi,

I am trying to retrieve data from the API using the metrics renaming feature. The structure I am providing to the API is slightly nested. Is this not support as data in the response is missing.

Have I misunderstood something here?

Here is the code I am using containing the structure I am providing:

from uk_covid19 import Cov19API
import pprint
    structure = {
        "date": "date",
        "name": "areaName",
        "code": "areaCode",
        "stats": {
            "cases": {
                "Daily cases": "newCasesByPublishDate",
                "Cumulative cases": "cumCasesByPublishDate",
                "Male cases": "maleCases",
                "Female cases": "femaleCases",
                "Hospital cases": "hospitalCases",
                "COVID-19 occupied beds with mechanical ventilators": "covidOccupiedMVBeds"
            },
            "deaths": {
                "Daily deaths": "newDeaths28DaysByDeathDate",
                "Cumulative deaths": "cumDeaths28DaysByDeathDate"
            },
            "admissions": {
                "New admissions": "newAdmissions",
                "Cumulative admissions": "cumAdmissions"
            },
            "tests": {
                "New pillar one tests by publish date": {
                    "metadata": "Pillar 1 (NHS and, in England, PHE)",
                    "value": "newPillarOneTestsByPublishDate"
                },
                "Cumulative pillar one tests by publish date": {
                    "metadata": "Pillar 1 (NHS and, in England, PHE)",
                    "value": "cumPillarOneTestsByPublishDate"
                },
                "New pillar two tests by publish date": {
                    "metadata": "Pillar 2 (Commercial partners)",
                    "value": "newPillarTwoTestsByPublishDate"
                },
                "Cumulative pillar two tests by publish date": {
                    "metadata": "Pillar 2 (Commercial partners)",
                    "value": "cumPillarTwoTestsByPublishDate"
                },
                "New pillar three tests by publish date": {
                    "metadata": "Pillar 3 (Antibody)",
                    "value": "newPillarThreeTestsByPublishDate"
                },
                "Cumulative pillar three tests by publish date": {
                    "metadata": "Pillar 3 (Antibody)",
                    "value": "cumPillarThreeTestsByPublishDate"
                },
                "New pillar four tests by publish date": {
                    "metadata": "Pillar 4 (Surveillance)",
                    "value": "newPillarFourTestsByPublishDate"
                },
                "Cumulative pillar four tests by publish date": {
                    "metadata": "Pillar 4 (Surveillance)",
                    "value": "cumPillarFourTestsByPublishDate"
                }
            }
        }
    }

    # Instantiate API object
    api = Cov19API(
        filters=["areaType=nation"],
        structure=structure,
        latest_by="newCasesByPublishDate"
    )

    # Obtain JSON from the API
    response = api.get_json()
    pprint.pprint(response)
    return response

Below is the response I receive. admissions and cases and deaths all have no data returned. However, tests which is heavily nested does return data 🤔

{'data': [{'code': 'E92000001',
           'date': '2020-09-18',
           'name': 'England',
           'stats': {'admissions': {'Cumulative admissions': 'cumAdmissions',
                                    'New admissions': 'newAdmissions'},
                     'cases': {'COVID-19 occupied beds with mechanical ventilators': 'covidOccupiedMVBeds',
                               'Cumulative cases': 'cumCasesByPublishDate',
                               'Daily cases': 'newCasesByPublishDate',
                               'Female cases': 'femaleCases',
                               'Hospital cases': 'hospitalCases',
                               'Male cases': 'maleCases'},
                     'deaths': {'Cumulative deaths': 'cumDeaths28DaysByDeathDate',
                                'Daily deaths': 'newDeaths28DaysByDeathDate'},
                     'tests': {'Cumulative pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                            '4 '
                                                                                            '(Surveillance)',
                                                                                'value': None},
                               'Cumulative pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                           '1 '
                                                                                           '(NHS '
                                                                                           'and, '
                                                                                           'in '
                                                                                           'England, '
                                                                                           'PHE)',
                                                                               'value': 5826554},
                               'Cumulative pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                             '3 '
                                                                                             '(Antibody)',
                                                                                 'value': 1628603},
                               'Cumulative pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                           '2 '
                                                                                           '(Commercial '
                                                                                           'partners)',
                                                                               'value': 9580578},
                               'New pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                     '4 '
                                                                                     '(Surveillance)',
                                                                         'value': None},
                               'New pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                    '1 '
                                                                                    '(NHS '
                                                                                    'and, '
                                                                                    'in '
                                                                                    'England, '
                                                                                    'PHE)',
                                                                        'value': 64639},
                               'New pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                      '3 '
                                                                                      '(Antibody)',
                                                                          'value': 6647},
                               'New pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                    '2 '
                                                                                    '(Commercial '
                                                                                    'partners)',
                                                                        'value': 133978}}}},
          {'code': 'N92000002',
           'date': '2020-09-18',
           'name': 'Northern Ireland',
           'stats': {'admissions': {'Cumulative admissions': 'cumAdmissions',
                                    'New admissions': 'newAdmissions'},
                     'cases': {'COVID-19 occupied beds with mechanical ventilators': 'covidOccupiedMVBeds',
                               'Cumulative cases': 'cumCasesByPublishDate',
                               'Daily cases': 'newCasesByPublishDate',
                               'Female cases': 'femaleCases',
                               'Hospital cases': 'hospitalCases',
                               'Male cases': 'maleCases'},
                     'deaths': {'Cumulative deaths': 'cumDeaths28DaysByDeathDate',
                                'Daily deaths': 'newDeaths28DaysByDeathDate'},
                     'tests': {'Cumulative pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                            '4 '
                                                                                            '(Surveillance)',
                                                                                'value': None},
                               'Cumulative pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                           '1 '
                                                                                           '(NHS '
                                                                                           'and, '
                                                                                           'in '
                                                                                           'England, '
                                                                                           'PHE)',
                                                                               'value': 196035},
                               'Cumulative pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                             '3 '
                                                                                             '(Antibody)',
                                                                                 'value': None},
                               'Cumulative pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                           '2 '
                                                                                           '(Commercial '
                                                                                           'partners)',
                                                                               'value': 222131},
                               'New pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                     '4 '
                                                                                     '(Surveillance)',
                                                                         'value': None},
                               'New pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                    '1 '
                                                                                    '(NHS '
                                                                                    'and, '
                                                                                    'in '
                                                                                    'England, '
                                                                                    'PHE)',
                                                                        'value': 1897},
                               'New pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                      '3 '
                                                                                      '(Antibody)',
                                                                          'value': None},
                               'New pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                    '2 '
                                                                                    '(Commercial '
                                                                                    'partners)',
                                                                        'value': 4612}}}},
          {'code': 'S92000003',
           'date': '2020-09-18',
           'name': 'Scotland',
           'stats': {'admissions': {'Cumulative admissions': 'cumAdmissions',
                                    'New admissions': 'newAdmissions'},
                     'cases': {'COVID-19 occupied beds with mechanical ventilators': 'covidOccupiedMVBeds',
                               'Cumulative cases': 'cumCasesByPublishDate',
                               'Daily cases': 'newCasesByPublishDate',
                               'Female cases': 'femaleCases',
                               'Hospital cases': 'hospitalCases',
                               'Male cases': 'maleCases'},
                     'deaths': {'Cumulative deaths': 'cumDeaths28DaysByDeathDate',
                                'Daily deaths': 'newDeaths28DaysByDeathDate'},
                     'tests': {'Cumulative pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                            '4 '
                                                                                            '(Surveillance)',
                                                                                'value': None},
                               'Cumulative pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                           '1 '
                                                                                           '(NHS '
                                                                                           'and, '
                                                                                           'in '
                                                                                           'England, '
                                                                                           'PHE)',
                                                                               'value': 616532},
                               'Cumulative pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                             '3 '
                                                                                             '(Antibody)',
                                                                                 'value': None},
                               'Cumulative pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                           '2 '
                                                                                           '(Commercial '
                                                                                           'partners)',
                                                                               'value': 850063},
                               'New pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                     '4 '
                                                                                     '(Surveillance)',
                                                                         'value': None},
                               'New pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                    '1 '
                                                                                    '(NHS '
                                                                                    'and, '
                                                                                    'in '
                                                                                    'England, '
                                                                                    'PHE)',
                                                                        'value': 6015},
                               'New pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                      '3 '
                                                                                      '(Antibody)',
                                                                          'value': None},
                               'New pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                    '2 '
                                                                                    '(Commercial '
                                                                                    'partners)',
                                                                        'value': 11274}}}},
          {'code': 'W92000004',
           'date': '2020-09-18',
           'name': 'Wales',
           'stats': {'admissions': {'Cumulative admissions': 'cumAdmissions',
                                    'New admissions': 'newAdmissions'},
                     'cases': {'COVID-19 occupied beds with mechanical ventilators': 'covidOccupiedMVBeds',
                               'Cumulative cases': 'cumCasesByPublishDate',
                               'Daily cases': 'newCasesByPublishDate',
                               'Female cases': 'femaleCases',
                               'Hospital cases': 'hospitalCases',
                               'Male cases': 'maleCases'},
                     'deaths': {'Cumulative deaths': 'cumDeaths28DaysByDeathDate',
                                'Daily deaths': 'newDeaths28DaysByDeathDate'},
                     'tests': {'Cumulative pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                            '4 '
                                                                                            '(Surveillance)',
                                                                                'value': None},
                               'Cumulative pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                           '1 '
                                                                                           '(NHS '
                                                                                           'and, '
                                                                                           'in '
                                                                                           'England, '
                                                                                           'PHE)',
                                                                               'value': 377348},
                               'Cumulative pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                             '3 '
                                                                                             '(Antibody)',
                                                                                 'value': None},
                               'Cumulative pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                           '2 '
                                                                                           '(Commercial '
                                                                                           'partners)',
                                                                               'value': 326229},
                               'New pillar four tests by publish date': {'metadata': 'Pillar '
                                                                                     '4 '
                                                                                     '(Surveillance)',
                                                                         'value': None},
                               'New pillar one tests by publish date': {'metadata': 'Pillar '
                                                                                    '1 '
                                                                                    '(NHS '
                                                                                    'and, '
                                                                                    'in '
                                                                                    'England, '
                                                                                    'PHE)',
                                                                        'value': 3396},
                               'New pillar three tests by publish date': {'metadata': 'Pillar '
                                                                                      '3 '
                                                                                      '(Antibody)',
                                                                          'value': None},
                               'New pillar two tests by publish date': {'metadata': 'Pillar '
                                                                                    '2 '
                                                                                    '(Commercial '
                                                                                    'partners)',
                                                                        'value': 7388}}}}],
 'lastUpdate': '2020-09-18T14:04:49.000000Z',
 'length': 4,
 'totalPages': 1}

Any help is much appreciated.

get_dataframe() mishandles index and date type

the dataframe returned by get_dataframe () is quite odd from a pandas point of view and doesn't appear to agree with the api documentation.

Shouldn't the 'date' column be of type pandas.Timestamp, rather than ?
Should time run forwards?
Should the 'date' column be the index of the returned dataframe?

unable to return newPeopleVaccinatedFirstDoseByPublishDate or newPeopleVaccinatedFirstDoseByPublishDate

If either of these metrics is included in a request, the string name of the metric is returned instead of the numeric data. The equivalent for 2nd vaccine dose works correctly

For example:
structure = {
"date": "date",
"name": "areaName",
"code": "areaCode",
"Dose1": "cumPeopleVaccinatedFirstDoseByPublishDate",
"Dose2": "cumPeopleVaccinatedSecondDoseByPublishDate",
}
returned data is
date,name,code,Dose1,Dose2
2021-01-13,United Kingdom,K02000001,cumPeopleVaccinatedFirstDoseByPublishDate,
2021-01-12,United Kingdom,K02000001,cumPeopleVaccinatedFirstDoseByPublishDate,428232
2021-01-11,United Kingdom,K02000001,cumPeopleVaccinatedFirstDoseByPublishDate,412167

README.rst link to "documentations website" broken, update to ukhsa-internal URL?

In README.rst the link to "documentations website" (under the Documentation heading) looks like it may need updating to https://ukhsa-internal.github.io/coronavirus-dashboard-api-python-sdk/

options call fails

The options call does not seem to work

Python 3.7.3 (default, Dec 20 2019, 18:57:59)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from uk_covid19 import Cov19API
>>> from pprint import pprint
>>> options = Cov19API.options()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gigo/pheapi/uk_covid19/api_interface.py", line 272, in options
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Resource Not Found for url: https://api.coronavirus.data.gov.uk/v1/data

C# SDK

Hi,

Apologies that this is not python related but I've written a C# SDK if you are interested in taking ownership of it? It has most of the functionality and a sample app to demonstrate it's use. I saw you had R, Python & Javascript and wondered if you'd want a C# one too?

Here's the repo if you'd like it - https://github.com/jchannon/coronavirus-dashboard-api-net-sdk

Thanks!

Values for filters

What are the values for filters? Been looking to use this with areaname filter for Lower and Upper tier LA. Some LTLA values from the MSOA list are not supported. So would be good to get a clear list of the values for these filter lists. @xenatisch

areaType

Can an example or method be provided for obtaining a list of supported "areaType"?

Historical data using the API

Thanks very much for publishing this API! I couldn't work out from the docs what the best way to get all historical records for a given resource is. Could you provide some guidance?

conjunctive conditions and multiple filters

hi, how do i get multiple areas in one response eg
https://api.coronavirus.data.gov.uk/v1/data?filters=areaName=barnsley;areaName=rotherham;areaName=sheffield&structure={"date":"date","areaName":"areaName","newCases":"newCasesByPublishDate"}

We wanting to make ONE api call, that returns new cases for multiple areas

500 Error on API request

Hello,

I have been using the API for daily downloads of the updated case counts for utla and ltla areas. This stopped working yesterday with a 500 error returned. If this is a timeout issue could you please allow longer queries? this was working last week and I have to produce a daily report. Many thanks, Carlo.

FailedRequestError: Request failed .... 500 - Internal Server Error Response .......... No response URL ............... https://api.coronavirus.data.gov.uk/v1/data?filters=areaType%3Dutla&structure=%7B%22areaType%22%3A%22areaType%22%2C%22areaName%22%3A%22areaName%22%2C%22areaCode%22%3A%22areaCode%22%2C%22specimenDate%22%3A%22date%22%2C%22dailyLabConfirmedCases%22%3A%22newCasesBySpecimenDate%22%2C%22totalLabConfirmedCases%22%3A%22cumCasesBySpecimenDate%22%7D&format=json&page=30 Decoded URL ....... https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=utla&structure={"areaType":"areaType","areaName":"areaName","areaCode":"areaCode","specimenDate":"date","dailyLabConfirmedCases":"newCasesBySpecimenDate","totalLabConfirmedCases":"cumCasesBySpecimenDate"}&format=json&page=30 Parameters: { 'filters': 'areaType=utla', 'format': 'json', 'page': 30, 'structure': '{"areaType":"areaType","areaName":"areaName","areaCode":"areaCode","specimenDate":"date","dailyLabConfirmedCases":"newCasesBySpecimenDate","totalLabConfirmedCases":"cumCasesBySpecimenDate"}'}

Only single record returned for Scottish areas

Probably an issue with the backend rather than the python API; not sure where best to report it.

Queries for Scottish areas seem to return only a single record, depending on what is requested in the structure.

from uk_covid19 import Cov19API

filters = ['areaCode=S12000049','areaType=utla']
structure = {
    'date':'date',
    'newCasesBySpecimenDate':'newCasesBySpecimenDate',
}
res = Cov19API(filters,structure).get_json()
print(res['length'], res['data'][-1])
# 1 {'date': '2020-08-12', 'newCasesBySpecimenDate': 6}

# Add another field not of interest, apparently always 0
structure['newCasesByPublishDate'] = 'newCasesByPublishDate'
res = Cov19API(filters,structure).get_json()
print(res['length'], res['data'][-1])
# 221 {'date': '2020-01-05', 'newCasesBySpecimenDate': None, 'newCasesByPublishDate': 0}

I'd expect to get the same number of records regardless of structure metrics. Something to do with None versus 0 perhaps?

Retrieve historical data for ltla

Although in the example it is demonstrated that data can be retrieved for historical dates as well

england_only = [
    'areaType=nation',
    'areaName=England'
]

However when we change filter of areaType to ltla, it only returns data for 4th august. Even adding a filter for a date does not provide output from the previous date.

ltla_filter = [
    'areaType=ltla',
    'areaName=Warwick'
]
cases_and_deaths = {
    "date":"date",
    "areaName":"areaName",
    "areaCode":"areaCode",
    "newCasesByPublishDate":"newCasesByPublishDate",
    "cumCasesByPublishDate":"cumCasesByPublishDate",
    "newDeathsByDeathDate":"newDeathsByDeathDate",
    "cumDeathsByDeathDate":"cumDeathsByDeathDate"
}
api = Cov19API(filters=ltla_filter, structure=cases_and_deaths)
data = api.get_json()  # Returns a dictionary
print(data)

{'data': [{'date': '2020-08-04', 'areaName': 'Warwick', 'areaCode': 'E07000222', 'newCasesByPublishDate': None, 'cumCasesByPublishDate': 445, 'newDeathsByDeathDate': None, 'cumDeathsByDeathDate': 78}], 'lastUpdate': '2020-08-04T21:35:21.000000Z', 'length': 1, 'totalPages': 1}

Historical demographic data through the API

Hi,

We're (https://github.com/GoogleCloudPlatform/covid-19-open-data) are interested in the time series of Covid positive cases aggregated by age and sex. Using the following structure for the API, I'm only able to get the most recent day's data, even though no latestBy argument is provided.

'structure': '{"date":"date","areaName":"areaName","maleCases":"maleCases","femaleCases":"femaleCases"}'}

Additionally, the API errors out (with error 500) when using any filter on areaType, but works when using areaName as a filter e.g. areaName=England is a fine filter, but areaType=nations does not work.

I note that we don't currently need the intersectional data of age+sex which is what the above provides, having the demographics data separately is also fine.

List of valid metrics for structure contains duplicate key missing out `cumCasesByPublishDateRate`

I know this is the SDK but there is nowhere to comment on the gov.uk docs.

Link to the docs: https://coronavirus.data.gov.uk/details/developers-guide#structure-metrics

Docs show:
cumCasesBySpecimenDateRate Rate of cumulative cases by publish date per 100k resident population
cumCasesBySpecimenDateRate Rate of cumulative cases by specimen date per 100k resident population

Should be:
cumCasesByPublishDateRate Rate of cumulative cases by publish date per 100k resident population
cumCasesBySpecimenDateRate Rate of cumulative cases by specimen date per 100k resident population

API endpoint redirected to normal page ...

There's a redirect in place meaning that the API does not work ...
... If the API is down, can't the service simply fail the request?

curl -si 'https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=nation;areaName=england&structure=%7B%22name%22:%22areaName%22%7D'
HTTP/1.1 307 Temporary Redirect
Content-Length: 0
Location: https://api.coronavirus-staging.data.gov.uk/v1/data?filters=areaType=nation;areaName=england&structure=%7B%22name%22:%22areaName%22%7D&format=json&page=1
Request-Context: appId=cid-v1:10ef5909-f7e8-4267-86f0-f575a2bdc89e
Date: Thu, 10 Sep 2020 09:27:21 GMT

Add support for API version 2

I'm looking into how we can move this SDK along to version 2, and wanted to suggest one approach, and discuss suitability or other options.

Firstly, I was expecting to retain the Cov19API class and just expand it to support both v1 and v2 APIs. The v1 API timestamp endpoint is used within this class, and this does not appear to be available on the v2 API so that would need to be retained even when obtaining data from v2.

• An alternative option would be to copy the Cov19API and make a Cov19APIv2 class in an entirely new module which could avoid making a 'Frankenstein' class trying to juggle needs of two distinct API versions. Any duplication could then simply be refactored into a base class from which both inherit.

Specifically, to retain backwards compatibility, I was expecting to initialise the Cov19API with a parameter version defaulting to "1", and storing a class attribute _supported_versions: list[str] = ["1", "2"] against which this is checked on initialisation.

Looking at the features of the v1 API and those in the new one, filters has been removed, and only partly reimplemented in the v2 API. One solution here would be to make the filters parameter optional, and then if provided enforce version == "2" else enforce v1.

Personally I suspect that separating the APIs out into separate classes may indeed be the cleaner approach for library design, but as a user you probably just want a single 'entry point' and not to pick between multiple classes. This however leaves the messy situation of 'retaining backward compatibility' meaning that said single class would default to v1, hence my idea to automatically use v2 if filters were not specified (the only way I can think to resolve both these constraints).

• typing.overload may be a nice way to cleanly specify distinct signatures for v1 and v2 usage of the class __init__ method

Other minor improvements I've already implemented or would like to:

• Use from __future__ import annotations (allows parameterised types like dict rather than Dict; allows | instead of Union)
• Use isort to ensure proper import structure; remove code comments indicating import convention
• Lint with black for consistency among multiple developers (potentially could also use pre-commit hooks; would let you decide if desirable)

Further to this, my initial motivation for contributing was to amass an exhaustive list (which in the context of an SDK could simply be one or more Enum classes) of the metrics etc. which are available, and also to indicate which are incompatible (as is mentioned in parts of the API documentation). As mentioned in this blog post:

"not all metrics on the page are available for all areas every day. For instance, vaccination data in England are published at MSOA level, while in Scotland, they are published at local authority level."

An additional ‘nice-to-have’ (both for users to avoid hitting rate limits but also UKHSA to avoid excessive query load) would be to have a default resource management feature such that when the same resource is re-requested then the locally proxied file is used instead. This could use for example a sqlite database stored as a temporary file in /var/tmp or /var/cache (i.e. directories that persist on reboot), or just use the tempfile.mkdtemp defaults for non-Unix OS’s (Windows)

API performance issues

Today I have been experiencing poor performance from the API.

The API has become slow and is returning status 500 internal server error.

Here is the error response and activityId:

Response .......... { "statusCode": 500, "message": "Internal server error", "activityId": "3ef0d497-18c5-47ff-9e1a-3f91cc6c0c6d" }

Full error response:

Request failed .... 500 - Internal Server Error
Response .......... { "statusCode": 500, "message": "Internal server error", "activityId": "3ef0d497-18c5-47ff-9e1a-3f91cc6c0c6d" }
URL ............... https://api.coronavirus.data.gov.uk/v1/data?filters=areaType%3Dnation&structure=%7B%22date%22%3A%22date%22%2C%22name%22%3A%22areaName%22%2C%22code%22%3A%22areaCode%22%2C%22stats%22%3A%7B%22cases%22%3A%7B%22New+cases%22%3A%7B%22value%22%3A%22newCasesByPublishDate%22%7D%2C%22Cumulative+cases%22%3A%7B%22value%22%3A%22cumCasesByPublishDate%22%7D%2C%22Hospital+cases%22%3A%7B%22value%22%3A%22hospitalCases%22%7D%2C%22COVID-19+occupied+beds+with+mechanical+ventilators%22%3A%7B%22value%22%3A%22covidOccupiedMVBeds%22%7D%7D%2C%22deaths%22%3A%7B%22Daily+deaths%22%3A%7B%22value%22%3A%22newDeaths28DaysByDeathDate%22%7D%2C%22Cumulative+deaths%22%3A%7B%22value%22%3A%22cumDeaths28DaysByDeathDate%22%7D%7D%2C%22admissions%22%3A%7B%22New+admissions%22%3A%7B%22value%22%3A%22newAdmissions%22%7D%2C%22Cumulative+admissions%22%3A%7B%22value%22%3A%22cumAdmissions%22%7D%7D%2C%22tests%22%3A%7B%22New+pillar+one+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+1+%28NHS+and%2C+in+England%2C+PHE%29%22%2C%22value%22%3A%22newPillarOneTestsByPublishDate%22%7D%2C%22Cumulative+pillar+one+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+1+%28NHS+and%2C+in+England%2C+PHE%29%22%2C%22value%22%3A%22cumPillarOneTestsByPublishDate%22%7D%2C%22New+pillar+two+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+2+%28Commercial+partners%29%22%2C%22value%22%3A%22newPillarTwoTestsByPublishDate%22%7D%2C%22Cumulative+pillar+two+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+2+%28Commercial+partners%29%22%2C%22value%22%3A%22cumPillarTwoTestsByPublishDate%22%7D%2C%22New+pillar+three+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+3+%28Antibody%29%22%2C%22value%22%3A%22newPillarThreeTestsByPublishDate%22%7D%2C%22Cumulative+pillar+three+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+3+%28Antibody%29%22%2C%22value%22%3A%22cumPillarThreeTestsByPublishDate%22%7D%2C%22New+pillar+four+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+4+%28Surveillance%29%22%2C%22value%22%3A%22newPillarFourTestsByPublishDate%22%7D%2C%22Cumulative+pillar+four+tests+by+publish+date%22%3A%7B%22metadata%22%3A%22Pillar+4+%28Surveillance%29%22%2C%22value%22%3A%22cumPillarFourTestsByPublishDate%22%7D%7D%7D%7D&latestBy=newCasesByPublishDate&format=json&page=1
Decoded URL ....... https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=nation&structure={"date":"date","name":"areaName","code":"areaCode","stats":{"cases":{"New+cases":{"value":"newCasesByPublishDate"},"Cumulative+cases":{"value":"cumCasesByPublishDate"},"Hospital+cases":{"value":"hospitalCases"},"COVID-19+occupied+beds+with+mechanical+ventilators":{"value":"covidOccupiedMVBeds"}},"deaths":{"Daily+deaths":{"value":"newDeaths28DaysByDeathDate"},"Cumulative+deaths":{"value":"cumDeaths28DaysByDeathDate"}},"admissions":{"New+admissions":{"value":"newAdmissions"},"Cumulative+admissions":{"value":"cumAdmissions"}},"tests":{"New+pillar+one+tests+by+publish+date":{"metadata":"Pillar+1+(NHS+and,+in+England,+PHE)","value":"newPillarOneTestsByPublishDate"},"Cumulative+pillar+one+tests+by+publish+date":{"metadata":"Pillar+1+(NHS+and,+in+England,+PHE)","value":"cumPillarOneTestsByPublishDate"},"New+pillar+two+tests+by+publish+date":{"metadata":"Pillar+2+(Commercial+partners)","value":"newPillarTwoTestsByPublishDate"},"Cumulative+pillar+two+tests+by+publish+date":{"metadata":"Pillar+2+(Commercial+partners)","value":"cumPillarTwoTestsByPublishDate"},"New+pillar+three+tests+by+publish+date":{"metadata":"Pillar+3+(Antibody)","value":"newPillarThreeTestsByPublishDate"},"Cumulative+pillar+three+tests+by+publish+date":{"metadata":"Pillar+3+(Antibody)","value":"cumPillarThreeTestsByPublishDate"},"New+pillar+four+tests+by+publish+date":{"metadata":"Pillar+4+(Surveillance)","value":"newPillarFourTestsByPublishDate"},"Cumulative+pillar+four+tests+by+publish+date":{"metadata":"Pillar+4+(Surveillance)","value":"cumPillarFourTestsByPublishDate"}}}}&latestBy=newCasesByPublishDate&format=json&page=1
Parameters:
{ 'filters': 'areaType=nation',
  'format': 'json',
  'latestBy': 'newCasesByPublishDate',
  'page': 1,
  'structure': '{"date":"date","name":"areaName","code":"areaCode","stats":{"cases":{"New '
               'cases":{"value":"newCasesByPublishDate"},"Cumulative '
               'cases":{"value":"cumCasesByPublishDate"},"Hospital '
               'cases":{"value":"hospitalCases"},"COVID-19 occupied beds with '
               'mechanical '
               'ventilators":{"value":"covidOccupiedMVBeds"}},"deaths":{"Daily '
               'deaths":{"value":"newDeaths28DaysByDeathDate"},"Cumulative '
               'deaths":{"value":"cumDeaths28DaysByDeathDate"}},"admissions":{"New '
               'admissions":{"value":"newAdmissions"},"Cumulative '
               'admissions":{"value":"cumAdmissions"}},"tests":{"New pillar '
               'one tests by publish date":{"metadata":"Pillar 1 (NHS and, in '
               'England, '
               'PHE)","value":"newPillarOneTestsByPublishDate"},"Cumulative '
               'pillar one tests by publish date":{"metadata":"Pillar 1 (NHS '
               'and, in England, '
               'PHE)","value":"cumPillarOneTestsByPublishDate"},"New pillar '
               'two tests by publish date":{"metadata":"Pillar 2 (Commercial '
               'partners)","value":"newPillarTwoTestsByPublishDate"},"Cumulative '
               'pillar two tests by publish date":{"metadata":"Pillar 2 '
               '(Commercial '
               'partners)","value":"cumPillarTwoTestsByPublishDate"},"New '
               'pillar three tests by publish date":{"metadata":"Pillar 3 '
               '(Antibody)","value":"newPillarThreeTestsByPublishDate"},"Cumulative '
               'pillar three tests by publish date":{"metadata":"Pillar 3 '
               '(Antibody)","value":"cumPillarThreeTestsByPublishDate"},"New '
               'pillar four tests by publish date":{"metadata":"Pillar 4 '
               '(Surveillance)","value":"newPillarFourTestsByPublishDate"},"Cumulative '
               'pillar four tests by publish date":{"metadata":"Pillar 4 '
               '(Surveillance)","value":"cumPillarFourTestsByPublishDate"}}}}'}

Build using PyPi source tarball fails due to missing requirements.txt

The 1.2.0 source tarball on PyPi appears to be incomplete, so that using it to build from source fails like so:

$ python3 setup.py build
Traceback (most recent call last):
File "setup.py", line 103, in
install_requires=get_requirements(),
File "setup.py", line 56, in get_requirements
with open(join(package_dir, 'requirements.txt')) as requirements:
FileNotFoundError: [Errno 2] No such file or directory: '/export/home/ptribble/ud/uk_covid19-1.2.0/requirements.txt'

Testing data by LTLA

The number of tests (capacity, processed) seems to only be available at the national level. Is there a plan to start releasing test numbers by LTLA?

Structure Change for Deaths

Has the structure changed for api calls for information relating to deaths? I keep getting 404 error codes using this structure:

Request failed .... 404 - Not Found
Response .......... {"response":"Invalid parameter 'newDeathsByDeathDate' in the requested JSON structure. Did you mean 'newDeaths28DaysByPublishDate'?","status_code":404,"status":"Not Found"}

Tried checking the documentation and there is no mention of this field.

Changing the field to newDeaths28DaysByPublishDate, stopped the python error, but no data was returned.

Have I missed a change announcement somewhere?

Thank you.

Internal server error

I am running the following code and it results in 500 error. It may be because high volume but at least the document specifies what should be maximum threshold? Due to this, my app stopped running, which gives insight about the local spread and may other statistics.

    ltla_filter = ['areaType=ltla']
    cases_and_deaths = {
                        "areaType":"areaType"
                        ,"areaName":"areaName"
                        ,"areaCode":"areaCode"
                        ,"specimenDate":"date"
                        ,"dailyLabConfirmedCases":"newCasesBySpecimenDate"
                        ,"totalLabConfirmedCases":"cumCasesBySpecimenDate"
                        }
    api = Cov19API(filters=ltla_filter, structure=cases_and_deaths)
    data = api.get_json()  # Returns a dictionary   

Error : "statusCode": 500, "message": "Internal server error", "activityId": "a7f790fe-e834-46d0-b8ed-0632c4b08274"

Typing/Documentation of cumDeaths28DaysByDeathDate

The metric 'cumDeaths28DaysByDeathDate' reads like it is a monotonically increasing value. However, the last value can be NaN, which arguably ought not to be possible.

If it is intended to return NaNs in the dataset, then the documentation ought, I think, to point this out as pandas has some default handling of NaNs, which will create confusion (typically they are treated as 0 in some aggregations, thus making the data element non-monotonic).

Otherwise, if 'cumDeaths28DaysByDeathDate' is a calculated value (it is the sum of 'newDeaths28DaysByDeathDate'), then the returned value ought to be the same as the previous day's value, rather than NaN.