dynamicgenetics/covid-19-community-response

Mapping community support in Wales during the COVID-19 pandemic

COVID-19 Community Response Map

This is a project to map community support in Wales throughout the COVID-19 pandemic.

We are using a range of open and government data sources to understand the population distributions that may be signigicant when planning community support, and displaying this alongside known existing sources of community support.

The project is a collaboration between the Dynamic Genetics group and Public Health Wales.

Interactive Map

To see and use the map generated by project please visit our COVID-19 Community Response Map.

Data Documentation

You can view the documentation for this project, and the data sources, on the Open Science Framework.

Technical Documentation

Documentation of the code used to create the map is available in the docs/ folder of this repository.

covid-19-community-response's Issues

Group Hover-Over URLS extend beyond display box

Either URLS wrap at a certain number of characters, or the box changes width responsively.

implement key per layer

Changelog:

• implement a key for each layer mapping colors to values in that layer

Selenium webdriver requires Safari

Needs to be updated to use headless Chrome so it will run on the server

borgLayers file needs manual reformat before use

currently the borgLayers file needs reformat before use (i.e., removing all "* and *" strings). Valerio suggested we use a dataclass to output layers from the borg in the format we want. This stems from the issue that we are using objective javascript objects:

Desired format (obj JS):
{ key : value, "quotedKey" : value }

Current output format (JSON):
{ "key" : value, "quotedKey" : value }

Adding new data to the map

Summary

We are hoping to add new data to the map.
This issue will be a place to decide what data we will be adding, or post links to useful sources.

Ideas

Here are some of the themes that we said we would consider going forwards for ideas!

Works in progress

• #105

• Twitter data

• Update documentation

• Update Welsh site

Variable definition is not robust enough in case one column in Master dataset is missing

If MasterDataset is missing the column mapped to a variable, the code will break as each variable accesses the corresponding data series (i.e. pd.Series) directly from master_dataset (which is a DataFrame).

Alternatively, a get method should be used, so that if the series is not there, the Variable won't be included in the generated data.json file.

(See backend.datasets.generate_json.py)

Links to Welsh/English version in header bar

Add reciprocal links to the Welsh and English sites in each language website

Metadata Update

Hello,
Your dataset was added to CoronaWhy (https://www.coronawhy.org/) Data Lake on Dataverse as a piece of common COVID-19 data frame http://datasets.coronawhy.org/dataset.xhtml?persistentId=doi:10.5072/FK2/TWFNK9
Would you be willing to help with the maintenance of your dataset in Dataverse, e.g. adding the relevant metadata and keeping the dataset up-to-date? That will help to make the dataset findable and accessible for the medical science community.

API Keys on server

Make somewhere to hold key files on server so they aren't lost when pulling new versions of the repository.

Vaccination data

I've looked into it now and the vaccination data is pulled from the same location as the COVID-19 case rate data is.
This should mean we can reuse the same functions for collecting the dataset and will just need to write new ones to process the spreadsheet.

I think we will want :
http://www2.nphs.wales.nhs.uk:8080/CommunitySurveillanceDocs.nsf/PublicPage?OpenPage&AutoFramed
title = "COVID19 vaccination downloadable data"
sheet = "HealthBoard_LocalAuthority"
filters: Risk group = "Wales residents aged 18 years and older"

There are columns for first and second dose so we might want to provide layers for both?

Data is not all scaled in the same direction

Data (e.g. COVID Vulnerable (comborbidity %)) is not all scaled in the same direction, or to the same scale.
This needs to be addressed for the interpretability of the map.

add on-hover to dots

Changelog:

• add on-hover data display for point layers (e.g., showing groups for individual points on hover)

Name of IMD variable not reflecting data level (LA/LSOA) selected

Possible fixes:

• if having LSOA data means we default to the LSOA data in all analysis then we can easily switch to prioritising the LSOA labels in the json meta data.
• if not, we'll need a frontend fix related to responding to the variable.
• alternatively, we just name both levels the same thing but make the name more general (e.g. 'Relative Deprivation'). This may be a better solution as this problem is quite unique to this variable.

Documentation structure and updating

Would it make sense to integrate our data documentation into the \docs\ folder of this repo so that all of the documentation would be available in one place as the docs for the code? This might look like having a code and data documentation sections on the website (present site is here) Some benefits of this might be...

• We could host meta-data about our datasets in the repository itself.
• Having datasets and documentation together will probably make it easier to keep docs up to date (OSF is already a bit out of date)
• We could probably integrate some kind of variables that keep track of when each dataset was last updated (especially for live datasets), and this would automatically update this information in the documentation too.

We made the Sphynx site quite last minute so I think there's a lot we could do to make it look nicer and fit better with the theme of the main site. We could then add a documentation option to the navbar?

Users should not be able to mix layers from different sections

Being able to mix layers from different sections allows users to mix colour palettes. To maintain data interpretability this should be discouraged.

To avoid this we need to implement functionality that prevents users from mixing layers from different sections, but allows them to 'layer' within a single section.

language by LSOA data does not work

the language data in demos_LSOA does not get put onto the map. It appears to be properly compiled from csv into geojson but the generateLayers does not produce a layer for it.

Put technical documentation on readthedocs.io?

After couple of fixes to the style, I would absolutely agree to put the docs on readthedocs.

As for the launch, I would just link the docs in the landing page with static link.

Therefore, I am converting this note into an issue pro memoria futura
(my tuppence tip-of-the-day in latin for you :D)

implament more data information

Changelog:

• implement more data information per layer (e.g., vulnerability definition from Jiao)

consider layer mixing

Changelog:

• human computer interaction considerations
• If we keep combining layers, add radio buttons / other way to prevent people mixing layers
• dynamically change opacity as a function of number of layers visible

higher resolution data

Changelog:

• get LSOA level data

implement category ordering system

Changelog:

• implement category ordering system in map.js
• implement category coloring system in borg.py
• Move community to top of list and bias to bottom

Data scale and direction is confusing

Data (e.g. COVID Vulnerable (comborbidity %)) is not all scaled in the same direction, or to the same scale.
This needs to be addressed for the interpretability of the map.

so:

1. decide direction and scale of layers
2. scale layers
3. set reverseColors property for data we want to reverse in colors

Nina's issue on this subject was slightly incorrect since the direction of colors (i.e., more means less or more means more of the measure) IS cureently supported (dataSources.py)

Sidebar not mobile friendly

Currently the sidebar of the map will take over the display when opened on mobile, so it is not possible to see the map.

We need a mobile implementation of the sidebar that allows it be opened, layers adjusted, and then closed again, so that mobile users can interact with the map.

Dataset import lag

Importing static dataset module takes a lot of time.

The issue is caused by global dataset constants to be defined in the module actually reading from files.

This issue could be easily fixed by replacing global constants with loading_functions.

Data labels on hover don't reflect data standardisation

E.G. On hover:

Support related tweets (PP): gives standardised number, which is not actually the number of support related tweets per person.

better on hover data readout

Changelog:

• add units to on hover data
• consider renaming layers to shorter names on on:hover readout
• build dictionary for these units & short names
• revise number formatting (e.g., so there aren't 0.000 values)

HTML tags for improved search engine optimisation

As already suggested by Chris, first thing to try is to include additional keywords in metadata tag in <head>

Update Mutual Aid data to use API

The COVID-19 mutual aid has an API endpoint that could be used. This is likely to be more stable that attempting to pull directly from Google Sheets.

https://github.com/Covid-Mutual-Aid/mutual-aid-wiki

Mongoexport needs scheduler.

Al will do this.

Mapbox Logo positioning

Request for mapbox logo to be placed separately from academic funders/partners.

Alisha wants group information pop-up for each dot

Quote:
"just a thought today.... can the blobs (!) of support actually show the organisations details? will be useful from a user perspective to then be getting in touch."

Remove Welsh language statement

From both Welsh and English sites

Project structure - potential for git submodules?

Making an issue for this discussion that we had a few month ago to decide whether we want to do this going forwards.
The idea was that we could have two repositories, one for the website and one for the data backend.
Making them separate might make it easier to:
a) update the map without having to pull the entire repo
b) treating the data behind the map and the frontend as separate outputs (e.g. someone might want to reuse the frontend, and would just need to edit it and then provide a data.json file that is formatted as expected).
(Tagging all of us in this one as something to potentially discuss as a group)

Some questions about this though.

• Do we want the dataset we create to have its own repo , or do we want the datasets and code together as they are now in the /backend/ folder?
• Would this be much work? i.e. is it worth doing vs benefit we'd get from it?

Sidebar: Make 'Risk of Bias' the last subheading & rename

'Risk of Bias' is currently the first subheading on the map. We would like it to appear last.

In the process, it should also be renamed from 'Risk of Bias' to 'Factors Affecting Data Quality'

Documentation structure does not reflect map layers

The documentation of data sources, and pages with explanations of sources, should match the layers on the map. This needs to be changed to make sure the map documentation is easy to navigate for those not familiar with the data.

fix scrolling issue

Changelog:

• scrolling moves the whole page (source: Nina)
• sidebar size overflows page (source: Nina)

move bias to new map

Changelog:

• consider whether bias should exist on a different map or could be a different map view on the same page / layer
• consider what risk of bias means (e.g., rename to factors affecting data quality)
• implement new bias map mode/view/page

Inconsistent geography file use

Scrapers files and the main data pipeline currently use different versions of the LA and LSOA boundaries files (in static/geoboundaries). Need to implement 'single point of failure' for loading and using these files in scripts, and to ensure all scripts are using the same version. This will also need to be integrated with the frontend.

revise language

Changelog:

• consider our audience when naming layers and ensure they make sense and are accessible