uoa-eresearch/edna

Heatmap system to geographically display eDNA (Environmental DNA) data from New Zealand

Biological Heritage National Science Challenge eDNA Virtual Hub Project

Goal of this pilot project is to produce a working example of a visualization system used to geographically display eDNA data from New Zealand. The display is, by default, a scalable grid of NZ, with samples falling into bins based on their latitude/longitude. The bins display as a heat map based on abundance. This will toggle between a sample view (where each sample contributes a count of one to the heat map), and a species view (where the heat map is made by summing abundances of all species). These heat map visualizations can be sub-selected by search functions of the metadata. Ultimately, it should be possible to sub-select any combination of metadata, but for the pilot this can just be some simple examples (e.g. elevation, basic environment type). This may also include some broad taxonomic divisions selectable as clickboxes (e.g. bacteria/archaea/animals/plants/fungi/protists). The species view should have a toggle between normalizing the heat-map to the number of samples from that grid bin, or not.

Clicking on a grid point (or hovering) brings up a list of all samples present in that grid point (or that fit the search criteria). In addition, perhaps as an active window alongside the map view, summary metrics of samples, either from clicking a grid point or sub-selecting the data, compared to all the data will be displayed. This is using the metric displays being developed by Andrew Dopheide (Biological Heritage National Science Challenge/Landcare Research).

For the pilot this is will be done from datasheets supplied, although ultimately the underlying data will come from a database being developed by us.

edna's Issues

add control to select samples from categorical amplicon types.

• Specified as ideally another leaflet (info) control beside the grid layer options.
• Modeled as a set of radio buttons but could also probably be implemented as a drop-down.

replace current contextual filters with the new contextual fields.

add new metafields to the contextual table. (Biome, Material etc)

eventually when gis data completion is working we can remove the old metafields.

add landuse mode to plot

Add a landuse coloured mode to the plot modes.

Samples titled ST* aren't being coloured correctly for elevation.

integrate andrew's graphs

Select2 focus switching to otu filter when contextual filter selected.

• When the contextual selection menu is pressed it sometimes switches the input focus to otu select2 element. Doesn't happen when using tab to switch focus from otu to contextual elements.

Add legend to plot

Add a legend to the plot

performance

Not needed yet but if performance gets too slow:

• compress the abundance response to send one set of sample context columns to remove the need for headers on every site included.

• even further: send colours of the boxes for the grid cells only + the coordinates - load everything as it's needed.

Heat layer and select2 element not working in Internet Explorer

Heat layer and select2 not working in Internet Explorer version 11.2007.14393.0

Combined the grid resolution controls

• Combine the grid resolution slider and text input into one leaflet control.

d3 chart pop up information not displaying the contextual site name at the moment.

• The site is showing as undefined. Probably to do with no longer using a global site look up or referencing the old look up table.

filtering for most generic taxonomic classification

Have a dropdown for just the kingdom. (Might be better to just implement the same multi-select set up in bpaotu australia)

make endemism filter apply independently from otu filtering.

Allow for endemism filtering when no filtering or only contextual filtering added.

Otu name lookups using otu pk failing sometimes

• I think this is due to filling the otu lookup via paginated results (50) whereas sample otu requests get as many ids as corresponds to a query (can include more than the 50 otus stored).
• Need to populate otu lookup table using sample otu requests I think.

Make leaflet control size proportional (%)

Returning no pathogen otu set on initial page load.

Has something to do with otu ids being return when querying without any filters.

Make select2 elements % size.

give min/max for site attributes as a heads up

Name search issues

Hi Warrick,
Here are some problems I've had when trying to search for taxon names. If I type in a name, it doesn't autocomplete like it used to do. That means that you have to actually scroll to the name, and don't have any obvious way of knowing whether that name is present or not (only tried this in the genus and species fields).
If I do select a name, the fields above (higher taxonomic levels) are restricted to those that contain whatever I selected (eg. that genus). This is cool, but not always useful, so it would be good to have the option to turn this on or off (off meaning you can select any name from the higher taxonomic rank).
Finally, the display is buggy, in that if I end up not selecting something or try to delete the selection, I can't get rid of the drop-down box to leave it blank/see the other fields. I've attached a screenshot that might help explain what I mean.
A-

add sidebar

Look into combine panels into a tabulated sidebar.

add integration tests for shepherd tutorial.

requires adding data-cy tags to shepherd elements to manipulate via cypress.

Update ingest to add landuse codes on import.

Add legend to map

Dynamically request missing otu names when loading popup content.

Currently taxonomic names are incrementally cached from otu filter suggestion results but selecting a grid cell requires all taxonomic names within that region including ones that have yet to be received by filter suggestion API. (e.g. when first 50 archaea names cached then clicking on a cell with 200 archaea species there will be at least 150 missing names.)

Solution: When a cell is clicked, calculate missing names, request the missing names then generate the popup content.

include endemism into the circle plot

Colour circles by year

Colour the plot circles by year. Currently contextual fields is just a string in the database which will need to be changed.

colour by environmental feature

Retroactively recording this as an issue. Was a requested feature by Austen d352913

add native filtering for otus

Search by text or give the option to search by otu name text when not using a filter option supplied by the api.

When using custom tags in otu search filter the url generated for the request uses text rather than a primary key set for the request. This does not currently work as the API currently only searches primary keys.

links with < in them don't work when pasted in an email

Graph coloring only working by elevation at the moment.

• The circle plot graph only colors by elevation at the moment.

Possibly make the taxonomic search suggestions construct more general search terms.

• e.g. Similar to Nick's previous version: If there is an organism classified up the 5 taxonomic ranks, create tags for the same organism but at the 4th, 3rd, 2nd and 1st ranks for users to easily select.

Load in other datasets

Currently contains fungi, bacteria and archaea. Need to import the rest.

re-implement cascading suggestions

Can no longer easily search for all fungi/etc as there's no option to search the parent of an otu entry. i.e. need one less level specificity to search a whole category.

Change d3 plot sizing.

Two ideas:

1

• scale circles so they should overlap less.
• magnify the hovered region so you can view the different plots in detail.

2

• increase the size and add a scrollbar

Make abundance heatmap the default view

taxonomic select initialization and contextual select initialization both call filter suggestions api.

Both dropdowns both call the filter suggestions api for their initial suggestion data but the API returns both contextual and taxonomic suggests (as filter suggestion api was originally created for a singular search filter). Either separate the API for taxonomic suggestions and contextual suggestions or call for the suggestions on page load once and pass the corresponding data to the select bars.

slider for grid resolution

Add analytics

color by biome (once biome contextual fields added)

• Color by the 2nd tier of biome classification

grid mode - circles should scale with zoom and fit in the matching grid circle

Make tags appear only on submission instead of every key press.

• Currently new tags get created too often so maybe change the condition for when a new tag is created from on key press (which is the default for select2) to on submission.

Add proportion of potential pathogens to plot.

Add proportion of potential pathogens to d3 circle plot.

Rewrite leafletGraphContro.update function

It depends on an old global siteAggregates variable which should be removed.

pie chart mode

add endemism filtering

Fix docker set up to allow for alteration to a back-end in a development environment.

Docker-compose breaks on muccg/* repositories. Need to fix this so the backend development stack can be run again and changes can be made safely.

Remove unused functions.

No longer using the old filtering and calculating functions so they can be removed.
Note: Need to rewrite the dropdown for the graph before this can be done as some of the d3 code relies on the dropdown (which relies on the old caulcating functions.)

Get missing otus from a search result.

• Currently the taxonomic lookup table is populated by filter option API responses. These responses are paginated (limit 50 results) whereas the abundances are not so there are sometime incompatible with otu entries in the lookup table.

• Need to get the missing ids then use them as a set to query the server then populate the popup content.