usgs-r/drb-temp-data-release-v2

A data release to support temperature modeling in the Delaware River Basin (2nd iteration)

drb-temp-data-release-v2's Issues

Update Zwart et al `Source code: Near-term forecasts ...` citation with 2022 when ready

Update Zwart et al Source code: Near-term forecasts ... citation with 2022 when ready.

This citation is currently found in in_text/text_SHARED.yml and in_text/text_03_driver.yml as:

  -
    authors: ['Jacob A. Zwart', 'Samantha K. Oliver', 'William D. Watkins','Jeffrey Sadler','Alison P. Appling','Hayley Corson-Dosch','Xiaowei Jia','Vipin Kumar','Jordan S. Read']
    title: >-
      Source code: Near-term forecasts of stream temperature using process-guided deep learning and data assimilation
    pubdate: 2021
    link: https://zenodo.org/record/5786612#.YtG75MZlBYx
  -

see commit f257992

Items that will need update is pubdate to 2022, and link. authors and title may need minor updating (to check once source code becomes available).

Create or Update file_functions.R for zipping ncdf files

file_functions zipping functions don't yet have a way to handle nc files.

Add code to handle nc files.

Clean in_data and out_data folders

First steps

• create IPDS record. Authors should include: Margaux + Jake + Sam + Alison...Sam will figure out the rest
• create SB item + child items that reflect the structure in #4
• Title: Data to support modeling water temperature in the Delaware River Basin
• Product Summary: Data release to support multiple modeling efforts in the Delaware River Basin
• team name could potentially be "AI/ML task within IWP PUMP"
• edit the files Sam added to this repository to reflect the child item structure, including the new SB IDs for the parent and child items.
• add gridMET to 3_drivers section + metadata to the appropriate in_text file

Overall data release structure

1. Spatial data [all locally built in delaware-model-prep]

• river shapefiles [delaware-model-prep]
• river segment metadata [delaware-model-prep]
• reservoir shapefiles [lake-temp-model-prep]
• reservoir metadata [delaware-model-prep?]
• site shapefile [delaware-model-prep]
• site metadata [delaware-model-prep]

2. Observations [all locally built in delaware-model-prep but with some other upstream pipeline dependencies]

• river temperature observations raw
• river temperature observations aggregated to single value per reach-day
• flow observations
• reservoir levels
• reservoir releases
• reservoir temperature observations
• reservoir diversions

3. Driver data

• NEW gridMET - on Caldera, which Jeff or @msleckman can point us to
• NEW GEFS historical/reanalysis product - @jzwart will point us to Caldera location
• PRMS-SNTemp ins/outs - built here and stored in /caldera/projects/usgs/water/iidd/datasci/water-prediction/run-prms-sntemp/4_model_run/out/uncal_sntemp_input_output.nc
• distance matrix (or does this go in spatial data?) - locally build in delaware-model-prep

Update any zipped rds file to csv.

Reservoir data uncertainties

Where should reservoir data live?

I think the data that can be broadly useful and easily updateable from public sources (NWIS) should remain in the "main" data release. I think the GLM-specific reservoir data/outputs should go into the "forecast" data release.

I think we can simplify the data provided in this release into fewer files including:

1. reservoir temperature observations
2. a simplified version of the daily reservoir water budget information, including historical interpolations, water levels, and reservoir releases (anything needed to run GLM)

str_detect() - no applicable method for 'type' applied to an object of class "AsIs"

drb-temp-data-release-v2/src/spatial_functions.R

Line 31 in 364bbb1

filter(str_detect(string = filename, pattern = layer_name)) %>% pull(filename)

Getting the following error on the filter(str_detect()) part of this function.

files_to_zip <- data.frame(filepath = dir(dsn, full.names = TRUE), stringsAsFactors = FALSE) %>%
     mutate(filename = basename(filepath)) %>%
     filter(str_detect(string = filename, pattern = layer_name))

Error in `filter()`:
! Problem while computing
  `..1 = str_detect(string =
  filename, pattern =
  layer_name)`.
Caused by error in `UseMethod()`:
! no applicable method for 'type' applied to an object of class "AsIs"
Run `rlang::last_error()` to see where the error occurred.

suggestion changing to :

files_to_zip <- data.frame(filepath = dir(dsn, full.names = TRUE), stringsAsFactors = FALSE) %>%
     mutate(filename = basename(filepath)) %>%
     filter(grepl(pattern = layer_name, x = filename)) %>%
     pull(filename)

replace in_data/drb_filtered_sites.rds with in_data/drb_filtered_sites_qa.rds

Data that should be added to this release

Most files are documented in the old data release yml files ported over. However, we'll also need to include:

• gridMET data
• GEFS historical reanalysis data

Keeping files as rds on science base?

@limnoliver

Is it ok to have keep these zipped datasets as .rds? Or should we turn them into csv. ?

drb-temp-data-release-v2/2_observations.yml

Lines 26 to 33 in 8a94e93

	out_data/unaggregated_temperature_observations_drb.zip:
	command: zip_obs(out_file = target_name, in_file = 'in_data/02_observations/obs_temp_drb_raw.rds')
	out_data/aggregated_temperature_observations_drb.zip:
	command: zip_obs(out_file = target_name, in_file = 'in_data/02_observations/obs_temp_drb.rds')
	out_data/flow_observations_drb.zip:
	command: zip_obs(out_file = target_name, in_file = 'in_data/02_observations/obs_flow_drb.rds')

Same question for in comment in forecast-data-release-v2

Move gridmet data in caldera to in_data

Placing this issue here to remember to copy over the gridmet data from caldera (/caldera/projects/usgs/water/impd/pump/gridmet/drb_gridmet_tools/data/out/drb_climate_2022_04_06.nc) to the in_data.

Creating sub-folders 01,02,03 in out_data?