s-enda/dmh

Data management handbook for S-ENDA partners

Data Management Handbook

This handbook describes general data management practices of the S-ENDA partners (available here):

• Norwegian Meteorological Institute (Meteorologisk institutt - MET)
• Norwegian Institute for Air Research (Norsk institutt for luftforskning - NILU)
• Norwegian Institute for Nature Research (Norsk institutt for naturforskning - NINA)
• Norwegian Institute for Water Research (Norsk institutt for vannforskning - NIVA)

It focuses on the management and sharing of dynamical geodata in line with the FAIR (Findable, Accessible, Interoperable, and Reusable) guiding principles for scientific data management.

Any suggestions for improvement are welcome by either creating an issue or pull request.

dmh's Issues

Data Structure and documenting - Introduction

          Text that is in section 2 is not important, but table is important and should be included. From other sections in mets template, 1.1.5 (FAIR), 1.1.3 (datasets) and 1.1.4 (metadata) should be included before the table. 

Vocabulary should be a part here

Originally posted by @lisemurberg in #12 (comment)

Information from MET

• Update Morten about our work
• Investigate if there is any material in MET’s DMH that must be kept in S-ENDA’s handbook. If so, take examples from MET’s official DMH.
• Add some MET-specific information on the Practical Guides

Add institute specific data managenet summary in roles.md

Add summary on data management on roles.md

New format - mdBook?

Please take a look into this before the next DMH-group update!

NINA would like to have a flat document, and suggested using mdBook to create the document. NIVA and NINA have experience with it.

MdBook is quite easy to make changes, and it is well documented. A couple of installations are required to run the document locally. Here is the installation instructions.

Here is the link to the draft document - https://s-enda.github.io/DMH/

Add bulletpoints from MET practical guide

Review text - Practical guides and data services

Contacts and roles

Fill out required information on the sheet Role in the FAIR.xlsx spreadsheet.

• NIVA
• NILU
• NINA
• #9

Access to data (without downloading it)

GDAL supports accessing to data remotely without having to download it in advance.

Should we mention it and provide some code snippets on how to do it in Python/R?

Here is an example of using GDAL virtual file system to access a remote NetCDF file without downloading it:

url='NETCDF:/vsicurl/https://thredds.met.no/thredds/fileServer/senorge/seNorge_2018/Archive/seNorge2018_2006.nc'
data2006 = stars::read_stars(url, sub='rr', proxy=TRUE)
[...]
st_crs(data2006)
reg1_st <- reg1 %>% st_as_sf(.,
                             coords = c("longitude", "latitude"), 
                             crs=4326)

reg1_st <- st_transform(reg1_st, crs=st_crs(data2006))
reg1_st$rr <- st_extract(data2006, reg1_st, time_column="date")

This snippet uses R stars library.

Such a URL works with gdalinfo from command line and on a wide variety of software using GDAL.

Maybe Benjamin Cretois can help with that as well?

Review text - Introduction, data governance

Introduction, data governance

Move Metadata types table to metadata subchapter

move from metadata.md to struct_metadata.md

FAIR convergance

Include FAIR convergence in FAIR chapter.

Figure in data management does not show in all browsers

https://s-enda.github.io/DMH/data_management.html

smaill issue with picture in data management section

Figure 1 shows as:

_

Figure 1. Value chain for data_

Discussion on table of contents

Come up with suggestions for slightly modified table of contents

FAIR should be moved.

Data Structure and documenting - Vocabulary

Include information about vocabulary for structuring and documenting data

Find data using Geonorge CSW

MET DMH explains how to use MET's CSW endpoint for finding data. Should we have a common section for using GeoNorge CSW endpoint to find S-ENDA data?

It looks that QGIS MetaSearch often comes with GeoNorge CSW endpoint already configured.

Data governance - Introduction

Should be moved to the top of the DMH, and be quite general as we think this is quite introductory.

The sections from MET are more or less ok, but needs to be rewritten a bit.

Its important to explain both what data governance is and why its important.

Example from NILU, The ACTRIS Data Management Plan: https://github.com/actris/data-management-plan

Update Acronyms

Acronym list is outdated and should be updated.

How to search geonorge for datasets

Describe and link to documentation on searching geonorge for datasets.

Review text - Structuring and documenting data

Read and review chapter Structuring and documenting data and its subchapters

S-ENDA architecture

Include as chapter 4.2 (move contact and roles to 4.3)

Include landskapsanalyse picture (updated?).

Include information from Arkitekturdokument

Add institute specific structuring and documenting data in struct_institute.md

To do for each institute:

• MET
• NILU
• NIVA
• NINA

Include how to achieve FAIR in practical guide.

How do we become Interoperable, Accessable, Reusable etc.

Wishes for practical guide - subchapters

Assign to WP leaders: Johannes and Markus

Guide on using CSW

Add a description on how to use catalog service for the web

Practical guides review

Practical guides should be reviewed by Johannes once #35 is closed

NIVA Tasks

• Create issues for actions in the agenda on GitHub
  Correct formatting in FAIR.xlsx (sheets Glossary and metadata)
• Update README.md

Data structure and documenting - Table

Each institute will fill our their current practices for Structuring and Documenting Data for Efficient Discovery and Use.

• MET
• NILU
• NINA
• NIVA

Remove the geonorge download API

Comment from Lars-Inge: Take away Geonorge's download API section. This is irrelevant and we should avoid national apis, but focus on the international apis.

Add chapter 7 from MET's DMH

Add relevant links from chapter 7 - How do find data

• Important to add context to links. Ex: Introduction to OpenSearch.
  The template hasn't been updated, need output from @mortenwh

Add glossary

Add Glossary from MET's common template

MET

DMH content

Look into the content of MET’s DMH, see what can be recycled

Data services text

update the data services text

NINA's tasks

• Practical guides example
• Add information about vocabulary

Guide on xarray and opendap

Add a practical guide on accessing opendap using xarray and creating example datasets.

Introduction for institute-specific data management

Data services

Data Services

3.1. Purpose
3.2. Implementation

User Portals and Documentation .

4.1. Purpose
4.2. Implementation …

User portal should be inside data services.

Data Structure and documenting- Systems FAIRness

Investigate a way to compare systems fairness

Data structure and documenting - Introduction

          Text that is in section 2 is not important, but table is important and should be included. From other sections in mets template, 1.1.5 (FAIR), 1.1.3 (datasets) and 1.1.4 (metadata) should be included before the table. 

Vocabulary should be a part here

Originally posted by @lisemurberg in #12 (comment)

Review text - Data management

Review chapter and its sub-chapters

Issue assignent!

Could you please assign yourself or someone you believe is the right person for the task? I have assigned myself for #63, #69, and #70, and left a suggestion on #59 and #67. Please mark the box(es) with the issues you have assigned.

• #68

• #67

• #66

• #65

• #64

• #61

• #60

• #59

Content from MET’s DMH that should be maintained in S-ENDA’s handbook. To be re-written to fit and structured accordingly

Move this information to our hanbook so we can start working on changes:
Content from MET’s DMH that should be maintained in S-ENDA’s handbook. To be re-written to fit and structured accordingly

• 1.1. The principles of data management for dynamic geodata.
• Structuring and Documenting Data for Efficient Discovery and Use
  • 2.1. Purpose.
  • 2.2. Implementation
  • 2.2.1. Current practice
• Data Services
  • 3.1. Purpose
  • 3.2. Implementation
• User Portals and Documentation .
  • 4.1. Purpose
  • 4.2. Implementation …
• Data Governance
  • 5.1. Purpose
  • 5.2. Data life cycle management
  • 5.2.1. Data Management Plan.
  • 5.3. Implementation …
• Practical Guides

Originally posted by @elinahkk in #2 (comment)

Update Glossary with relevant terminology

Width of hanbook

Try to increase the width of the handbook
https://rust-lang.github.io/mdBook/format/configuration/general.html

Practical guides

• Add some MET-specific information on the Practical Guides (from issue #2 )
• NINA come with examples on practical guides (from issue #5 )

include m-2-m metadata access points for table "user portals relevant to S-ENDA partners"

For completeness of the table, it would be useful to include m-2-m metadata access points as well.

Practical guides - what do we need?

It would be good to have a guide on how we find S-ENDA's datasets in Geonorge. We need suggestions from the other groups, specially Dataset-group

Clean up old files

There are some old files (ex. DMH.Rmd, FAIR.xlsx) in the main branch. This should be cleaned up/removed.

Update general data management text

Update general data management text in chapter 4. to work for whole chapter, not just information about data management roles.

Should cover:
4.1 Institute specific data management
4.2 S-ENDA Architecture
4.3 Contact and roles