jhamman/ingestor

A library to facilitate the ingestion and normalization of climate datasets

Hi there 👋

I am a climate scientist and engineer. Around GitHub, you'll find me contributing to open source projects like Xarray and Pangeo. I am the CTO at Earthmover, a startup building the cloud platform for scientific data teams. Previously, I was a co-founder and the technology director at CarbonPlan, a non-profit working on data science and policy issues surrounding carbon removal and climate solutions, and a project scientist in the Climate and Global Dynamics Laboratory at the National Center for Atmospheric Research. I'm a licensed Professional Engineer and I have a Ph.D. in Civil and Environmental Engineering from the University of Washington.

ingestor's Issues

change default number of jobs

This seems to break with the default 20 parallel jobs, change default to 4 (or 8?)

Multiple datasets

Needs to be able to handle more input datasets (MERRA and CMIP5 being high priorities)

Add ESGF Class

ESGF hosts large datasets such as CMIP3/5. Access is either through wget or OpenDAP. I've recently tried to get some openDAP examples to work (ESGF/esgf-pyclient#26), without too much success.

Add CEDADataSet class

CEDA provides both FTP and OpenDAP access to CMIP5 (as well as other related datasets).

http://www.ceda.ac.uk/

Variable names

Names change from dataset to dataset, listing commonly used and or all available variables for each dataset, ideally with a short description would be a good functionality to have.

It would be even better to have a dictionary of ~20 popular variables that the ingestor can automatically translate into getting the correct variable from the correct dataset.

Add NASADataSet Class (or classes)

NASA datasets are spread among a number of DACs. For MERRA (1/2) and a number of other hydrologic datasets (e.g. LDAS), we can use the GES DISC OpenDAP servers.

https://disc.gsfc.nasa.gov/information/tools?title=OPeNDAP%20and%20GDS

Document installation and usage

briefly document in the readme how to install and which other libraries are required, along with a single simple example that gets a list of variables subset to a time and region. (this can pretty much just be cut and past from the last test example)

Default variable sets

A nice (easy) option would be to permit specifying "icar" inputs (or "GARD" or "BCSD-like") to get a pre-populated list of variables.

Similarly, default named regions (e.g. CONUS = "area": "55/-130/20/-60")