pangeo-data/esgf2xarray

utilities for loading esgf archives as xarray datasets

esgf2xarray's Issues

make this an actual package!

We need a

• setup.py
• README.md
• add tests for aggregation
• travisci integration

compare search functionality with intake-esm

https://github.com/NCAR/intake-esm
https://intake-esm.readthedocs.io/en/latest/index.html

We want to make sure that the dataframes returned by both search modules are compatible with the aggregation functions. (Consistent column names, necessary fields, etc.)

add functionality which flags potential issues before aggregation

The files dataframe returned by esgf_search is ideal for identifying issues which might arise during aggregation. For example, there can be both ri1p1f1 and ri1p1f2 type ensemble members which should probably not be combined. Or, if there are two versions (v20190101 and v20190202, so use the latest) of the same file or if one ensemble member covers only a subset of the time it would be good to flag these issues before aggregation.

I am keeping track of all the anomalies I see in the CMIP6 data and propose to write a new function to check files , flagging potential issues so the they can be dealt with on a case by case basis rather than silently dealing with them in combine_files.

rethink logic of decode_times in open_dataset

This how we load individual files:

esgf2xarray/esgf2zarr/aggregate.py

Lines 58 to 63 in 9cfb9f0

	def open_dataset(url, default_chunk_size='12MiB'):
	# try to use smaller chunks
	with dask.config.set({'array.chunk-size': '12MiB'}):
	ds = xr.open_dataset(url, chunks={'time': 'auto'}, decode_times=False)
	ds.attrs['history'] = f"{datetime.now()} xarray.open_dataset('{url}')"
	ds = set_coords(ds)

The problem is that time_bnds can't be auto chunked if it is a cftime type. Now we are avoiding that by not decoding time at all; however, this could possibly lead to errors if different files have incompatible reference times e.g. days since 1950-01-01.

An alternative would be to eagerly load time_bnds so dask doesn't complain about its object dtype.

keep track of xarray ncml progress

pydata/xarray#2697

Eventually most of the aggregation logic can hopefully be replaced by xarray.open_ncml.