Comments (4)
Hi @aladinor - have you checked that this behaviour is actually different from what xr.open_dataset
will do for one dataset? Because the aim here should be compliance with that behaviour, and clarity as to what options should be passed if you want different behaviour (e.g. passing chunks={}
).
from datatree.
Hi @TomNicholas. This is a really good question. So the main thing that I found out is that xarray.open_dataset
has its default parameters. Thus, when passing the **kwargs
dictionary from datatree.open_datatree
to xarray.open_dataset
, it will overwrite all the default parameters and set them None
. At least in the test I performed, if none of those default arguments were passed, xr.open_dataset
would behave as I mentioned in #328. I think that If we want to keep xr.open_dataset
, all the parameters should be passed within the **kwargs
dictionary or try to find a way to not overwrite the default args in xr.open_dataset
. I could not find a way to do that without retyping/passing all of them from datatree.open_datatree
to xr.open_dataset
. That's why I felt using xr.open_zarr
was more convenient. However, I will recheck it using one dataset
from datatree.
Hi @TomNicholas. This could be a reproducible example for you to check.
import xarray as xr
import datatree
import numpy as np
import pandas as pd
def main():
data = 283 + 5 * np.random.randn(5, 3, 4)
times = pd.date_range("2018-01-01", periods=5)
lons = np.linspace(-120, -90, 4)
lats = np.linspace(25, 55, 3)
temp = xr.DataArray(data, coords=[times, lats, lons], dims=["time", "lat", "lon"]).chunk({'time': 1,
'lat': -1,
'lon': -1})
ds = xr.Dataset(data_vars={'Temp': temp})
ds2 = xr.Dataset(data_vars={'Temp_C': temp - 273.15}).chunk({'time': 1, 'lat': 1, 'lon': 1})
dtree = datatree.DataTree(data=ds)
datatree.DataTree(ds2, parent=dtree, name='set2')
print(dtree.Temp.chunksizes)
print(dtree.set2.Temp_C.chunksizes)
path_zarr = "test.zarr"
dtree.to_zarr(path_zarr)
dtree_back = datatree.open_datatree(path_zarr, engine='zarr')
print(dtree_back.Temp.chunksizes)
print(dtree_back.set2.Temp_C.chunksizes)
...
if __name__ == "__main__":
main()
And the output
Frozen({'time': (1, 1, 1, 1, 1), 'lat': (3,), 'lon': (4,)})
Frozen({'time': (1, 1, 1, 1, 1), 'lat': (1, 1, 1), 'lon': (1, 1, 1, 1)})
Frozen({})
Frozen({})
The expected output without passing any parameters to open_datatree
should be to keep the same chunk sizes. However, it didn't.
Let me know your thoughts.
from datatree.
Hi @TomNicholas. I looked again at xr.open_dataset
documentation and I found this
In order to reproduce the default behavior of xr.open_zarr(...) use xr.open_dataset(..., engine='zarr', chunks={}). chunks='auto' will use dask auto chunking taking into account the engine preferred chunks. See dask chunking for more details.
I think based on I will close this issue and the PR. Thanks for your time!
from datatree.
Related Issues (20)
- Resolve dimension coordinates from parents HOT 8
- Tree linkage + dataset in one object nonideal for UI tree view HOT 6
- v0.0.14 not on pypi HOT 1
- setting node name breaks tree linkage HOT 7
- When creating a DataTree from a Dataset with path-like variable, subgroups are expected to be created HOT 8
- Actually you're right, I don't know if the docs currently mention anywhere that assigning to `.ds` is allowed!
- Typing: `DataTree[Unknown]`
- `drop_vars` issue? HOT 1
- Creating DataTree from DataArrays
- Opening a datatree from S3 bucket HOT 3
- Typing issue: Pylance complains with DataTree inequalities HOT 1
- open_datatree() keeps the hdf file open preventing writes HOT 2
- `open_datatree` performance HOT 2
- decision analysis in datatree? HOT 1
- Collapse by default the "Attributes" section in rich display HOT 1
- Add an `attrs` keyword argument to the constructor: `DataTree(attrs={})` HOT 1
- Describe a DataTree: adidng visualization and summarization capabilities HOT 1
- Rich display width is broken HOT 4
- Auto-plotting capabilities HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datatree.