Comments (8)
Thank you for spotting this!
This flat Dataset containing path-like variable name is expected to produce groups and subgroups once injected into a DataTree.
Yes, or a simpler solution would be just to raise an error if someone tries to pass a Dataset which has path-like variable names. We are then basically assuming that (a) any Dataset passed is supposed to represent a group, and (b) a single group cannot have path-like variable names.
I think there is also an argument to be made with round-tripping. I think we want this property to hold always:
ds # start with a dataset
dt = DataTree(ds)
new_ds = dt.to_dataset() # expect new_ds == ds
If ds contains path-like variable names, that are then automatically turned into different groups, this property won't work. If we just forbid it, we can make sure it works in all other cases.
from datatree.
I personally think this little loop is fine - it's clear, explicit and should't be buggy (once we fix the bug you reported in this issue!). If many other people appear to say that they would like a special function for ingesting this type of non-compliant dataset then we could revisit the idea.
from datatree.
Note: this issue was generated from a notebook. You can use it to reproduce locally the bug.
from datatree.
Hello, thanks for your answer!
Indeed the round-trip capability seems the cleanest default behaviour. A Dataset is supposed to be a group. I think what I needed already exists: it's DataTree.from_dict
.
Line 1035 in 0afaa6c
I can have a look to fix the bug (forbidding Datasets containing path-like variable names), and I can suggest an error message like this to help the user that probably wanted this "auto-creating-group" behaviour but did not knew about .from_dict
:
Given Dataset contains path-like variable names.
A Dataset represents a group, and a single group cannot have path-like variable names.
Consider `DataTree.from_dict` to automatically create groups from a mapping of paths to data objects
(if you have an idea to shorten and improve it I am interested!)
Here is what I really wanted to do when creating the issue:
from pathlib import PurePosixPath
# Note: paths represent variable names ; the mapping should only contain groups, hence `.parent`
# Note 2: the named DataArray is renamed to only keep the name part of the path.
mapping_path_to_dataarray = {
PurePosixPath(str(varname)).parent: xda.rename(PurePosixPath(str(varname)).name)
for varname, xda in xds.items()
}
mapping_path_to_dataarray
{PurePosixPath('group/subgroup'): <xarray.DataArray 'my_variable' (label: 3, z: 18)>
dask.array<xarray-<this-array>, shape=(3, 18), dtype=int64, chunksize=(2, 4), chunktype=numpy.ndarray>
Coordinates:
* label (label) <U1 'a' 'b' 'c'
* z (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17}
Though this logic will work, it seems a little bit over-complicated to me.
However, a Dataset carrying variables names containing paths is a little bit
a misuse of a Dataset. A DataTree should be used in the first place
if possible.
xdt = dt.DataTree.from_dict(mapping_path_to_dataarray)
xdt if INTERACTIVE else print(xdt)
DataTree('None', parent=None)
└── DataTree('group')
└── DataTree('subgroup')
Dimensions: (label: 3, z: 18)
Coordinates:
* label (label) <U1 'a' 'b' 'c'
* z (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Data variables:
my_variable (label, z) int64 dask.array<chunksize=(2, 4), meta=np.ndarray>
Note: Pylance is not happy of usage of dict of PurePosixPaths. It only expects str as keys.
Maybe it could be worth it to make it accepts Pure Paths too?
Argument of type "dict[PurePosixPath, DataArray]" cannot be assigned to parameter "d" of type "MutableMapping[str, Dataset | DataArray | DataTree[Unknown] | None]" in function "from_dict"
"dict[PurePosixPath, DataArray]" is incompatible with "MutableMapping[str, Dataset | DataArray | DataTree[Unknown] | None]"
Type parameter "_KT@MutableMapping" is invariant, but "PurePosixPath" is not the same as "str"
Type parameter "_VT@MutableMapping" is invariant, but "DataArray" is not the same as "Dataset | DataArray | DataTree[Unknown] | None"PylancereportArgumentType
Reconverting the DataTree gives an empty Dataset, as the root contains only groups
but no concrete variables:
reconverted = xdt.to_dataset()
reconverted if INTERACTIVE else print(reconverted)
<xarray.Dataset>
Dimensions: ()
Data variables:
*empty*
Reconverting the subgroup containing the variable is successful
reconverted = xdt["group/subgroup"].to_dataset()
reconverted if INTERACTIVE else print(reconverted)
<xarray.Dataset>
Dimensions: (label: 3, z: 18)
Coordinates:
* label (label) <U1 'a' 'b' 'c'
* z (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Data variables:
my_variable (label, z) int64 dask.array<chunksize=(2, 4), meta=np.ndarray>
from datatree.
Complement:
After re-reading all of this, I think the most natural behaviour from a user perspective is the one of __setitem__
:
xdt = dt.DataTree()
for varname, xda in xds.items():
xdt[varname] = xda
The only issue I have is that this make the for loop unavoidable as it contains an assignment (imperative not declarative)
The over-complicated solution...
mapping_path_to_dataarray = {
PurePosixPath(str(varname)).parent: xda.rename(PurePosixPath(str(varname)).name)
for varname, xda in xds.items()
}
...was only to get a dict-comprehension rather than a for loop with assignments.
from datatree.
(if you have an idea to shorten and improve it I am interested!)
That looks good - I think long error messages are good if they are adding more context, like yours is. Only thing I would add is an explicit list of the offending variable names.
Note: Pylance is not happy of usage of dict of PurePosixPaths. It only expects str as keys.
Maybe it could be worth it to make it accepts Pure Paths too?
The Posix part is to forbid backslashes, only allowing forward slashes. Maybe we can forbid those at runtime whilst relaxing the type though.
After re-reading all of this, I think the most natural behaviour from a user perspective is the one of
__setitem__
:
Why? What do you dislike about the "one dataset per group" logic?
from datatree.
Okay for adding the offending variable names!
I completely agree with the "one dataset per group" logic. This is more from a usage perspective where I find these 3 lines to be the simplest to convert a Dataset containing paths to a DataTree:
xdt = dt.DataTree()
for varname, xda in xds.items():
xdt[varname] = xda
The user does not have to think that much: the assignment will take care of creating groups and renaming the DataArrays from the source Dataset accordingly. The only thing I quite dislike is having to make the for-loop and assignments, not having a declarative comprehension-like approach to achieve what these lines do, without becoming verbose. But I must admit this is mainly about aesthetics rather than a true need, so I don't think anything more needs to be done on this topic!
from datatree.
Related Issues (20)
- Resolve dimension coordinates from parents HOT 8
- Tree linkage + dataset in one object nonideal for UI tree view HOT 6
- v0.0.14 not on pypi HOT 1
- setting node name breaks tree linkage HOT 7
- Actually you're right, I don't know if the docs currently mention anywhere that assigning to `.ds` is allowed!
- Typing: `DataTree[Unknown]`
- `drop_vars` issue? HOT 1
- Creating DataTree from DataArrays
- Opening a datatree from S3 bucket HOT 3
- Typing issue: Pylance complains with DataTree inequalities HOT 1
- open_datatree() keeps the hdf file open preventing writes HOT 2
- open_datatree() from zarr creates issues with `kwargs` HOT 4
- `open_datatree` performance HOT 2
- decision analysis in datatree? HOT 1
- Collapse by default the "Attributes" section in rich display HOT 1
- Add an `attrs` keyword argument to the constructor: `DataTree(attrs={})` HOT 1
- Describe a DataTree: adidng visualization and summarization capabilities HOT 1
- Rich display width is broken HOT 4
- Auto-plotting capabilities HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datatree.