Comments (10)
I've updated my issue over at HDFGroup/hdf5#1388. Let's see if there will be any movement.
from h5netcdf.
Currently setting an existing attribute via h5py is creating an new temporary attribute, deleting the existing attribute followed by renaming the temporary attribute to the existing name. That deletion/rename is the source of our issue, as it triggers the above mentioned bug in hdf5.
We could just try to modify the existing attribute, but we will be bound to the existing datatype. In the case of your variable_id
this is a fixed length string of size 3:
HDF5 "test.nc" {
ATTRIBUTE "variable_id" {
DATATYPE H5T_STRING {
STRSIZE 3;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "tas"
}
}
}
Using h5py AttributeManager
's modify
-function we can change the content to another fixed length-string of size 3. But we can only change it to a fixed length string of size 2, since h5py uses NULLPAD
but the origin uses NULLTERM
(most likely netcdf-c).
To establish an extendable string you would need to delete and recreate the attribute as variable length string:
with h5netcdf.File(testfile, "r+") as f:
print(f.attrs["variable_id"])
del f.attrs["variable_id"]
# create variable-length string
f.attrs["variable_id"] = b"changed_with_h5netcdf"
print(f.attrs["variable_id"])
with h5netcdf.File(testfile, "r+") as f:
print(f.attrs["variable_id"])
# call h5py-modify
f.attrs._h5attrs.modify("variable_id", b"changed_with_h5netcdf_again")
print(f.attrs["variable_id"])
with h5netcdf.File(testfile, "r+") as f:
print(f.attrs["variable_id"])
# call h5py-modify
f.attrs._h5attrs.modify("variable_id", b"changed_with_h5netcdf_again_and_again_to_make_up_for_a_huge_string")
print(f.attrs["variable_id"])
tas
changed_with_h5netcdf
changed_with_h5netcdf
changed_with_h5netcdf_again
changed_with_h5netcdf_again
changed_with_h5netcdf_again_and_again_to_make_up_for_a_huge_string
To me, it looks like this change is best located upstream at h5py. I'll open an issue in h5py.
from h5netcdf.
Wow really cool, in our code I have also implemented the workaround now, checking if the type of the attribute already is a string with variable size and only then use modify, if it's a fixed size string, delete the attribute and create a new one. I'm already quite happy with that. 😄
from h5netcdf.
@Frooob Just a heads-up, the above mentioned issue was upleveled to Priority High 1.
at HDF5 Group. So we might just wait it out, now.
from h5netcdf.
@Frooob Thanks for raising this. Unfortunately that error is originating from hdf5.
Please have a look in
and
for more context.
I've tried to fix this over at hdf5, but unfortunately my C/C++ skills are not sufficient for that task.
If you strongly rely on that feature, please try to convince upstream to fix this.
from h5netcdf.
Also relevant #202
from h5netcdf.
@Frooob After recollecting my thoughts on that issue, we might actually be able to make this work by changing h5py.
In your use-case the dtype of your attribute is the same throughout consecutive changes, right?
Update: Or we might just change it here in h5netcdf.
from h5netcdf.
Yes, nothing in the dataset besides the attributes should be changing
from h5netcdf.
@Frooob I've opened a PR over at h5py/h5py#2206.
It might nevertheless be a good idea to add that workaround in h5netcdf to cover for older h5py releases.
from h5netcdf.
This is fixed in h5py (h5py/h5py#2274)
from h5netcdf.
Related Issues (20)
- Corrupted headers when serialising using dask.distributed client HOT 9
- Test failures with NetCDF 4.9.0 HOT 6
- Remove h5py2 related code and CI builds HOT 3
- FAILED h5netcdf/tests/test_h5netcdf.py::test_group_names HOT 6
- AttributeError for '_phony_dim_count' when trying to convert a file made with h5py HOT 12
- Tests test_more_than_7_attr_creation_track_order and test_bool_slicing_length_one_dim fail in the test suite HOT 18
- very slow partial reading when saved with index shift HOT 10
- h5py minimum version update? HOT 5
- Improving performance for h5netcdf HOT 13
- Documentation request: Alternative way to obtain h5netcdf HOT 1
- Segmentation fault after upgrading to h5netcdf==1.1.0 HOT 14
- ValueError raised when attribute has type `h5py.Reference` HOT 5
- support for HDF5 dimension scales with null dataspace HOT 2
- Better Error for illegal variable names HOT 3
- md5 checksum mismatch for identical files/data HOT 7
- Question: does `h5netcdf` bring in the entire data from a netCDF file on a remote disk (like S3)? HOT 5
- Provide an example with time dimension readable by paraview HOT 3
- transient vs committed types HOT 5
- string decoding produces pairs of low surrogates HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from h5netcdf.