Modifying attributes safely is not possible with all datasets.

[Frooob]

Modifying attributes safely is not possible with all datasets.

[kmuehlbauer]

I've updated my issue over at HDFGroup/hdf5#1388. Let's see if there will be any movement.

[kmuehlbauer]

@Frooob

Currently setting an existing attribute via h5py is creating an new temporary attribute, deleting the existing attribute followed by renaming the temporary attribute to the existing name. That deletion/rename is the source of our issue, as it triggers the above mentioned bug in hdf5.

We could just try to modify the existing attribute, but we will be bound to the existing datatype. In the case of your variable_id this is a fixed length string of size 3:

HDF5 "test.nc" {
ATTRIBUTE "variable_id" {
   DATATYPE  H5T_STRING {
      STRSIZE 3;
      STRPAD H5T_STR_NULLTERM;
      CSET H5T_CSET_ASCII;
      CTYPE H5T_C_S1;
   }
   DATASPACE  SCALAR
   DATA {
   (0): "tas"
   }
}
}

Using h5py AttributeManager's modify-function we can change the content to another fixed length-string of size 3. But we can only change it to a fixed length string of size 2, since h5py uses NULLPAD but the origin uses NULLTERM (most likely netcdf-c).

To establish an extendable string you would need to delete and recreate the attribute as variable length string:

with h5netcdf.File(testfile, "r+") as f:
    print(f.attrs["variable_id"])
    del f.attrs["variable_id"]
    # create variable-length string
    f.attrs["variable_id"] = b"changed_with_h5netcdf"
    print(f.attrs["variable_id"])
with h5netcdf.File(testfile, "r+") as f:
    print(f.attrs["variable_id"])
    # call h5py-modify
    f.attrs._h5attrs.modify("variable_id", b"changed_with_h5netcdf_again")
    print(f.attrs["variable_id"])
with h5netcdf.File(testfile, "r+") as f:
    print(f.attrs["variable_id"])
    # call h5py-modify
    f.attrs._h5attrs.modify("variable_id", b"changed_with_h5netcdf_again_and_again_to_make_up_for_a_huge_string")
    print(f.attrs["variable_id"])

tas
changed_with_h5netcdf
changed_with_h5netcdf
changed_with_h5netcdf_again
changed_with_h5netcdf_again
changed_with_h5netcdf_again_and_again_to_make_up_for_a_huge_string

To me, it looks like this change is best located upstream at h5py. I'll open an issue in h5py.

[Frooob]

Wow really cool, in our code I have also implemented the workaround now, checking if the type of the attribute already is a string with variable size and only then use modify, if it's a fixed size string, delete the attribute and create a new one. I'm already quite happy with that. 😄

[kmuehlbauer]

@Frooob Just a heads-up, the above mentioned issue was upleveled to Priority High 1. at HDF5 Group. So we might just wait it out, now.

[kmuehlbauer]

@Frooob Thanks for raising this. Unfortunately that error is originating from hdf5.

Please have a look in

h5py/h5py#1385

and

HDFGroup/hdf5#1388

for more context.

I've tried to fix this over at hdf5, but unfortunately my C/C++ skills are not sufficient for that task.

If you strongly rely on that feature, please try to convince upstream to fix this.

[kmuehlbauer]

Also relevant #202

[kmuehlbauer]

@Frooob After recollecting my thoughts on that issue, we might actually be able to make this work by changing h5py.

In your use-case the dtype of your attribute is the same throughout consecutive changes, right?

Update: Or we might just change it here in h5netcdf.

[Frooob]

Yes, nothing in the dataset besides the attributes should be changing

[kmuehlbauer]

@Frooob I've opened a PR over at h5py/h5py#2206.

It might nevertheless be a good idea to add that workaround in h5netcdf to cover for older h5py releases.

[Frooob]

This is fixed in h5py (h5py/h5py#2274)