Can numpy objects be supported?

[adam2392]

Can numpy objects be supported?

[kmuehlbauer]

@adam2392 Thanks for raising this. A pure h5netcdf MCVE would be nice nevertheless.

For NetCDF4 data model

Attributes are always scalar values or 1D arrays, which can be associated with either a variable or the file as a whole. Although there is no enforced limit, the user is expected to keep attributes small.

So this means that this would be a non-standard usage which h5netcdf could only support using invalid_netcdf keyword arg.

Also HDF5 speaks of Attributes as small datasets.

How small is "small" is not defined by the library and is up to the user’s interpretation.

As the request is not conforming to the NetCDF4 standard I'm inclined to keep the current behavior. You might go the suggested way (HDF5-link above) of adding the data as another variable with an attribute pointing to it.

[kmuehlbauer]

Xref: mne-tools/mne-connectivity#87

[kmuehlbauer]

After reading the discussion over at mne-tools/mne-connectivity#87 I think I might be missing a bit of context.

@adam2392 In the mne-connectivity docs the output format is mentioned as NetCDF. What you request would finally create non-standard NetCDF files. Maybe I'm missing something. Could you shed some light?

[shoyer]

It isn't possible to directly put arbitrary NumPy object arrays into HDF5 files, because these can contain arbitrary Python objects that may not be supported by HDF5.

If desired, arbitrary Python objects could be serialized into bytes with pickle. But this would not be appropriate for h5netcdf unless users need to explicitly opt-in each time they open a file with pickle support (e.g., by specifying something like allow_pickle=True), because deserializing pickle data allows arbitrary code execution.

[kmuehlbauer]

I'm closing this as wont-fix. If there is more discussion needed, please reopen @adam2392.