Support for container or `Eigen::Map`

[mlund]

Support for container or `Eigen::Map`

[DavidAce]

I'm happy to take a look.

Currently h5pp supports contiguous containers such as std::array, c-style arrays, std::vectors or Eigen (or Eigen::Map!) with POD data. As far as I can tell, there is no way to tell HDF5 to write a non-contiguous block of data (i.e. to specify a stride), but I'm still learning. I have decided not to support ranges because not all STL containers are contiguous, like std::list. I think offering the possibility would send mixed messages: I would either have to copy the list data to a contiguous vector, or write elements one by one to individual datasets, but these are decisions a user should make instead. So I decided to simply check if the container has a ".data()" method - contiguous containers usually do.

However, going back to your problem, it is possible to write entire POD structs, or (contiguous) containers thereof. It's just a matter of registering and defining a compound HDF5 type before writing it. In fact, I do this at initialization to enable support for std::complex, and CUDA structs like struct double2{double x,y;}. Code is found in include/h5pp/details/h5ppTypeCompoundCreate.h

So for instance, a struct like,

struct Particle {
    double x,y,z,t;
    int dim;
    
}

is registered in HDF5 with

        // hid_t can be replaced by a safer h5pp::hid::h5t below,
        hid_t MY_HDF5_PARTICLE_TYPE = H5Tcreate(H5T_COMPOUND, sizeof(Particle));
        H5Tinsert(MY_HDF5_PARTICLE_TYPE, "x", HOFFSET(Particle, x), H5T_NATIVE_DOUBLE);
        H5Tinsert(MY_HDF5_PARTICLE_TYPE, "y", HOFFSET(Particle, y), H5T_NATIVE_DOUBLE);
        H5Tinsert(MY_HDF5_PARTICLE_TYPE, "z", HOFFSET(Particle, z), H5T_NATIVE_DOUBLE);
        H5Tinsert(MY_HDF5_PARTICLE_TYPE, "t", HOFFSET(Particle, t), H5T_NATIVE_DOUBLE);
        H5Tinsert(MY_HDF5_PARTICLE_TYPE, "dim", HOFFSET(Particle, dim), H5T_NATIVE_INT);
    

If you'd find this useful I can look into a way of letting users register their own structs and pass them to h5pp in some way. I'm thinking something along the lines of

std::vector<Particle> particles(10);
...
//register as above
...

myfile.writeDataset(particles, MY_HDF5_PARTICLE_TYPE,"particles");

Another option is to use HDF5 Tables which is specifically geared at time-series data in low-latency settings. h5pp doesn't currently support this, because the entries in the time-series have to be registered in the same way.

Also, check out todays massive update, although I suspect it would not address this particular issue.

[mlund]

Thanks for looking into this. I still do not understand why the m1 approach above works as Map should use a stride to access the POD for the positions, only?

[DavidAce]

I think I've found the reason. When writing Eigen types, these are automatically transformed to row-major (as wanted by HDF5) by coyping into a temporary object. In the case of Eigen::Map, the temporary object is actually a row-major instance of the wrapped type.
This "copy" probably just does the correct thing and honor's the stride.
The only way to avoid the copy is to either pass row-major eigen types, or simpy to pass the raw pointer + size, which of course will make the data look wrong in the file.

[DavidAce]

In any case, I have implemented the user-defined types as I mentioned yesterday. It turns out that I need this feature for my own project. More specifically I need to make tables of user-defined HDF5 compound types, so now I have implemented that too. Check out the new examples 5 and 6 in the wiki. I believe HDF5 tables could be a cleaner solution for time-series data.
The new features are in the master branch and will be available in v1.5.1

[DavidAce]

I have decided to close this issue for now. The library has support for Eigen::Map and Eigen::TensorMap, and I have added support for user-defined POD structs. However, non-contiguous containers and non-POD structs with dynamic data seems much harder. Honestly at the moment I don't know how to implement that or if it is even possible. It would be a nice feature though, so pointing me to an implementation or a pull-request is appreciated. You are welcome to open a new issue in that case.