Comments (5)
I'm happy to take a look.
Currently h5pp supports contiguous containers such as std::array, c-style arrays, std::vectors or Eigen (or Eigen::Map!) with POD data. As far as I can tell, there is no way to tell HDF5 to write a non-contiguous block of data (i.e. to specify a stride), but I'm still learning. I have decided not to support ranges because not all STL containers are contiguous, like std::list. I think offering the possibility would send mixed messages: I would either have to copy the list data to a contiguous vector, or write elements one by one to individual datasets, but these are decisions a user should make instead. So I decided to simply check if the container has a ".data()" method - contiguous containers usually do.
However, going back to your problem, it is possible to write entire POD structs, or (contiguous) containers thereof. It's just a matter of registering and defining a compound HDF5 type before writing it. In fact, I do this at initialization to enable support for std::complex
, and CUDA structs like struct double2{double x,y;}
. Code is found in include/h5pp/details/h5ppTypeCompoundCreate.h
So for instance, a struct like,
struct Particle {
double x,y,z,t;
int dim;
}
is registered in HDF5 with
// hid_t can be replaced by a safer h5pp::hid::h5t below,
hid_t MY_HDF5_PARTICLE_TYPE = H5Tcreate(H5T_COMPOUND, sizeof(Particle));
H5Tinsert(MY_HDF5_PARTICLE_TYPE, "x", HOFFSET(Particle, x), H5T_NATIVE_DOUBLE);
H5Tinsert(MY_HDF5_PARTICLE_TYPE, "y", HOFFSET(Particle, y), H5T_NATIVE_DOUBLE);
H5Tinsert(MY_HDF5_PARTICLE_TYPE, "z", HOFFSET(Particle, z), H5T_NATIVE_DOUBLE);
H5Tinsert(MY_HDF5_PARTICLE_TYPE, "t", HOFFSET(Particle, t), H5T_NATIVE_DOUBLE);
H5Tinsert(MY_HDF5_PARTICLE_TYPE, "dim", HOFFSET(Particle, dim), H5T_NATIVE_INT);
If you'd find this useful I can look into a way of letting users register their own structs and pass them to h5pp in some way. I'm thinking something along the lines of
std::vector<Particle> particles(10);
...
//register as above
...
myfile.writeDataset(particles, MY_HDF5_PARTICLE_TYPE,"particles");
Another option is to use HDF5 Tables which is specifically geared at time-series data in low-latency settings. h5pp doesn't currently support this, because the entries in the time-series have to be registered in the same way.
Also, check out todays massive update, although I suspect it would not address this particular issue.
from h5pp.
Thanks for looking into this. I still do not understand why the m1
approach above works as Map should use a stride to access the POD for the positions, only?
from h5pp.
I think I've found the reason. When writing Eigen types, these are automatically transformed to row-major (as wanted by HDF5) by coyping into a temporary object. In the case of Eigen::Map, the temporary object is actually a row-major instance of the wrapped type.
This "copy" probably just does the correct thing and honor's the stride.
The only way to avoid the copy is to either pass row-major eigen types, or simpy to pass the raw pointer + size, which of course will make the data look wrong in the file.
from h5pp.
In any case, I have implemented the user-defined types as I mentioned yesterday. It turns out that I need this feature for my own project. More specifically I need to make tables of user-defined HDF5 compound types, so now I have implemented that too. Check out the new examples 5 and 6 in the wiki. I believe HDF5 tables could be a cleaner solution for time-series data.
The new features are in the master branch and will be available in v1.5.1
from h5pp.
I have decided to close this issue for now. The library has support for Eigen::Map and Eigen::TensorMap, and I have added support for user-defined POD structs. However, non-contiguous containers and non-POD structs with dynamic data seems much harder. Honestly at the moment I don't know how to implement that or if it is even possible. It would be a nice feature though, so pointing me to an implementation or a pull-request is appreciated. You are welcome to open a new issue in that case.
from h5pp.
Related Issues (16)
- write fails for custom structs of length >3 HOT 3
- h5pp compatibility issues with hdf5 1.13 HOT 3
- Some issues with H5PP on Mac HOT 8
- Cmake error while building h5pp HOT 1
- How to write dateaset with struct and std::string HOT 2
- `std::is_pod_v` deprecated in C++20 HOT 1
- Overwrite cmake flag CMAKE_CXX_STANDARD HOT 1
- Cmake 3.16: Configuration error in find_package(h5pp REQUIRED) HOT 2
- Compilation Failure on M1 MacOS due to Missing -lrt and -lsz HOT 2
- Add example of how to read or write a slice of a dataset HOT 4
- Add readDataset() overload that takes hid::h5t. HOT 4
- How to config the headers in QtCreator? HOT 3
- h5pp without CMake (HDF5 windows binary installer) HOT 1
- How to traverse a hdf5 file with h5pp? HOT 1
- Can this library be use in .net enviroment? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from h5pp.