vbrankov/hdf5-ocaml

OCaml wrapper for HDF5

hdf5-ocaml's Issues

super simple example

Hello,
Is it possible to use your bindings to implement a key-value store?
With both keys and values being strings.
Thanks!

project name hdf5 vs hdf5-ocaml

There seems to be a name clash. When upgrading hdf5-0.1.4 to hdf5-0.1.5 I get an error:
Error: I don't know about package hdf5 (passed through --only-packages/--release)

It looks like the opam package name has changed from hdf5 to hdf5-ocaml, as the following works:
opam pin add hdf5-ocaml <path/to/local/repo> while opam pin add hdf5 <path/to/local/repo> fails with the same message as above.

Did you intentionally rename it to hdf5-ocaml?

Don't use "%caml_string_*" special primitives with Bytes.t

use the bytes equivalent instead.

H5a.read_float does not work for HDF5 attributes in 32-bit float format

The C stub passes a pointer to a double. It should be pointer to float when 32-bit float.

value hdf5_h5a_read_float(value attr_v, value mem_type_v)
{
CAMLparam2(attr_v, mem_type_v);
double buf; <<<<<<
raise_if_fail(H5Aread(Hid_val(attr_v), Hid_val(mem_type_v), &buf));
CAMLreturn(caml_copy_double(buf));
}

hdf5 does not compile on mac OSX

Config: MAC OS X Sierra. Ocaml: 4.02.0 (also tried with 4.02.3)
Installed HDF5 lib by homebrew
Compilation error is:
ocaml setup.ml -build
/Users/username/.opam/4.02.0/bin/ocamlopt.opt unix.cmxa -I /Users/username/.opam/4.02.0/lib/ocaml/ocamlbuild /Users/username/.opam/4.02.0/lib/ocaml/ocamlbuild/ocamlbuildlib.cmxa myocamlbuild.ml /Users/username/.opam/4.02.0/lib/ocaml/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
/Users/username/.opam/4.02.0/bin/ocamlfind ocamlc -ccopt -I/usr/include/hdf5/serial -c lib/raw/h5a_stubs.c
mv h5a_stubs.o lib/raw/h5a_stubs.o
/Users/username/.opam/4.02.0/bin/ocamlfind ocamlc -ccopt -I/usr/include/hdf5/serial -c lib/raw/h5d_stubs.c
mv h5d_stubs.o lib/raw/h5d_stubs.o
/Users/username/.opam/4.02.0/bin/ocamlfind ocamlc -ccopt -I/usr/include/hdf5/serial -c lib/raw/h5f_stubs.c

• /Users/username/.opam/4.02.0/bin/ocamlfind ocamlc -ccopt -I/usr/include/hdf5/serial -c lib/raw/h5f_stubs.c
  lib/raw/h5f_stubs.c:123:8: error: no member named 'super_ext_size' in 'struct H5F_info2_t'
  info.super_ext_size = Int_val(Field(info_v, 0));

lib/raw/h5f_stubs.c:135:40: error: no member named 'super_ext_size' in 'struct H5F_info2_t'
Store_field(info_v, 0, Val_int(info->super_ext_size));
~~~~ ^
/Users/username/.opam/4.02.0/lib/ocaml/caml/mlvalues.h:75:29: note: expanded from macro 'Val_int'
#define Val_int(x) Val_long(x)
^
/Users/username/.opam/4.02.0/lib/ocaml/caml/mlvalues.h:71:36: note: expanded from macro 'Val_long'
#define Val_long(x) (((intnat)(x) << 1) + 1)
^
/Users/username/.opam/4.02.0/lib/ocaml/caml/memory.h:239:27: note: expanded from macro 'Store_field'
value caml__temp_val = (val);
^~~
2 errors generated.
Command exited with code 2.

HDF5 now requires PPX libraries?

When building with current HDF5 libraries, we seem to now require a Ppx_inline_test_lib Runtime module. Is this intentional? I'd guess that this is a development requirement that has slipped into the release.

best, Mark

File "_none_", line 1:
Error: No implementations provided for the following modules:
         Inline_test_config referenced from /home/mhayden/.opam/linux.dev/lib/hdf5/caml/hdf5_caml.cmxa(Hdf5_caml__Struct)
         Ppx_inline_test_lib__Runtime referenced from /home/mhayden/.opam/linux.dev/lib/hdf5/caml/hdf5_caml.cmxa(Hdf5_caml__Type),
           /home/mhayden/.opam/linux.dev/lib/hdf5/caml/hdf5_caml.cmxa(Hdf5_caml__Field),
           /home/mhayden/.opam/linux.dev/lib/hdf5/caml/hdf5_caml.cmxa(Hdf5_caml__H5),
           /home/mhayden/.opam/linux.dev/lib/hdf5/caml/hdf5_caml.cmxa(Hdf5_caml__Struct_intf),
           /home/mhayden/.opam/linux.dev/lib/hdf5/caml/hdf5_caml.cmxa(Hdf5_caml__Struct),
           /home/mhayden/.opam/linux.dev/lib/hdf5/caml/hdf5_caml.cmxa(Hdf5_caml)
Makefile:1168: recipe for target 'obj/seaiq_ml.o' failed
make: *** [obj/seaiq_ml.o] Error 2

read_attribute_string fails to read variable-length string

I have a file with an attribute which is a variable-length string:
http://vectors.erikbern.com/fashion-mnist-784-euclidean.hdf5 (sorry, 217 Mo)

The attribute is named "distance" and its value is "euclidean".

H5.read_attribute_string fails to read it (it returns garbage, which corresponds to an address ; when reading a vl string, H5Aread expects the third arg to be a pointer to a pointer, allocates the result on its side, and returns the address of the string).

(Is there a special function for reading vl string attributes that I have missed?)

My code is below:

open Base
open Stdio
open Hdf5_caml

module Dataset = struct
  type t = {
    train : (float, Bigarray.float32_elt, Bigarray.c_layout) Bigarray.Array2.t;
    test : (float, Bigarray.float32_elt, Bigarray.c_layout) Bigarray.Array2.t
  }

  let read f =
    let data = H5.open_rdonly ("../../../data/" ^ f) in
    printf "%s: %s\n" f (Sexp.to_string_hum @@ [%sexp_of : string list] @@ H5.ls data);
    let distance = H5.read_attribute_string data "distance" in
    let train = H5.Float32.read_float_array2 data "train" Bigarray.c_layout in
    let test = H5.Float32.read_float_array2 data "test" Bigarray.c_layout in
    printf "  train:%dx%d test: %dx%d distance: %s\n"
      (Bigarray.Array2.dim1 train)
      (Bigarray.Array2.dim2 train)
      (Bigarray.Array2.dim1 test)
      (Bigarray.Array2.dim2 test)
      distance;
    { train; test }
  ;;
end;;

Dataset.read "fashion-mnist-784-euclidean.hdf5";;

error with read_float_genarray

The following example uses Owl to create a Bigarray genarray (that's what Owl matrices are), write it to a file, and then read it back in. The example works perfectly in hdf5-ocaml v. 0.1.3, but not in the current version of hdf5-ocaml (with the addition of PR #11).

open Hdf5_caml

let () =
  let m = Owl.Mat.uniform 10 10 in
  let f = H5.create_trunc "foo.h5" in
  H5.write_float_genarray f "m" m;
  H5.close f;

  let f = H5.open_rdonly "foo.h5" in
  let m' = H5.read_float_genarray f "m" Bigarray.c_layout in
  H5.close f;
  Printf.printf "original data = data from file: %b\n" (m = m')

With the current version, this line

  let m' = H5.read_float_genarray f "m" Bigarray.c_layout in

causes these error messages, and the rest of the code isn't executed:

HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
  #000: H5Dio.c line 269 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 345 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dio.c line 638 in H5D__write(): no write intent on file
    major: Dataset
    minor: Write failed
Fatal error: exception Hdf5_raw.H5i.Fail

It's strange that the error messages are about writes, since it seems to be a read that causes the eeror. However, commenting out the read_float_genarray line gets rid of the errors, and afaics every other line seems to execute without problem.

There is no similar problem with write_float_array and read_float_array, btw. I have only seen the problem with read_float_genarray.

(I did some searching for similar errors on the web. There are quite a few cases in which these HDF5 errors have come up, but none of them seemed similar to this case. For example, some of the instances of these errors had to do with writing files of certain large sizes.)

weird segfault

I have a weird segfault

Program received signal SIGSEGV, Segmentation fault.
0x00005555555e9674 in caml_empty_minor_heap () at minor_gc.c:381

which occurs when

• I create a large bigarray, and
• I subsequently open an hdf5 file

In the code below, the segfault disappears if either

• I make the array size n smaller by one (found this threshold of 16_776_796 through bisection), OR
• I comment out the last 2 lines (i.e. if I don't open the hdf5 file)

let () =
  let open Bigarray in
  let open Hdf5_raw in
  let n = 16_776_796 in
  let a = Array1.create Float64 c_layout n in
  for i=0 to n-1 do a.{i} <- Random.float 1. done;
  let file = H5f.create "test.h5" [ H5f.Acc.TRUNC ] in
  H5f.close file

Any idea what might be going on?
Can anyone reproduce this? I have never had any issues with large Bigarrays so far, so I doubt this is a faulty-RAM-type-of-issue.

example project using the ppx extension

I was trying to locate a project using the ppx %%h5struct extension but couldn't find one. There should be an example oasis or jbuilder project demonstrating how to use it. If that is too much effort, perhaps README.md should make a note on how to add to it to jbuilder, or at least how to run the ppx extension manually on the command line.

Difficulties with Hdf5_caml

This continues a discussion that started in #3.

OK, I don't understand why I can't #require "hdf5_caml" in utop if caml_copy_string is supposed to be available at the top level; it's not. I've posted a question about on discuss.ocaml.org.

I copied a couple of the example files to a project and listed them in a jbuild file. I can compile examples/h5_intro/create_example.ml into a native executable that runs just fine, although there's a warning about the missing serial directory that I mentioned in #3. This example file uses Hdf5_raw.

I also copied examples/caml/test.ml to my project. This uses Hdf5_caml. When I add that to the jbuild file, it causes an error that I find very puzzling.

jbuild:

(jbuild_version 1)

(executables
 ((names (create_example test))
	(public_names (create_example test))
  (package yo)
  (libraries (bigarray hdf5_raw hdf5_caml))
 ))

(The version that worked with create_example.ml is the same, but with both instances of "test" deleted and with "hdf5_caml" deleted.)

The error:

$ make
jbuilder build @install
      ocamlc src/yo/test.{cmi,cmo,cmt}
      ocamlc src/yo/create_example.{cmi,cmo,cmt}
    ocamlopt src/yo/test.{cmx,o}
    ocamlopt src/yo/test.exe (exit 2)
(cd _build/default && /Users/marshall/.opam/4.05.0+flambda/bin/ocamlopt.opt -w -40 -g -o src/yo/test.exe -I /Users/marshall/.opam/4.05.0+flambda/lib/hdf5_caml -I /Users/marshall/.opam/4.05.0+flambda/lib/hdf5_raw -I /Users/marshall/.opam/4.05.0+flambda/lib/ocaml /Users/marshall/.opam/4.05.0+flambda/lib/ocaml/unix.cmxa /Users/marshall/.opam/4.05.0+flambda/lib/hdf5_raw/hdf5_raw.cmxa /Users/marshall/.opam/4.05.0+flambda/lib/ocaml/bigarray.cmxa /Users/marshall/.opam/4.05.0+flambda/lib/hdf5_caml/hdf5_caml.cmxa src/yo/test.cmx)
File "_none_", line 1:
Error: No implementations provided for the following modules:
         Bigarray referenced from /Users/marshall/.opam/4.05.0+flambda/lib/hdf5_raw/hdf5_raw.cmxa(Hdf5_raw)
make: *** [all] Error 1

This seems strange because test.ml makes no reference to Bigarray, and I am including it in the libraries entry of jbuild.

Sorry to raise all of these issues at once. I'll keep working on them, but was wondering whether you had any insight. Thanks.

directory -> group/path

great to see that hdf5 support is finally coming to ocaml!

on cursory reading of the highlevel api and docs i often see the term 'directory'. while this is of course intuitive, the standard terminology is path if it's the string or 'group' (and/or 'dataset') if it's the object i believe.

https://www.hdfgroup.org/HDF5/doc/Glossary.html#Glossary-Group

i know hdf5 via h5py which is a joy to use. this is of course based on numpy and heavily uses overloading of slicing syntax, which is probably not a good match for ocaml. but there might be other aspects where it could serve as inspiration for the api.

https://github.com/h5py/h5py/tree/master/h5py/_hl

some ideas for the highlevel interface (may be nonsense and/or already implemented, sorry if so.)

• i would have expected a single file creation function with some variant argument for the file mode. but i guess that's taste?
• would it be good to provide a 'with file' construct/function which would work like a context manager in python?
• fold/iter/map functions over hdf5 groups? (this may be nonsense since the links in an hdf5 file can be duplicate and circular as i just found out)
• hdf5 capabilities depend on the format version a lot. highlevel verification of version requirements for opened files?

disclaimer: have not actually used, only looked at the .mli's

required hdf version?

i have problems loading hdf5-ocaml from utop. i have hdf5.10 installed. i get

#require "hdf5_caml";;
Error: The external function `hdf5_h5tb_write_records_bytecords' is not available

this is after a reinstall of hdf5-ocaml via opam. before i got the message

cannot load required shared library dllhdf5_raw_stubs.
  Reason: /Users/<USER>/.opam/4.02.3/lib/stublibs/dllhdf5_raw_stubs.so:
dlopen(/Users/<USER>/.opam/4.02.3/lib/stublibs/dllhdf5_raw_stubs.so, 138):
Library not loaded: /usr/local/lib/libhdf5.9.dylib Referenced from:
/Users/<USER>/.opam/4.02.3/lib/stublibs/dllhdf5_raw_stubs.so
  Reason: image not found.
Error: The external function `hdf5_h5l_exists' is not available

HDF5 now requires PPX libraries? (Re-opened)

[This issue was closed before I was able to respond to previous comment. I've re-opened and included additional detail below.]

When building with current HDF5 libraries, we seem to now require a Ppx_inline_test_lib Runtime module. Is this intentional? I'd guess that this is a development requirement that has slipped into the release.

best, Mark

Sorry for not replying sooner.

Short answer: Unless I'm missing something, YES!!!! these are causing lots of problems. Long answer is below.

This change draws in what seems like a massive number of other files (see portion of Makefile included at the bottom of this comment). I just finished working through the dependencies-of-dependencies-of-dependencies-of-dependencies. Using ocaml hdf5 now requires all the libraries listed below just to use read some arrays from HDF5 files. Also, my software makes use of Path module which has a name conflict with one of the modules in one of the many libraries (maybe there are more of these conflicts?).

Please understand that I distribute my software to be used on mobile devices which are cross-compiled. With this change, I now need to distribute many additional libraries and increase the size of our executables (downloaded over cellular connections). Also, I now need to make sure these libraries cross-compile for Windows, IOS, and Android... a massive undertaking that could easily take 1 month of effort just devoted to this... and then would need to be fixed every time there is a change/update to these libraries. This is all on top of the general concern of having a ton of additional software included that we don't know much about.

I hope this all makes sense. If there is some way these dependencies can be removed, that would be extremely helpful if not essential to our continued use of your otherwise very useful library. I think if there is going to be an isolated use that triggers dependencies on so many things that it might make sense to split that off as an optional HDF library.

`HDF5_INCLUDES =
-I $(OCAMLLIB)/compiler-libs
-I $(OCAMLSYSLIB)/ocaml-compiler-libs/shadow
-I $(OCAMLSYSLIB)/ocaml-migrate-parsetree
-I $(OCAMLSYSLIB)/base
-I $(OCAMLSYSLIB)/base/caml
-I $(OCAMLSYSLIB)/base/shadow_stdlib
-I $(OCAMLSYSLIB)/stdio
-I $(OCAMLSYSLIB)/sexplib0
-I $(OCAMLSYSLIB)/ppxlib
-I $(OCAMLSYSLIB)/ppxlib/ast
-I $(OCAMLSYSLIB)/ppxlib/print_diff
-I $(OCAMLSYSLIB)/ppxlib/traverse
-I $(OCAMLSYSLIB)/ppxlib/traverse_builtins
-I $(OCAMLSYSLIB)/ppx_derivers
-I $(OCAMLSYSLIB)/ppx_inline_test
-I $(OCAMLSYSLIB)/ppx_inline_test/config
-I $(OCAMLSYSLIB)/ppx_inline_test/runtime-lib
-I $(OCAMLSYSLIB)/ppx_inline_test/libname
-I $(OCAMLSYSLIB)/hdf5/raw
-I $(OCAMLSYSLIB)/hdf5/caml

LIBS_HDF5 =
ocaml_shadow.cma
ocamlcommon.cma
ocamloptcomp.cma
sexplib0.cma
caml.cma
shadow_stdlib.cma
base.cma
stdio.cma
ppx_derivers.cma
migrate_parsetree.cma
ppxlib_traverse_builtins.cma
ppxlib_ast.cma
ppxlib_print_diff.cma
ppxlib.cma
ppxlib_traverse.cma
ppx_inline_test_libname.cma
ppx_inline_test.cma
ppx_inline_test_lib.cma
inline_test_config.cma
hdf5_raw.cma
hdf5_caml.cma
`

Dependency issue

Hi thanks for this work. I just find that the lib libhdf5-dev is required before installation. So I was thinking maybe it might be helpful to add a line of dependency in the readme file?

'hdf5.h' file not found

With OCaml 4.05.0+flambda, I get this error trying to install hdf5-ocaml with opam:

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[ERROR] The compilation of hdf5 failed at "make".

#=== ERROR while compiling hdf5.0.1.4 =========================================#
# context      2.0.0~beta4 | darwin/x86_64 | ocaml-variants.4.05.0+flambda | https://opam.ocaml.org/2.0#2017-12-06 19:04
# path         ~/.opam/4.05.0+flambda/.opam-switch/build/hdf5.0.1.4
# command      make
# exit-code    2
# env-file     ~/.opam/log/hdf5-32892-05296d.env
# output-file  ~/.opam/log/hdf5-32892-05296d.out
### output ###
# [...]
# ocaml setup.ml -build
# ocamlfind ocamlopt -package unix -package ocamlbuild -linkpkg -package cppo_ocamlbuild myocamlbuild.ml /Users/marshall/.opam/4.05.0+flambda/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
# /Users/marshall/.opam/4.05.0+flambda/bin/ocamlfind ocamlc -g -ccopt -I/usr/include/hdf5/serial -c lib/raw/h5a_stubs.c
# + /Users/marshall/.opam/4.05.0+flambda/bin/ocamlfind ocamlc -g -ccopt -I/usr/include/hdf5/serial -c lib/raw/h5a_stubs.c
# In file included from lib/raw/h5a_stubs.c:9:
# lib/raw/hdf5_caml.h:2:10: fatal error: 'hdf5.h' file not found
# #include "hdf5.h"
#          ^
# 1 error generated.
# Command exited with code 2.
# E: Failure("Command ''/Users/marshall/.opam/4.05.0+flambda/bin/ocamlbuild' lib/raw/libhdf5_raw_stubs.a lib/raw/dllhdf5_raw_stubs.so lib/raw/hdf5_raw.cma lib/raw/hdf5_raw.cmxa lib/raw/hdf5_raw.a lib/raw/hdf5_raw.cmxs lib/ppx/ppx_h5struct.native lib/caml/hdf5_caml.cma lib/caml/hdf5_caml.cmxa lib/caml/hdf5_caml.a lib/caml/hdf5_caml.cmxs -use-ocamlfind -plugin-tags 'package(cppo_ocamlbuild)' -tag[...]
# make: *** [build] Error 1

Maybe this is the same problem as #8? It's not clear to me what it is I should install, in that case. libhdf5-dev is an Ubuntu package, it seems. I'm on MacOS. I installed the only hdf5-related package (hdf5) that's available in homebrew. I can install something else from source, but I'm not sure what's missing.

I'll try installing hdf5-ocaml from source later today.

Does not build for ocaml 4.13 or later

This does not appear to build for Ocaml 4.13 or later. Output below is for Ocaml 4.14.

• opam install hdf5
  [ERROR] Package conflict!

• No agreement on the version of ocaml:
  • (invariant) -> ocaml-variants = 4.14.0+options -> ocaml = 4.14.0
  • hdf5 -> ocaml < 4.03.0
    You can temporarily relax the switch invariant with `--update-invariant'
• Missing dependency:
  • hdf5 -> ppx_tools_versioned -> ocaml-migrate-parsetree < 2.0.0 ->
    ocaml-variants = 4.08.0+beta2 -> ocaml-beta
    unmet availability conditions: 'enable-ocaml-beta-repository'
• Missing dependency:
  • hdf5 -> ppx_tools_versioned -> ocaml-migrate-parsetree < 2.0.0 ->
    ocaml-variants = 4.08.0+beta3 -> ocaml-beta
    unmet availability conditions: 'enable-ocaml-beta-repository'

No solution found, exiting

Remove caml_copy_string so Hdf5_caml can be used in bytecode?

I think it's valuable to be able to use Hdf5_caml in utop, which was possible in 0.1.3. The current use of caml_copy_string in 0.1.4 prevents this, apparently. I believe this is because caml_copy_string is unavailable in bytecode, since I also get an error concerning caml_copy_string when I try to compile a standalone bytecode executable:

jbuilder build @install
      ocamlc src/yo/test.bc (exit 2)
(cd _build/default && /Users/mabrams/.opam/4.05.0+flambda/bin/ocamlc.opt -w -40 -g -o src/yo/test.bc -I /Users/mabrams/.opam/4.05.0+flambda/lib/bytes -I /Users/mabrams/.opam/4.05.0+flambda/lib/ctypes -I /Users/mabrams/.opam/4.05.0+flambda/lib/eigen -I /Users/mabrams/.opam/4.05.0+flambda/lib/gsl -I /Users/mabrams/.opam/4.05.0+flambda/lib/hdf5_caml -I /Users/mabrams/.opam/4.05.0+flambda/lib/hdf5_raw -I /Users/mabrams/.opam/4.05.0+flambda/lib/integers -I /Users/mabrams/.opam/4.05.0+flambda/lib/ocaml -I /Users/mabrams/.opam/4.05.0+flambda/lib/owl -I /Users/mabrams/.opam/4.05.0+flambda/lib/plplot /Users/mabrams/.opam/4.05.0+flambda/lib/ocaml/unix.cma /Users/mabrams/.opam/4.05.0+flambda/lib/ocaml/bigarray.cma /Users/mabrams/.opam/4.05.0+flambda/lib/integers/integers.cma /Users/mabrams/.opam/4.05.0+flambda/lib/ctypes/ctypes.cma /Users/mabrams/.opam/4.05.0+flambda/lib/ocaml/str.cma /Users/mabrams/.opam/4.05.0+flambda/lib/ctypes/cstubs.cma /Users/mabrams/.opam/4.05.0+flambda/lib/eigen/eigen.cma /Users/mabrams/.opam/4.05.0+flambda/lib/gsl/gsl.cma /Users/mabrams/.opam/4.05.0+flambda/lib/plplot/plplot.cma /Users/mabrams/.opam/4.05.0+flambda/lib/hdf5_raw/hdf5_raw.cma /Users/mabrams/.opam/4.05.0+flambda/lib/owl/owl.cma /Users/mabrams/.opam/4.05.0+flambda/lib/hdf5_caml/hdf5_caml.cma src/yo/test.cmo)
File "_none_", line 1:
Error: Error while linking /Users/mabrams/.opam/4.05.0+flambda/lib/hdf5_caml/hdf5_caml.cma(Hdf5_caml):
The external function `caml_copy_string' is not available

I'm hoping that in a future version of hdf5-ocaml, some other way of accomplishing the goal for which calm_copy_string can be developed which will allow Hdf5_caml to be used in toplevels.

(caml_copy_string is mentioned in #9 and #10, but those remarks are mixed in with other topics.)