drup/ocaml-lmdb

Ocaml bindings for lmdb.

OCaml-lmdb

The LMDB database is a fast in-file database that supports ACID transactions.

These bindings expose a typesafe yet low-overhead API. Both transactions and cursors are available. Database implementations are specialized both by keys and values. Two module are predefined: Lmdb.Db (string keys and string values) and Lmdb.IntDb (int keys and string values). New implementation (which can use special LMDB features such as multi-values) can be added via a functorial interface.

Please consult the documentation and a simple example.

let open Lmdb in
let env = Env.(create Rw ~flags:Flags.no_subdir ~max_maps:1) "mydb" in
let map = Map.create Nodup ~key:Conv.string ~value:Conv.string
	    ~name:"Camelidae" env in
Map.add map "Bactrian camel" "Elegant and beautiful animal with two humps."

ocaml-lmdb's Issues

FreeBSD compatibility

Not sure if it's really an issue but to install the opam package on FreeBSD 12 i had to explicitly set C_INCLUDE_PATH and LIBRARY_PATH:

setenv C_INCLUDE_PATH /usr/local/include/
setenv LIBRARY_PATH /usr/local/lib/

May be there is a standard way with opam or dune to avoid this manual setting?

Consider updating the opam package.

The api has changed so much that a new opam release might be necessary.

Dunify

Opam doesn't install package correctly

I've installed the Lmdb package via Opam and the build goes find, but I can't actually open up the Lmdb package in the repl or when compiling a source file.

A type puzzle with permissions.

Here is a nice puzzle for type apprentices: Figure out a signature of M so that this works

module rec M : sig
  type +'a cap
  type r = [`R]
  type w = [`W]
  val r : r cap
  val rw : 'any cap

  type +'a env
  val create : 'a cap -> 'a env

  val read : _ env -> int
  val write : w env -> int -> unit

  type +'a txn
  val txn : ?c:'a cap -> 'a env -> ('a txn -> 'r) -> 'r
  val readT : _ txn -> int
  val writeT : w txn -> int -> unit

end = struct

  type r = M.r
  type w = M.w

  type _ cap = C
  let r = C
  let rw = C

  type _ env = E
  let create C = E

  let read E = 1
  let write E _ = ()
   
  type _ txn = T
  let txn ?c:_ E f = f T
  let readT T = 1
  let writeT T _ = ()
end
open M
open M


let envr = create r
let envw = create rw

let i = read envr
let i2 = read envw

let () = write envw 2
let () = write envr 1 (* MUST FAIL *)

let _ = txn envw @@ fun t -> readT t
let _ = txn envw @@ fun t -> writeT t

let _ = txn envr @@ fun t -> readT t
let _ = txn envr @@ fun t -> writeT t (* MUST FAIL *)

let _ = txn ~c:r envw @@ fun t -> readT t
let _ = txn ~c:r envw @@ fun t -> writeT t (* MUST FAIL *)

The current version works, but as the downside that this doesn't:

let f env =
  let i = txn ~c:r env @@ fun t -> readT t in
  write env i

Add some conditional compilation to handle different lmdb versions

Lmdb.Values.db_val is opaque

Lmdb.Values.db_val is opaque and thus no module of type Lmdb.Values.S can be constructed.

not in opam?!

What is the show stopper?
Is it not production ready?

Non-declared dependency on conf-pkg-config

lmdb implicitly (via dune-configurator) uses (at least on Linux) pkg-config to discover lmdb.h. However, this dependency is not explicitly defined in depexts. This means that if pkg-config happens to not be present, the lmdb package can't build.

Error is as follows:

File "dune", line 6, characters 0-70:
6 | (rule
7 |  (targets cflags.sexp clibs.sexp)
8 |  (action (run ./discover.bc)))
(cd _build/default && ./discover.bc)
which: pkg-config
-> not found
Fatal error: exception Failure("lmdb.h not found")

With pkg-config installed, the package seems to build correctly.

Segfault when using Cursor.go

The following program segfaults in the middle of the cursor transaction. The resulting database can be dumped using the md_dump program with no issues. Pinned to master, on macOS, lmdb version 0.9.23:

let () =
  (* Create env and db *)
  let kibi = 8 * 1024 in
  let mibi = kibi * 1024 in
  let env = Lmdb.Env.create ~map_size:(10 * mibi) ~max_dbs:1 "testenv" in
  let t = Lmdb.Db.create ~create:true env "testdb" in
  (* Put some entries *)
  let rec put_count t = function
    | 0 -> ()
    | count ->
        let value_bytes = Bytes.make (10 * kibi) '1' in
        Lmdb.Db.put t (string_of_int count) (Bytes.to_string value_bytes) ;
        put_count t (count - 1)
  in
  let count = 250 in
  put_count t count ;
  assert ((Lmdb.Db.stats t).entries = count) ;
  (* Iterate using cursor and print keys *)
  Lmdb.Db.Cursor.go Lmdb.ro t (fun cur ->
      (* Triggering GC here also SEGFAULTs *) 
      (* Gc.full_major () ; *)
      let rec print_keys = function
        | 0 -> ()
        | count ->
            let key, _ = Lmdb.Db.Cursor.next cur in
            print_endline key ;
            print_keys (count - 1)
      in
      print_keys count )

Note: GC segfaulting in the code above.

I tried to get to the bottom of it, but wasn't successful.

My best guess is that the OCaml GC attempts to free the memory pointed to by the Bigarray created here:

ocaml-lmdb/src/lmdb.ml

Line 373 in 2e163ad

(* no need to keep a reference to mvp here, because the memory the bigarray

Which would violate http://www.lmdb.tech/doc/group__mdb.html#ga8bf10cd91d3f3a83a34d04ce6b07992d
As a hack, I've tried adding all values returned from this function to a list ref to prevent them from being collected, that didn't help though...