bmatsuo/lmdb-go

Bindings for the LMDB C library

#lmdb-go

Go bindings to the OpenLDAP Lightning Memory-Mapped Database (LMDB).

Packages

Functionality is logically divided into several packages. Applications will usually need to import lmdb but may import other packages on an as needed basis.

Packages in the exp/ directory are not stable and may change without warning. That said, they are generally usable if application dependencies are managed and pinned by tag/commit.

Developers concerned with package stability should consult the documentation.

####lmdb

import "github.com/bmatsuo/lmdb-go/lmdb"

Core bindings allowing low-level access to LMDB.

####lmdbscan

import "github.com/bmatsuo/lmdb-go/lmdbscan"

A utility package for scanning database ranges. The API is inspired by bufio.Scanner and the python cursor implementation.

####exp/lmdbpool

import "github.com/bmatsuo/lmdb-go/exp/lmdbpool"

A utility package which facilitates reuse of lmdb.Txn objects using a sync.Pool. Naively storing lmdb.Txn objects in sync.Pool can be troublesome. And the lmdbpool.TxnPool type has been defined as a complete pooling solution and as reference for applications attempting to write their own pooling implementation.

The lmdbpool package is relatively new. But it has a lot of potential utility. And once the lmdbpool API has been ironed out, and the implementation hardened through use by real applications it can be integrated directly into the lmdb package for more transparent integration. Please test this package and provide feedback to speed this process up.

####exp/lmdbsync

import "github.com/bmatsuo/lmdb-go/exp/lmdbsync"

An experimental utility package that provides synchronization necessary to change an environment's map size after initialization. The package provides error handlers to automatically manage database size and retry failed transactions.

The lmdbsync package is usable but the implementation of Handlers are unstable and may change in incompatible ways without notice. The use cases of dynamic map sizes and multiprocessing are niche and the package requires much more development driven by practical feedback before the Handler API and the provided implementations can be considered stable.

Key Features

###Idiomatic API

API inspired by BoltDB with automatic commit/rollback of transactions. The goal of lmdb-go is to provide idiomatic database interactions without compromising the flexibility of the C API.

NOTE: While the lmdb package tries hard to make LMDB as easy to use as possible there are compromises, gotchas, and caveats that application developers must be aware of when relying on LMDB to store their data. All users are encouraged to fully read the documentation so they are aware of these caveats.

Where the lmdb package and its implementation decisions do not meet the needs of application developers in terms of safety or operational use the lmdbsync package has been designed to wrap lmdb and safely fill in additional functionality. Consult the documentation for more information about the lmdbsync package.

###API coverage

The lmdb-go project aims for complete coverage of the LMDB C API (within reason). Some notable features and optimizations that are supported:

• Idiomatic subtransactions ("sub-updates") that allow the batching of updates.

• Batch IO on databases utilizing the MDB_DUPSORT and MDB_DUPFIXED flags.

• Reserved writes than can save in memory copies converting/buffering into []byte.

For tracking purposes a list of unsupported features is kept in an issue.

###Zero-copy reads

Applications with high performance requirements can opt-in to fast, zero-copy reads at the cost of runtime safety. Zero-copy behavior is specified at the transaction level to reduce instrumentation overhead.

err := lmdb.View(func(txn *lmdb.Txn) error {
    // RawRead enables zero-copy behavior with some serious caveats.
    // Read the documentation carefully before using.
    txn.RawRead = true

    val, err := txn.Get(dbi, []byte("largevalue"), 0)
    // ...
})

###Documentation

Comprehensive documentation and examples are provided to demonstrate safe usage of lmdb. In addition to godoc documentation, implementations of the standand LMDB commands (mdb_stat, etc) can be found in the cmd/ directory and some simple experimental commands can be found in the exp/cmd/ directory. Aside from providing minor utility these programs are provided as examples of lmdb in practice.

##LMDB compared to BoltDB

BoltDB is a quality database with a design similar to LMDB. Both store key-value data in a file and provide ACID transactions. So there are often questions of why to use one database or the other.

###Advantages of BoltDB

• Nested databases allow for hierarchical data organization.

• Far more databases can be accessed concurrently.

• Operating systems that do not support sparse files do not use up excessive space due to a large pre-allocation of file space. The exp/lmdbsync package is intended to resolve this problem with LMDB but it is not ready.

• As a pure Go package bolt can be easily cross-compiled using the go toolchain and GOOS/GOARCH variables.

• Its simpler design and implementation in pure Go mean it is free of many caveats and gotchas which are present using the lmdb package. For more information about caveats with the lmdb package, consult its documentation.

###Advantages of LMDB

• Keys can contain multiple values using the DupSort flag.

• Updates can have sub-updates for atomic batching of changes.

• Databases typically remain open for the application lifetime. This limits the number of concurrently accessible databases. But, this minimizes the overhead of database accesses and typically produces cleaner code than an equivalent BoltDB implementation.

• Significantly faster than BoltDB. The raw speed of LMDB easily surpasses BoltDB. Additionally, LMDB provides optimizations ranging from safe, feature-specific optimizations to generally unsafe, extremely situational ones. Applications are free to enable any optimizations that fit their data, access, and reliability models.

• LMDB allows multiple applications to access a database simultaneously. Updates from concurrent processes are synchronized using a database lock file.

• As a C library, applications in any language can interact with LMDB databases. Mission critical Go applications can use a database while Python scripts perform analysis on the side.

##Build

There is no dependency on shared libraries. So most users can simply install using go get.

go get github.com/bmatsuo/lmdb-go/lmdb

On FreeBSD 10, you must explicitly set CC (otherwise it will fail with a cryptic error), for example:

CC=clang go test -v ./...

Building commands and running tests can be done with go or with make

make bin
make test
make check
make all

On Linux, you can specify the pwritev build tag to reduce the number of syscalls required when committing a transaction. In your own package you can then do

go build -tags pwritev .

to enable the optimisation.

##Documentation

###Go doc

The go doc documentation available on godoc.org is the primary source of developer documentation for lmdb-go. It provides an overview of the API with a lot of usage examples. Where necessary the documentation points out differences between the semantics of methods and their C counterparts.

###LMDB

The LMDB homepage and mailing list (archives) are the official source of documentation regarding low-level LMDB operation and internals.

Along with an API reference LMDB provides a high-level summary of the library. While lmdb-go abstracts many of the thread and transaction details by default the rest of the guide is still useful to compare with go doc.

###Versioning and Stability

The lmdb-go project makes regular releases with IDs X.Y.Z. All packages outside of the exp/ directory are considered stable and adhere to the guidelines of semantic versioning.

Experimental packages (those packages in exp/) are not required to adhere to semantic versioning. However packages specifically declared to merely be "unstable" can be relied on more for long term use with less concern.

The API of an unstable package may change in subtle ways between minor release versions. But deprecations will be indicated at least one release in advance and all functionality will remain available through some method.

##License

Except where otherwise noted files in the lmdb-go project are licensed under the BSD 3-clause open source license.

The LMDB C source is licensed under the OpenLDAP Public License.

##Links

####github.com/bmatsuo/raft-mdb (godoc)

An experimental backend for github.com/hashicorp/raft forked from github.com/hashicorp/raft-mdb.

####github.com/bmatsuo/cayley/graph/lmdb (godoc)

Experimental backend quad-store for github.com/google/cayley based off of the BoltDB implementation.

lmdb-go's Issues

all: Drop support for go1.3 and earlier

The .travis.yml specifies that tests should be run for go1.2 and go1.3. This is somewhat anachronistic. And lately travis seems to be flaking out occasionally on some of the tests. Reducing the number of go versions that we are testing against well help determine if there is actually a problem in a realistic environment.

It still seems relevant to support go1.4. My understanding is that some some developers have continued to use it because of various temporary impediments with go1.5 (and tip?).

env.Update don't block other goroutine's env.Update?

I my mind, I thought write txn is something like exclusive lock, it'll block other threads/processes to write db, but in my code, before a write txn exit, another goroutine get
a write txn, I don't know why. Below is my code:

func (t *lmdbTopic) persistMessages(msgs []*Message) {
    isFull := false
    err := t.queueEnv.Update(func(txn *lmdb.Txn) error {
        log.Printf("In persistMessages, gid: %d get Update Txn", GoID())
        offset := t.persistedOffset(txn)
        log.Printf("In persistMessages, gid: %d, offset: %d", GoID(), offset)
        offset, err := t.persistToPartitionDB(offset, msgs)
        if err == nil {
            log.Printf("In persistMessages, gid: %d, update offset: %d", GoID(), offset)
            t.updatePersistOffset(txn, offset)
            log.Printf("In persistMessages, gid: %d will release Update Txn", GoID())
            return nil
        }
        return err
    })
    if err == nil {
        return
    }
    if lmdb.IsMapFull(err) {
        isFull = true
    } else {
        panic(err)
    }
    if isFull {
        log.Printf("In persistMessages, gid: %d will rotate persist partititon", GoID())
        t.rotatePersistPartition()
        t.persistMessages(msgs)
    }
}

Below is my log:

2016/08/31 16:23:56 In persistMessages, gid: 152 get Update Txn, t.queueEnv's addr: 0xc420356100
2016/08/31 16:23:56 In persistMessages, gid: 152, offset: 27394
2016/08/31 16:23:56 In persistMessages, gid: 152, update offset: 27395
2016/08/31 16:23:56 In persistMessages, gid: 152 will release Update Txn
2016/08/31 16:23:56 In persistMessages, gid: 152 get Update Txn, t.queueEnv's addr: 0xc420356100
2016/08/31 16:23:56 In persistMessages, gid: 152, offset: 27395
2016/08/31 16:23:56 In persistMessages, gid: 40 get Update Txn, t.queueEnv's addr: 0xc420356110
2016/08/31 16:23:56 In persistMessages, gid: 152, update offset: 27396
2016/08/31 16:23:56 In persistMessages, gid: 40, offset: 27395
2016/08/31 16:23:56 In persistMessages, gid: 152 will release Update Txn
panic: mdb_put: MDB_KEYEXIST: Key/data pair already exists

From log I see goroutine 40 call queueEnv.Update before goroutine 152 exit queueEnv.Update.

lmdb: Reduce number of heap allocations of MDB_val

I'm currently looking at allocation counts in the codebase I am working on, and I noticed that lmdb-go allocates short lived MDB_val objects, e.g. on every call to Txn.Get.

Would it be possible / safe to cache one or two MDB_val in Txn, and re-use them in Txn.Get and friends? I think there might be an issue with RawRead Txns, but I'm not sure I understand the code base sufficiently.

If this sounds like it could work I'd be happy to contribute a PR. It is not a straight up performance win, but it should reduce the pressure on the GC when querying lots and lots.

lmdb: 0.9.17 has been released

LMDB/lmdb@2011eb3 on the mdb.RE/0.9 branch.

Update LMDB to 0.9.19

A new version of LMDB has been officially tagged on 2016-12-28.

Should I call runtime.SetFinalizer(qe.env, qe.env.Close) in my NewQEnv?

Execute me, @bmatsuo
Below is my code:

type QEnvOpt struct {
    maxTopicNum  int
    topicMapSize int64
}

type QEnv struct {
    mu       sync.Mutex
    root     *string
    env      *lmdb.Env
    topicMap map[string]*QTopic
}

func NewQEnv(root *string, opt *QEnvOpt) (*QEnv, error) {
    env, err := lmdb.NewEnv()
    if err != nil {
        return nil, err
    }
    qe := new(QEnv)
    qe.root = root
    qe.env = env
    if opt != nil {
        qe.env.SetMapSize(opt.topicMapSize)
        qe.env.SetMaxDBs(opt.maxTopicNum)
    } else {
        qe.env.SetMapSize(DEFAULT_TOPIC_MAP_SIZE)
        qe.env.SetMaxDBs(DEFALUT_MAX_TOPOC_NUM)
    }
    err = qe.env.Open(*qe.root+ENV_META_NAME, lmdb.NoSync|lmdb.NoSubdir, 0644)
    if err != nil {
        qe.env.Close()
        return nil, err
    }
    dead, err := qe.env.ReaderCheck()
    if err != nil {
        return nil, err
    }
    log.Println("ReaderCheck clears: ", dead)
    runtime.SetFinalizer(qe.env, qe.env.Close)
    return qe, nil
}

I'm not sure if I need call

runtime.SetFinalizer(qe.env, qe.env.Close)

and when golang gc, will golang collect the pointers in a struct object?

type QEnv struct {
    mu       sync.Mutex
    root     *string
    env      *lmdb.Env
    topicMap map[string]*QTopic
}

lmdb: finalizer for Txn

Relates to #20

A finalizer for Txn should only be required on unmanaged Txn types. Because managed Txn types have
a deferred Abort operation their resources will always be released.

Adequate benchmarks for unmanaged Txn objects must be in place before implementation and benchcmp should verify the performance overhead of a finalizer. Further, adequate benchmarks of Txn.Reset and Txn.Renew must be in place so that their benefits can be further quantified and compared.

PutMulti does not return the number of items written

When MDB_MULTIPLE is given to the mv_data field on the "length" parameter is set to the number of bytes written. Presumably items are written in page order (which is not necessarily sorted order from my experimentation). It's not clear exactly whether this is supposed to interplay with another flag like MDB_NOOVERWRITE or merely with errors like MDB_MAP_FULL.

From the official docs:

MDB_MULTIPLE - store multiple contiguous data elements in a single request. This flag may only be specified if the database was opened with MDB_DUPFIXED. The data argument must be an array of two MDB_vals. The mv_size of the first MDB_val must be the size of a single data element. The mv_data of the first MDB_val must point to the beginning of the array of contiguous data elements. The mv_size of the second MDB_val must be the count of the number of data elements to store. On return this field will be set to the count of the number of elements actually written. The mv_data of the second MDB_val is unused.

For backwards compatibility I can't change the signature of PutMulti. However for #99 I was probably going to add a new function for writing Multiple values. So it would make sense to address this issue with that new function.

lmdb: Support integer flags (MDB_INTEGERKEY/MDB_INTEGERDUP)

I would like to attempt support for MDB_INTEGERKEY and MDB_INTEGERDUP (See mdb_dbi_open). Any user attempting benefit from them would be in tricky "unsafe" territory without more direct support. But the two flags actually open a wide variety of database schemata (and query structure).

The API extension to support the flags needs be thought out carefully.

• Mock implementation and benchmark link

lmdb: possible CGO pointer argument panic using bytes.Buffer

My experimental raft backend at github.com/bmatsuo/raft-mdb issues a panic due to the new cgo pointer restrictions. The problem occurs when a bytes.Buffer with a small amount of data is written using the Bytes method.

$ go test
--- FAIL: TestMDB_Logs (0.00s)
panic: runtime error: cgo argument has Go pointer to Go pointer [recovered]
    panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 11 [running]:
panic(0x896ec0, 0xc82000bc00)
    /usr/lib/go/src/runtime/panic.go:464 +0x3e6
testing.tRunner.func1(0xc8200805a0)
    /usr/lib/go/src/testing/testing.go:467 +0x192
panic(0x896ec0, 0xc82000bc00)
    /usr/lib/go/src/runtime/panic.go:426 +0x4e9
github.com/bmatsuo/lmdb-go/lmdb._cgoCheckPointer0(0x7d23c0, 0xc8200541e4, 0x0, 0x0, 0x0, 0xc82000bba8)
    ??:0 +0x4d
github.com/bmatsuo/lmdb-go/lmdb.(*Txn).Put(0xc8200c1f40, 0xc800000002, 0xc82000bba8, 0x8, 0x8, 0xc8200541e4, 0x1f, 0x40, 0x0, 0x0, ...)
    /home/b/src/github.com/bmatsuo/lmdb-go/lmdb/txn.go:297 +0x19d
github.com/bmatsuo/raft-mdb.(*MDBStore).StoreLogs.func1(0xc8200c1f40, 0x0, 0x0)
    /home/b/src/github.com/bmatsuo/raft-mdb/mdb_store.go:180 +0x31d
github.com/bmatsuo/lmdb-go/lmdb.(*Env).run(0xc8200280f0, 0x489301, 0x0, 0xc820049b90, 0x0, 0x0)
    /home/b/src/github.com/bmatsuo/lmdb-go/lmdb/env.go:461 +0x16a
github.com/bmatsuo/lmdb-go/lmdb.(*Env).Update(0xc8200280f0, 0xc820049b90, 0x0, 0x0)
    /home/b/src/github.com/bmatsuo/lmdb-go/lmdb/env.go:437 +0x45
github.com/bmatsuo/raft-mdb.(*MDBStore).StoreLogs(0xc820116600, 0xc820049bf0, 0x1, 0x1, 0x0, 0x0)
    /home/b/src/github.com/bmatsuo/raft-mdb/mdb_store.go:186 +0x63
github.com/bmatsuo/raft-mdb.(*MDBStore).StoreLog(0xc820116600, 0xc820015c80, 0x0, 0x0)
    /home/b/src/github.com/bmatsuo/raft-mdb/mdb_store.go:164 +0x84
github.com/bmatsuo/raft-mdb.TestMDB_Logs(0xc8200805a0)
    /home/b/src/github.com/bmatsuo/raft-mdb/mdb_store_test.go:167 +0x944
testing.tRunner(0xc8200805a0, 0xd85f40)
    /usr/lib/go/src/testing/testing.go:473 +0x98
created by testing.RunTests
    /usr/lib/go/src/testing/testing.go:582 +0x892
exit status 2
FAIL    github.com/bmatsuo/raft-mdb 0.006s

Calling convention does not conform to proposed cgo tool restritions

Proposal: https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md

C functions that take an MDBval* are not meeting the restrictions proposed in go#12416. The pointers in the argument's mv_data argument is usually a Go pointer. The infractions should be fixed before the implementation is finalized and lands on tip.

lmdb: strange numbers of allocations reported by benchmarks

@lmb pointed out some anomalous looking numbers reported by go test -bench=. -benchmem in #61, Specifically regarding the BenchmarkTxn_Put function.

$ go test -test.run=None -test.bench='BenchmarkTxn_Put$' -benchmem
PASS
BenchmarkTxn_Put-4       1000000              1759 ns/op              96 B/op          4 allocs/op
--- BENCH: BenchmarkTxn_Put-4
        bench_test.go:444: initializing random source data
ok      github.com/bmatsuo/lmdb-go/lmdb 2.616s

The 4 allocations reported are difficult to track down. One should be happening in the benchmark function itself (edit: that is false, allocations in the benchmark function get amortized and do not have a significant effect). But the others are not easy to account for. In a normal (non-error) case Txn.Put does not appear to make allocations inside the Go runtime.

@lmb ran the pprof tool using -alloc_objects and the following information about allocations in the benchmark function was comptued:

(pprof) list lmdb.BenchmarkTxn_Put.func1
Total: 506370
ROUTINE ======================== github.com/bmatsuo/lmdb-go/lmdb.BenchmarkTxn_Put.func1 in /Users/lorenz/go/src/github.com/bmatsuo/lmdb-go/lmdb/bench_test.go
         0     440425 (flat, cum) 86.98% of Total
         .          .     77:   err = env.Update(func(txn *Txn) (err error) {
         .          .     78:       b.ResetTimer()
         .          .     79:       defer b.StopTimer()
         .          .     80:       for i := 0; i < b.N; i++ {
         .          .     81:           k := ps[rand.Intn(len(ps)/2)*2]
         .         21     82:           v := makeBenchDBVal(&rc)
         .     440404     83:           err := txn.Put(dbi, k, v, 0)
         .          .     84:           if err != nil {
         .          .     85:               return err
         .          .     86:           }
         .          .     87:       }
         .          .     88:       return nil

Nothing seems to be adding up correctly here.

lmdbscan: allow opnext to be set without opset?

It's unclear if the lmdbscan.Scanner needs an alternative to lmdb.SetNext that sets the "opnext" operation without first setting the "opset" operation.

This relates to #17.

Add Env.ReaderCheck to primary usage examples

This method is actually pretty critical to using MDB in production. And it is not being called out at all in any documentation. If an application crashes for some (potentially unrelated) reason during a read transaction you can easily end up with stale readers and machines hitting their disk quotas.

lmdb: finalizer for Env

Relates to #20

A finalizer for lmdb.Env should be straightforward and low overhead. The finalizer needs to check if Env._env is nil, closing it and setting it to nil otherwise. The finalizer must not panic if the Env was closed correctly.

lmdb: Unsupported features

Unsupported C functions:

• mdb_env_get_fd* (#3)
• mdb_env_set_userctx*/mdb_env_get_userctx*
  • See comment below.
  • Why not put a field in the Env struct instead to avoid cgo call overhead?
• mdb_env_set_assert* (wontfix)
• mdb_set_compare**/mdb_set_dupsort** and mdb_cmp/mdb_dcmp (#12)(#61)
• mdb_set_relfunc*/mdb_set_relctx*
  • See comment.

* -- It's currently unclear what benefits this brings
** -- If this can be achieved it will not be trivial for users to instrument

Unusable flags:

• MDB_INTEGERKEY/MDB_INTEGERDUP (#11)

Invalid constant valMaxSize in lmdb/val.go when run in 32 bit OS

Getting "array bound is too large" from Go in lmdb/val.go

Originates from valMaxSize constant. On 32-bit processors this is 1<<31-1 (2GB) rather than 1<<32-1 (which would be the unsigned integer length). Also on 64-bit processors the max value could be much larger for LMDB. Although as stated in comments this is for portability. However this strongly constrains database size.

lmdb: Update to LMDB 0.9.18

Released on 2016-02-05

lmdb: failure to defer leads to resource leaks

There is some possibility of memory leaks in the lmdb package because garbage collection will not collect allocations performed by C. The package must go through an audit of the core types so that potential leaks may be found. Finalizers should be added and benchcmp run in to determine the runtime cost.

If it is determined that a type should not have a finalizer for the sake of performance it must be clear in the type's documentation that it can leak when misused.

• Env (#26)
  • Allocation must be performed by C in mdb_env_create
  • Leak when Env.Close is not called. Makes rules for Env.Close inconsistent with those stated in C documentation.
  • Finalizer is probably low overhead due to infrequent creation/destruction of environments.
• Txn (#27)
  • Allocation must be performed by C in mdb_txn_begin.
  • Leak when Txn.Commit or Txn.Abort is not called. Managed transactions cannot leak memory.
  • Finalizer may introduce overhead. Txn.Reset and Txn.Renew can reduce overhead for readonly transactions.
• Cursor (#28)
  • Allocation must be performed by C in mdb_cursor_open
  • Leak when Cursor.Close is not called. Makes rules for Cursor.Close inconsistent with those stated in C documentation.
  • Finalizer may introduce overhead. Cursor.Reset can reduce overhead for readonly cursors.
• mdbVal
  • Must be checked on a use-by-use basis.
  • As long as the an *mdbVal is allocated by Go there shouldn't be problems. The data should always either be a Go pointer or a C pointer managed by LMDB (must not be freed by GC).

Update LMDB to 0.9.19

The GitHub mirror of the project is out-of-date and a new version of LMDB has been released. It is available on the mdb.RE/0.9 branch of the OpenLDAP repository (commit 27f6160).

lmdbsync: default MapResizeHandler strategy should have exponential backoff

Right now there is a constant delay. This is doesn't seem terrible for the default (low) number of retries. But increasing the number of retries is not as effective as it should be.

Also, it seems a little weird that the delay function only gets called after one immediate attempt. It seems more flexible and consistent to always call the delay function.

Support for `mdb_set_compare` and `mdb_set_dupsort`.

I think there is potential value here. But there needs to be some research and, in general, the C API needs to change for applications to take full advantage of these functions.

Background

The LMDB C API lets an application specify custom functions used to sort keys and values in a database. And this feature is not supported in lmdb-go despite potential benefits.

It is common for applications to store data in LMDB with fixed-width integer keys. Currently these keys must be encoded as big-endian byte slices and are compared as c-strings when scanned.

It may also be desirable to store data using exotic keys or exploit data structure when sorting duplicates. These cases are extreme but should be possible (and useful) with LMDB. A combination of the LMDB API and cgo prevent such things from being practical today. But were circumstances to change an lmdb-go API that could eventually support this functionality would be ideal.

Proposal

Define a new Cmp type that is an opaque handle for a comparison function. And, when arbitrary comparison functions can be provided, two functions for initializing and deinitializing a Cmp respectively will be be defined.

type Cmp uintptr

// These functions would be defined once the LMDB C API makes it possible use
// arbitrary Go for sorting.
func CompareFunc(func (a, b []byte) int) Cmp
func DestroyCompareFunc(Cmp)

With the Cmp type two new transaction methods can be added for supplying alternate comparison functions.

func (txn *Txn) SetCmp(dbi DBI, cmp Cmp) int)

func (txn *Txn) SetCmpDup(dbi DBI, cmp Cmp) int)

To provide some utility before the LMDB C API changes several C-based sorting functions can be defined.

const (
    CmpBE64 Cmp = 0
    CmpBE32 Cmp = 1
    CmpBE16 Cmp = 2
)

These Go constants will have corresponding C functions defined in the lmdbgo C library.

int lmdbgo_cmp_func_sortbe64(const MDB_val *a, const MDB_val *b);
int lmdbgo_cmp_func_sortbe32(const MDB_val *a, const MDB_val *b);
int lmdbgo_cmp_func_sortbe16(const MDB_val *a, const MDB_val *b);

Examples

Examples of the various facets covered in this proposal are given below. In each case the comparison function is set immediately when the database handle is open, which itself immediately follows the call to Env.Open. This is the only calling pattern that will be supported.

Arbitrary Cmp Functions

In this example an arbitrary comparison function is used to create a Cmp value that is then used to set the key comparison function for the database.

// ...
err = env.Open("/path/to/db", 0, 0644)
if err != nil {
    panic(err)
}

var dbi lmdb.DBI
dbcmp := lmdb.CompareFunc(doCompareCapnp)
defer lmdb.DestroyCompareFunc(dbcmp) 
err = env.Update(func(txn *lmd.Txn) (err error) {
    dbi, err = txn.OpenDBI("mydb", lmdb.Create)
    if err != nil {
        return err
    }
    err = txn.SetCmp(dbcmp)
    return err
})

// ...

Many to Many uint mapping

This example demonstrates a common real world case of database foreign keys. In this case 64-bit database IDs are mapped to one-another using a builtin comparison function to optimize seek performance.

// ...
err = env.Open("/path/to/db", 0, 0644)
if err != nil {
    panic(err)
}

var dbi lmdb.DBI
err = env.Update(func(txn *lmd.Txn) (err error) {
    dbi, err = txn.OpenDBI("mydb", lmdb.Create|lmdb.DupSort|lmdb.DupFixed)
    if err != nil {
        return err
    }
    err = txn.SetCmp(lmdb.CmpBE64)
    if err != nil {
        return err
    }
    err = txn.SetCmpDup(lmdb.CmpBE64)
    return err
})

// ...

Consequences

Existing programs will continue to function the same as they due. Their binary size may grow slightly.

Programs using builtin big-endian comparison functions will function the same semantically, but should have increased performance. Having a known length and encoding allows fast decoding to an integer format that can be quickly compared. Using a big-endian integer encoding is fortuitous because the Env.Copy* methods work as expected.

Risks

Big-endian integer encodings may see no performance increase when using builtin comparison functions.

The C API and cgo may never support hooking up arbitrary Go functions as comparison functions. Or the cgo bridge required may be too slow for practical use.

Alternatives

The MDB_INTEGERKEY and MDB_INTEGERDUP DBI flags perform very similar tasks for integer valued keys/data. The integer flags may even outperform custom functions. However their introduction into the lmdb-go API has not been finalized. The flags also give up specificity about key encoding (big-/little-endian) for performance which will have benefits and drawbacks.

Non-opaque values such as capnproto buffers cannot typically be used as keys in a database. Instead the values must either be stored in a separate database with opaque keys or appended to an opaque fixed-width prefix before stored as a key.

fewer examples as tests

Given the fairly complex nature of setting up, populating, and testing a database it seems more appropriate to have one or two executable examples and more examples that are reusable snippets in example_test.go simply so that their compilation can be verified.

The following example functions need to be evaluated and have any necessary tests and code examples added before being removed.

• ExampleTxn_dupFixed (#5)
• ExampleTxn_dupSort (#5)
• ExampleEnv (#5)
• ExampleTxn (#5)
• ExampleCursor (#5)
• ExampleTxn_OpenDBI (#8)

lmdb: Txn.Reset clears the Txn finalizer when it should not

The docs for Txn.Reset do not mention that the method clears the finalizer on the Txn object. They do indicate however that everything the finalizer is meant to prevent is still possible. So it does seem like retaining the finalizer until Abort is called is the correct action.

I probably did this because of a benchmark, without realizing the impact of it. So the performance impact should be quantified.

How readonly txn.Reset and txn.Renew use?

I create a read-only txn in a goroutine, then this txn will called in another goroutine frequently.

The scene is:

1. four writer goroutines write to the db, then four reader goroutines read this db(this four readers don't share txn.)
2. each time, a reader read some data from db (such as 20 key one time)，if db has not enough data, return, after 50ms, read again.

so I call txn.Reset after I create txn in a goroutine, before read call txn.Renew, after read call txn.Reset again. But some time, txn.Renew return error:

mdb_cursor_renew: invalid argument

So

1. How to use these two funcs?
2. Need I call cursor.Renew after txn.Renew?

lmdb: finalizer for Cursor

Relates to #20

Adding a finalizer for Cursor should be very easy. But care should be taken to ensure that benchmarks are in place and benchcmp is run before and after to quantify the overhead. Further, benchmarks for Cursor.Renew should be in place to further quantify its benefits.

Can't compile on windows

Windows 7 Pro, 64 bit. mingw-w64. go 1.5.

Compilation fails with error "can't convert -1 to mdb_filehandle_t".

This appears to be caused by an assumption that the mdb_filehandle_t will be an int, which is not the case on Windows.

func (env *Env) FD() (uintptr, error) {
    var mf C.mdb_filehandle_t
    ret := C.mdb_env_get_fd(env._env, &mf)
    err := operrno("mdb_env_get_fd", ret)
    if err != nil {
        return 0, err
    }
    if mf == C.mdb_filehandle_t(-1) { <<< error. assumes POSIX file handle.
        return 0, errNotOpen
    }
    return uintptr(mf), nil
}

/** An abstraction for a file handle.
 *  On POSIX systems file handles are small integers. On Windows
 *  they're opaque pointers.
 */
#ifdef _WIN32
typedef void *mdb_filehandle_t;
#else
typedef int mdb_filehandle_t;
#endif

lmdbscan: Scanner.Scan panics if lmdbscan.New() failed.

If lmdbscan.New() fails then the error will be wiped out when the scanner's Scan method will panic attempting to access the underlying cursor. This is counter to the goals of the lmdbscan package, where ideally errors are checked for once at the end of the transaction.

scanner.New(txn, 1234)
defer scanner.Close()
for scanner.Scan() { // panic!
  log.Printf("k=%q v=%q", scanner.Key(), scanner.Val())
}
return scanner.Err()

Opening existing database in read-only mode fails

I have to compatibility with c code.

(headerFile's type is uint64_t, val is 0)

  MDB_txn *otxn;
  mdb_txn_begin(_env, NULL, 0, &otxn);
  char dbName[21];
  sprintf(dbName, "%" PRIu64, headFile);
  mdb_dbi_open(otxn, dbName, MDB_CREATE, &_db);
  mdb_txn_commit(otxn);

go code:

topic.currentPartitionID's val is 0

err = env.View(func(txn *lmdb.Txn) error {
        topic.currentPartitionDB, err = txn.CreateDBI(uInt64ToString(topic.currentPartitionID))
        return err
    })
    if err != nil {
        log.Println("Call env.View failed: ", err)
        return err
    }
}

when run, I got error:

Call env.View failed: mdb_dbi_open: permission denied

How to debug this?

Windows compilation error

When trying to build on windows getting the following error. Any suggestion on how to fix?

.\mdb.c:4625:13: warning: implicit declaration of function 'pthread_mutexattr_setrobust' [-Wimplicit-function-declaration]
    || (rc = pthread_mutexattr_setrobust(&mattr, PTHREAD_MUTEX_ROBUST))
             ^
.\mdb.c:4625:49: error: 'PTHREAD_MUTEX_ROBUST' undeclared (first use in this function)
    || (rc = pthread_mutexattr_setrobust(&mattr, PTHREAD_MUTEX_ROBUST))
                                                 ^
.\mdb.c:4625:49: note: each undeclared identifier is reported only once for each function it appears in
.\mdb.c: In function 'mdb_mutex_failed':
.\mdb.c:351:37: warning: implicit declaration of function 'pthread_mutex_consistent' [-Wimplicit-function-declaration]
 #define mdb_mutex_consistent(mutex) pthread_mutex_consistent(mutex)
                                     ^
.\mdb.c:10002:10: note: in expansion of macro 'mdb_mutex_consistent'
    rc2 = mdb_mutex_consistent(mutex);
          ^

The main issue is that PTHREAD_MUTEX_ROBUST is undefined.

lmdb: Cursor.DBI incorrectly returns 0 after the Cursor is closed

The implementation of Cursor.DBI explicitly returns 0 when the cursor is closed. The value DBI(0) is valid and references the environment freelist. So this is incorrect.

lmdb: limited support for mdb_set_compare and mdb_set_dupsort

Trying to figure out how to support these functions has been a long road that is at least partially documented in #12. But I have finally collected all the information I need about the practical requirements for using the APIs and the inherent limitations/guarantees of cgo. I believe the final pieces of the puzzle were delivered from Ian Lance Taylor on the golang-nuts mailing list.

https://groups.google.com/forum/#!topic/golang-nuts/gvezyWFEAYI

I can proceed with the tentative plan I developed in #12 with work I have started on my branch, bmatsuo/compare-funcs. I will need to write substantial documentation with adequate warnings plastered around the docs. Making use of it in an application is about the trickiest thing I have seen in terms of cgo. Hopefully there will be minimal foot-shooting after the feature's introduction.

Basic requirements for use

Writing a custom comparison function in C is the fastest and simplest way to use custom comparison. The lmdb.h file must be copied from lmdb-go into a local project directory in order to ensure that the C.MDB_val struct is visible (using the same version of the header file is safest). Then a C function with MDB_compare_func signature can be defined. And its pointer can be passed to Txn.SetCompare().

package main

/*
#include "lmdb.h"

static inline int my_comparison_func(const MDB_val *a, const MDB_val *b) {
    // ...
}
*/
import "C"

import "github.com/bmatsuo/lmdb-go/lmdb"

func main() {
    //...
    var dbi lmdb.DBI
    env.Update(func(txn *Txn) (err error) {
        dbi, err = txn.OpenDBI("foodb", lmdb.Create)
        if err != nil {
            return err
        }
        txn.SetCompare(dbi, unsafe.Pointer(&C.my_comparison_func))
    }) 
}

lmdb: Txn.OpenDBI return value is not specified and not implemented clearly

I took a look at the import graph generated by godoc.org. It pointed out some interesting things. Most significantly it seems like math.NaN() is used to generate an "invalid" DBI in case of failure in mdb_dbi_open.

    if ret != success {
        return DBI(math.NaN()), operrno("mdb_dbi_open", ret)
    }

This is weird but it also turns out to be pointless. Converting math.NaN() to DBI (a uint type) results in 0. This turns out to be questionable behavior because 0 is not actually an invalid DBI.

It is probably more appropriate to return ^DBI(0), -1. This should not be a valid DBI (it may even be guaranteed due to a limit from some constant #define).

Deadlock under Linux

Hello! I've been using my own bindings for a while, but recently switched to your package because of handy API and because Go doesn't allow to link the same C symbols under different package names.

The problem is, the code you're using in the package is somehow differs from their tip version on GitHub. I've been using the tip 0.9.70 from here: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/lmdb.h

It has no release date, at first I was confused (see the title change of this issue).

There isn't much difference in the API, except some documentation improve, but it has one big feature called MDB_USE_SYSV_SEM that allows to use different mutex provider under Linux environment.

Without this flag enabled, txn, err := l.env.BeginTxn(nil, 0) may deadlock, not Go kind of deadlock, but some deeply internal deadlock that can be only killed by kill -9. The holds true for Amazon AWS Linux or desktop Arch linux at least.

My advise is to switch to the tip LMDB version (no degradation AFAIK), and enable MDB_USE_SYSV_SEM by default for Linux systems. See the pull request related as an example of such transition - I just replaced C files and fixed types in two places in Go wrapping.

all: internal package for creating, populating, and tearing down environments.

There are functions that do this in several places. A single set of functions should be defined in an "lmdbtest" package in directory internal/lmdbtest/.

• Create a new environment at a temporary path with some given options (MapSize, MaxDBs, MaxReaders, Flags)
• Close an environment and remove its path.
• Open a named DBI
• Write a set of items to a database.

How offset and when should ReaderCheck be called?

Should I call ReaderCheck everytime before

func (env *Env) Update(fn TxnOp) error 

and

func (env *Env) View(fn TxnOp) error 

Exception with cursor.Get(k, nil, Set)

I have db with dup values (not sure if relevant). When I'm trying to do k, v, e = cursor.Get(buf.Bytes(), nil, lmdb.Set) I get exception (see below). With k, v, e = cursor.Get(buf.Bytes(), nil, lmdb.SetKey) it works. Just Set positions cursor at the key but doesn't read value. But cursor.Get still attempts to read it which leads to the exception.

panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 21 [running]:
panic(0x56f360, 0xc0420b0260)
        C:/Go/src/runtime/panic.go:500 +0x1af
github.com/bmatsuo/lmdb-go/lmdb._cgoCheckPointer0(0xc0420bc024, 0x0, 0x0, 0x0, 0x0)
        ??:0 +0x60
github.com/bmatsuo/lmdb-go/lmdb.getBytesCopy(0xc0420b0170, 0xc0420b0170, 0x23190e0, 0x556ee0)
        D:/Projects/Go/src/github.com/bmatsuo/lmdb-go/lmdb/val.go:120 +0x51
github.com/bmatsuo/lmdb-go/lmdb.(*Txn).bytes(0xc0420b6120, 0xc0420b0170, 0x25, 0x40, 0xf)
        D:/Projects/Go/src/github.com/bmatsuo/lmdb-go/lmdb/txn.go:276 +0x7a
github.com/bmatsuo/lmdb-go/lmdb.(*Cursor).Get(0xc0420b01a0, 0xc0420bc024, 0x25, 0x40, 0x0, 0x0, 0x0, 0xf, 0xa0, 0x61, ...)
        D:/Projects/Go/src/github.com/bmatsuo/lmdb-go/lmdb/cursor.go:155 +0x147
github.com/bmatsuo/lmdb-go/lmdbscan.(*Scanner).Set(0xc0420ba120, 0xc0420bc024, 0x25, 0x40, 0x0, 0x0, 0x0, 0xf, 0x4c)
        D:/Projects/Go/src/github.com/bmatsuo/lmdb-go/lmdbscan/scanner.go:74 +0xc4
github.com/bmatsuo/lmdb-go/lmdbscan.(*Scanner).SetNext(0xc0420ba120, 0xc0420bc024, 0x25, 0x40, 0x0, 0x0, 0x0, 0xf, 0x9, 0x25)
        D:/Projects/Go/src/github.com/bmatsuo/lmdb-go/lmdbscan/scanner.go:85 +0x9b
bitbucket.org/agileharbor/lmdbbenchmark/lmdbLayer.(*Reader).ReadDupRecords.func1(0xc0420b6120, 0x0, 0x0)
        D:/Projects/Go/src/bitbucket.org/agileharbor/lmdbbenchmark/lmdbLayer/reader.go:103 +0x83d
github.com/bmatsuo/lmdb-go/lmdb.(*Env).run(0xc042072040, 0x412c00, 0x20000, 0xc0420ae0f0, 0x0, 0x0)
        D:/Projects/Go/src/github.com/bmatsuo/lmdb-go/lmdb/env.go:461 +0x157
github.com/bmatsuo/lmdb-go/lmdb.(*Env).View(0xc042072040, 0xc0420ae0f0, 0x9, 0xc000000002)
        D:/Projects/Go/src/github.com/bmatsuo/lmdb-go/lmdb/env.go:423 +0x4a

lmdb: deprecated conversion from pointers to slices

The cgo wiki page suggests a different way to turn arrays into slices.

https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices

The current method using the "reflect" package isn't guaranteed to be compatible.

https://godoc.org/reflect#SliceHeader

The docs claims the method is not portable. This may be due to os/arch restrictions on the amount of memory available for use and the slice implementation.

https://github.com/golang/go/blob/6b0688f7421aeef904d40a374bae75c37ba0b8b4/src/runtime/slice.go#L17-L34
https://github.com/golang/go/blob/a03bdc3e6bea34abd5077205371e6fb9ef354481/src/runtime/malloc.go#L151-L164

lmdbscan: Scanner needs to expose the cursor somehow

Having the Del method on the scanner itself seems odd. And there may be a desire to call other things like Count (or Put). There are a couple ways the cursor could be exposed.

1. Change the Scanner constructor to take a Cursor instead of a DBI. This would make the Scanner API as simple as possible but it complicates Cursor construction.
2. Add the constructor above along side the existing one. This would make the common case straightforward. But closing semantics are a little more complex.
3. Add a method to get the cursor from the scanner (e.g. s.Cursor().Del(0)). This is pretty flexible. Cursor closing semantics are already a little weird.
4. Export the Cursor struct field in a Scanner (e.g. s.Cursor.Del(0)). Slightly more convenient than above. A little less safe.

mdb_cursor_get: MDB_PAGE_NOTFOUND: Requested page not found

When will this error happen?
What's the influence?
May it be ignore?

lmdbsync: Remove exp/cmd/lmdb_example_resize

The command is basically a test meant to be run in parallel in background processes of a shell or in a multiplexer. The example should be pared down as much as possible, rerooted under exp/lmdbsync, and run by go test for that package (it must run as two forked processes because only one open map of an environment can exist in a process).

lmdbscan: stabilize API and move out of exp/

The lmdbscan API hasn't changed in a long time. There is an open question regarding its usage (#18). But anything coming out of that issue should be backwards compatible.

People really should be using the package. I'm a little worried that no one is using it primarily because it is under exp/.

I need anyone who has backward incompatible suggestions for exp/lmdbscan to bring them up soon. No real deadline, but at some point I will move things and incompatible changes will not happen in the foreseeable future.

lmdbscan: better ability to handle duplicates in a special way

Scanning over databases and treating values for duplicate keys specially seems to be fairly cumbersome. It is always possible to iterate strictly using Cursor.Next. But a simple action like collecting all values for duplicate keys and printing them, is not as easy is it potentially could be. This is an example of scanning over a database using the NextNoDup and NextDup flags.

    err := env.View(func(txn *lmdb.Txn) (err error) {
        scanner := lmdbscan.New(txn, dbi)
        defer scanner.Close()

        for {
            scanner.Set(nil, nil, lmdb.NextNoDup)
            if !scanner.Scan() {
                return scanner.Err()
            }  
            k := scanner.Key()
            vals := [][]byte{scanner.Val()}
            scanner.SetNext(nil, nil, lmdb.NextDup, lmdb.NextDup)
            for scanner.Scan() {
                vals = append(vals, scanner.Val())
            }  
            if scanner.Err() != nil {
                return scanner.Err()
            }
            log.Printf("k=%q vals=%q", k, vals)
        }  
    }) 
    if err != nil {
        panic(err)
    }  

It should not be so clumsy. In particular, scanner.SetNext(nil, nil, lmdb.NextDup, lmdb.NextDup) is pretty lame.

How to build with glide?

I'm using golang v1.6.1, and glide for vendor manage.

I run

glide rebuild 

In my project, which has been add to $GOPATH, get error:

[WARN] The rebuild command is deprecated and will be removed in a future version
[WARN] Use the go install command instead
[INFO] Building dependencies.
[INFO] Running go build github.com/bmatsuo/lmdb-go
[WARN] Failed to run 'go install' for github.com/bmatsuo/lmdb-go: can't load package: package ./vendor/github.com/bmatsuo/lmdb-go: no buildable Go source files in /Users/zwb/Code/Go/queue/vendor/github.com/bmatsuo/lmdb-go

I hope to build lmdb-go first, so I can get auto-complete with gocode set lib-path.

lmdb: Calling convention does not conform to proposed cgo tool restritions

Proposal: https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md

C functions that take an MDB_val* are not meeting the restrictions proposed in go#12416. The pointer in the arguments' mv_data field is usually a Go pointer. The infractions should be fixed when the design is finalized and by the time the implementation lands on "tip".

PutMulti incorrectly panics when passed an empty page

The lmdb package has to adjust empty inputs to allow CGO calls to succeed in the general case. But this adjustment causes Cursor.PutMulti to panic when it tries to validate the consistency of its arguments.

Unfortunately, in the current release of LMDB fixing this bug would only cause LMDB to segfault moments later. But once this bug in LMDB is fixed this bug should be addressed.

Note that the plan in #99 is to introduce new function for writing Multiple values. This function will not have the bug. This means that it will probably segfault with a nil-pointer reference until an fix for LMDB is released, or until a fix can be confirmed so much that it can be emulated.

The fix in LMDB will likely trigger an EINVAL error in these cases, which would not be hard to emulate. It is just a matter of making sure the LMDB folks have committed to this.

Release 1.6.0

There is a sizable chunk of CHANGES not tagged in a release. I think after the bug fix #57, which caused a performance degration, and the optimization #62 I think things are in a decent spot for a new tag to be created.

The tag will probably be made in the next couple days. After the freshest commits have cured for a short time.

lmdbsync: Env should be an argument of HandleTxnErr

The docs for golang.org/x/net/context recommend that Context not be used to pass arguments to between functions. The Env is truly and argument for the functions. It is not always used. But it is common enough when you want to use a handler that it deserves inclusion in the arg list.

So the Handler interface should look as follows:

type Handler interface {
    HandleTxnErr(ctx context.Context, env *Env, err error) (context.Context, error)
}

lmdbsync: investigate use of golang.org/x/net/context

The hand-rolled Bag implementation is a little silly. And it may be bloating the implementation too much. I will put a simple branch together that relies on context.Context.

At some point this is going to get pulled into the stdlib. And it won't be under a "net" prefix.

with lmdb.Append get error: mdb_put: MDB_KEYEXIST: Key/data pair already exists

I call

 txn.Put(qp.partition, []byte("12345"), v.mem, lmdb.Append)

Got error:

mdb_put: MDB_KEYEXIST: Key/data pair already exists

Below is my code, the qp.partition type is lmdb.DBI, inited before I call push

qp.partition, err = txn.CreateDBI(uInt64ToString(qp.curPartitionID))

func (qp *QProducer) push(batch []QPItemType) error {
    isFull := false
    err := func() error {
        err := qp.qt.qe.env.Update(func(txn *lmdb.Txn) error {
            offset, err := qp.qt.getProducerMeta(txn)
            if err != nil {
                return errors.Wrap(err, "Call getProducerMeta failed")
            }
            for _, v := range batch {
                offset++
                err = txn.Put(qp.partition, []byte("12345"), v.mem, lmdb.Append)
                if err != nil {
                    if err == lmdb.MapFull {
                        isFull = true
                        break
                    }
                    return errors.Wrap(err, "Call txn.Put failed")
                }
            }
            if !isFull {
                qp.qt.updateProducerMeta(txn, offset)
            }
            return nil
        })
        if err != nil && err == lmdb.MapFull {
            isFull = true
            return nil
        }
        return errors.Wrap(err, "Call env.Update failed")
    }()

The loop is only once, and I delete db everytime before I run this test, also set the key to a sepcial key,

err = txn.Put(qp.partition, []byte("12345"), v.mem, lmdb.Append)

So It cann't EXIST in db.

I don't know how to debug this.