snej/nimdbx

Fast persistent key-value store for Nim, based on libmdbx

nimdbx's Issues

Open a collection inside a write transaction

I'm getting this error:

MDBX_BUSY: Another write transaction is running, or environment is already used while opening with MDBX_EXCLUSIVE flag

Is there a workaround for this?

Build errors on Windows for the first test

import nimdbx

let CollectionName = "stuff"
var db = openDatabase(".")
var coll = db.createCollection(CollectionName)

error as follows:

# Running make
#   Path: C:\Users\asus\.nimble\pkgs\nimdbx-0.4.1\libmdbx-dist
stack trace: (most recent call last)
C:\Users\asus\.nimble\pkgs\nimterop-0.6.13\nimterop\build\getheader.nim(485, 23)
C:\Users\asus\.nimble\pkgs\nimterop-0.6.13\nimterop\build\getheader.nim(234, 11) buildLibrary
C:\Users\asus\.nimble\pkgs\nimterop-0.6.13\nimterop\build\tools.nim(196, 19) make
C:\Users\asus\.nimble\pkgs\nimterop-0.6.13\nimterop\build\shell.nim(90, 11) execAction
C:\Nim\lib\system\assertions.nim(30, 26) failedAssertImpl
C:\Nim\lib\system\assertions.nim(23, 11) raiseAssert
C:\Nim\lib\system\fatal.nim(53, 5) sysFatal
C:\Users\asus\.nimble\pkgs\nimdbx-0.4.1\nimdbx\private\libmdbx.nim(17, 10) template/generic instantiation of `getHeader` from here
C:\Nim\lib\system\fatal.nim(53, 5) Error: unhandled exception: C:\Users\asus\.nimble\pkgs\nimterop-0.6.13\nimterop\build\shell.nim(90, 7) `not die` Command failed: 2
cmd: cmd /c C: && cd C:\Users\asus\.nimble\pkgs\nimdbx-0.4.1\libmdbx-dist && make -j 16
result:
The system cannot find the path specified.
'case' is not recognized as an internal or external command,
operable program or batch file.
'Darwin*' is not recognized as an internal or external command,
operable program or batch file.
'test' is not recognized as an internal or external command,
operable program or batch file.
'test' is not recognized as an internal or external command,
operable program or batch file.
'test' is not recognized as an internal or external command,
operable program or batch file.
'which' is not recognized as an internal or external command,
operable program or batch file.
cc -O2 -g -Wall -Werror -Wextra -Wpedantic -ffunction-sections -fPIC -fvisibility=hidden -std=gnu11 -pthread -Wno-error=attributes  -DNDEBUG=1 '-DMDBX_CONFIG_H="config.h"' -ULIBMDBX_EXPORTS -c mdbx.c -o mdbx-static.o
GNUmakefile:101: recipe for target 'mdbx-static.o' failed
process_begin: CreateProcess(NULL, cc -O2 -g -Wall -Werror -Wextra -Wpedantic -ffunction-sections -fPIC -fvisibility=hidden -std=gnu11 -pthread -Wno-error=attributes -DNDEBUG=1 -DMDBX_CONFIG_H=\"config.h\" -ULIBMDBX_EXPORTS -c mdbx.c -o mdbx-static.o, ...) failed.
make (e=2): ϵͳҲָļ

make: *** [mdbx-static.o] Error 2
 [AssertionDefect]

then I tried to build manually to see detail, which is as follows:

PS C:\Users\asus\.nimble\pkgs\nimdbx-0.4.1\libmdbx-dist> gcc -std=gnu11 -O2 -g -Wall -Werror -Wextra -Wpedantic -ffunction-sections -fPIC -fvisibility=hidden -pthread -Wno-error=attributes  -DNDEBUG=1 -DMDBX_CONFIG_H="config.h" -ULIBMDBX_EXPORTS -c mdbx.c -o mdbx-static.o
mdbx.c:16:10: error: #include expects "FILENAME" or <FILENAME>
 #include MDBX_CONFIG_H
          ^~~~~~~~~~~~~
mdbx.c:1371:29: error: unknown type name 'FILE_INFO_BY_HANDLE_CLASS'
     _In_ HANDLE hFile, _In_ FILE_INFO_BY_HANDLE_CLASS FileInformationClass,
                             ^~~~~~~~~~~~~~~~~~~~~~~~~
mdbx.c:1373:19: error: unknown type name 'MDBX_GetFileInformationByHandleEx'
 MDBX_INTERNAL_VAR MDBX_GetFileInformationByHandleEx
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mdbx.c:1392:29: error: unknown type name 'FILE_INFO_BY_HANDLE_CLASS'
     _In_ HANDLE hFile, _In_ FILE_INFO_BY_HANDLE_CLASS FileInformationClass,
                             ^~~~~~~~~~~~~~~~~~~~~~~~~
mdbx.c:1394:19: error: unknown type name 'MDBX_SetFileInformationByHandle'
 MDBX_INTERNAL_VAR MDBX_SetFileInformationByHandle
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mdbx.c: In function 'bad_page':
mdbx.c:3617:22: error: ISO C does not support the 'I64' ms_printf length modifier [-Werror=format=]
                      "corrupted page #%u, mod-txnid %" PRIaTXN " \n",
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mdbx.c: In function 'thread_key_create':
mdbx.c:4124:14: error: ISO C does not support the 'I64' ms_printf length modifier [-Werror=format=]
   mdbx_trace("&key = %p, value %" PRIuPTR ", rc %d",
...

Automatically build libmdbx

Use Nimterop to build the C library.

Ideally, statically link it for better performance (and fewer problems finding the dylib.)

Writeable cursors (Cursor.put, etc.)

MDBX_cursor supports writing to the collection, if it's in a write-transaction. Should support this (with a WriteCursor subclass.)

Typed Collections

The key/value API is awkward because keys and values can be anything. Data and DataOut help, but aren't very safe.

It would be nice to have generic KEY and VALUE parameters on Collection, that can take values like string, seq[byte], int32, Collatable, etc. Then those also need to be added to Snapshot, Transaction, Cursor, and all the CRUD functions.

The problem is, if I just make everything generic, there's a ton of code duplication since the functions get instantiated for every generic type. So it'd be better to make this stuff a wrapper around the existing untyped code.

'pastMinKey' can have side effects

~/nimdbx/nimdbx/Cursor.nim(106, 6) Error: 'pastMinKey' can have side effects
> ~/nimdbx/nimdbx/Cursor.nim(106, 43) Hint: 'pastMinKey' calls `.sideEffect` 'compareKey'
>> ~/nimdbx/nimdbx/Cursor.nim(92, 6) Hint: 'compareKey' called by 'pastMinKey'

Zero-copy put

Curious if you know whether it might be possible to put a Nim table into nimdbx without any copying, or is that nonsensical? I'm basically wondering if it's possible to skip serialization altogether and effectively transfer ownership of the memory taken by a Nim table over to nimdbx. Thanks!

nimble test errors on Windows

  Compiling C:\nimdbx\tests\test1_Data (from package nimdbx) using c backend
# Including library C:\nimdbx\libmdbx-dist\libmdbx.a
# Importing C:\nimdbx\libmdbx-dist\mdbx.h
# Saved to C:/Users/asus/nimcache/nimterop/toastCache/nimterop_2496732934.nim
C:\nimdbx\nimdbx\private\libmdbx.nim(56, 8) template/generic instantiation of `cImport` from here
C:\Users\asus\.nimble\pkgs\nimterop-0.6.13\nimterop\cimport.nim(704, 10) template/generic instantiation of `cImport` from here
C:\Users\asus\nimcache\nimterop\toastCache\nimterop_2496732934.nim(988, 48) Error: undeclared identifier: 'HANDLE'
       Tip: 17 messages have been suppressed, use --verbose to show them.
     Error: Execution failed with exit code 1
        ... Command: C:\Nim\bin\nim.exe c --noNimblePath -d:NimblePkgVersion=0.4.1 --path:C:\Users\asus\.nimble\pkgs\nimterop-0.6.13 --path:C:\Users\asus\.nimble\pkgs\regex-0.16.2 --path:C:\Users\asus\.nimble\pkgs\unicodedb-0.9.0 --path:C:\Users\asus\.nimble\pkgs\unicodeplus-0.8.0 --path:C:\Users\asus\.nimble\pkgs\unicodedb-0.9.0 --path:C:\Users\asus\.nimble\pkgs\segmentation-0.1.0 --path:C:\Users\asus\.nimble\pkgs\unicodedb-0.9.0 --path:C:\Users\asus\.nimble\pkgs\cligen-1.2.2 --hints:off -r --path:. C:\nimdbx\tests\test1_Data

libmdbx submodule doesn't get installed

Using nimble install, it installed cleanly without error.

A program with only an import and an echo will not build.
Looks like the libmdbx files are missing, but I can't find a way to get them added.
..................................................
stack trace: (most recent call last)
/home/jason/.nimble/pkgs/nimterop-0.6.13/nimterop/build/getheader.nim(485, 23)
/home/jason/.nimble/pkgs/nimterop-0.6.13/nimterop/build/getheader.nim(238, 11) buildLibrary
/home/jason/Local_Install/nim-1.4.2/lib/system/assertions.nim(30, 26) failedAssertImpl
/home/jason/Local_Install/nim-1.4.2/lib/system/assertions.nim(23, 11) raiseAssert
/home/jason/Local_Install/nim-1.4.2/lib/system/fatal.nim(49, 5) sysFatal
/home/jason/.nimble/pkgs/nimdbx-0.4.0/nimdbx/private/libmdbx.nim(17, 10) template/generic instantiation of getHeader from here
/home/jason/Local_Install/nim-1.4.2/lib/system/fatal.nim(49, 5) Error: unhandled exception: /home/jason/.nimble/pkgs/nimterop-0.6.13/nimterop/build/getheader.nim(238, 12) buildStatus.built
Build configuration failed - No build files found in /home/jason/.nimble/nimdbx/vendor/libmdbx
[AssertionDefect]
The terminal process "/bin/bash '-c', 'nim c -r -o:dbxnim /home/jason/Files/Projects/Nim/Test/dbxnim.nim'" terminated with exit code: 1.

Stop using notnil until it's more stable

I found your thread: https://forum.nim-lang.org/t/6301#38854

I'm trying to use getCollectionOrNil but I can't prove to the compiler that the collection is non-nil by using raise within a if col == nil statement (even if I nest it in an else block).

proc getCollection*(query: OpleQuery, name: string): Collection not nil =
  let col = query.database.openCollectionOrNil name
  if col == nil:
    raise newException(Defect, "bad ref")
  else: col

"No locks available" calling mdbx_env_open

[Suite] Basic
[OK] CollectionFlags

[Suite] Database
**** Raising No locks available
/mnt/c/nimdbx/tests/test.nim(27) test
/mnt/c/nimdbx/src/nimdbx/Database.nim(121) openDatabase
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(26) check
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(19) throw

Unhandled exception: No locks available [MDBXError]

[FAILED] create DB
**** Raising Invalid argument
/mnt/c/nimdbx/tests/test.nim(26) test
/mnt/c/nimdbx/src/nimdbx/Database.nim(59) eraseDatabase
/mnt/c/nimdbx/src/nimdbx/Database.nim(52) deleteDatabase
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(26) check
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(19) throw

Unhandled exception: Invalid argument [MDBXError]

[FAILED] Sequences
**** Raising Invalid argument
/mnt/c/nimdbx/tests/test.nim(26) test
/mnt/c/nimdbx/src/nimdbx/Database.nim(59) eraseDatabase
/mnt/c/nimdbx/src/nimdbx/Database.nim(52) deleteDatabase
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(26) check
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(19) throw

Unhandled exception: Invalid argument [MDBXError]

[FAILED] Put and get record
**** Raising Invalid argument
/mnt/c/nimdbx/tests/test.nim(26) test
/mnt/c/nimdbx/src/nimdbx/Database.nim(59) eraseDatabase
/mnt/c/nimdbx/src/nimdbx/Database.nim(52) deleteDatabase
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(26) check
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(19) throw

Unhandled exception: Invalid argument [MDBXError]

[FAILED] Cursors
**** Raising Invalid argument
/mnt/c/nimdbx/tests/test.nim(26) test
/mnt/c/nimdbx/src/nimdbx/Database.nim(59) eraseDatabase
/mnt/c/nimdbx/src/nimdbx/Database.nim(52) deleteDatabase
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(26) check
/mnt/c/nimdbx/src/nimdbx/private/utils.nim(19) throw

Unhandled exception: Invalid argument [MDBXError]

[FAILED] Int Keys
Error: execution of an external program failed: '/mnt/c/nimdbx/tests/bin/test '

you might want to change git submodule from ssh protocol to https as in .git/config [submodule] section and write dynlib as

when defined(windows):
  const libmdbx = "libmdbx.dll"
elif defined(macosx):
  const libmdbx = "libmdbx.dylib"
else:
  const libmdbx = "libmdbx.so"

after make and make install into /usr/local/lib

CI failure on Windows, libmdbx doesn't build in Github Actions

There seems to be a problem with libmdbx building on Windows in a Github Action. I filed erthink/libmdbx#138 .

Until then I'll disable Windows in CI.

mdbx_ symbols not inspectable in IDE

I get Error Type when hovering over mdbx_ symbols in VS Code. Is this expected?

• nim 1.5.1

Test failures on Ubuntu (CRUD, Cursor)

As first reported in #1, and reproduced in a Github Action some tests are failing on Ubuntu.

It looks like a problem serializing/deserializing keys. Not sure what the difference is from macOS that would cause this.