ray2501/tcl-lmdb

The Tcl interface to the Lightning Memory-Mapped Database

tcl-lmdb's Issues

Makefile does not account for cross-compiling

tcl-lmdb/Makefile.in

Lines 152 to 161 in 992c681

	# Workaround for glibc pthread robust mutex support (glibc < 2.12) fix
	ifneq ("$(OS)","Windows_NT")
	UNAME_S := $(shell uname -s)
	ifeq ($(UNAME_S),Linux)
	GLIBCCHECK := $(shell expr `ldd --version | grep ^ldd | sed 's/^.* //g'` \< 2.12)
	ifeq "$(GLIBCCHECK)" "1"
	PKG_CFLAGS += -DMDB_USE_ROBUST=0
	endif
	endif
	endif

This assumes that because one is building ON Linux that one is building FOR Linux, this does not hold true when cross-compiling.

Segfault when you use an env without opening it

I was trying out tcl-lmdb (thanks for making it!) and ran into a segfault issue. Details follow.

Code to reproduce

package require lmdb 0.3.5
set e [lmdb env]
set db [lmdb open -env $e]

Versions

• Commit: 9ac936b
• Tcl 8.6.7
• OS: openSUSE Tumbleweed 20171117 x86_64

Shell transcript

> tclsh segfault-lmdb.tcl 
fish: “tclsh segfault-lmdb.tcl” terminated by signal SIGSEGV (Address boundary error)
> valgrind tclsh segfault-lmdb.tcl 
==28436== Memcheck, a memory error detector
==28436== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==28436== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==28436== Command: tclsh segfault-lmdb.tcl
==28436== 
==28436== Invalid read of size 4
==28436==    at 0x6339041: mdb_txn_renew0 (in /usr/lib64/tcl/lmdb0.3.5/liblmdb0.3.5.so)
==28436==    by 0x633A373: mdb_txn_begin (in /usr/lib64/tcl/lmdb0.3.5/liblmdb0.3.5.so)
==28436==    by 0x6334C71: LMDB_MAIN (in /usr/lib64/tcl/lmdb0.3.5/liblmdb0.3.5.so)
==28436==    by 0x4E7E795: TclNRRunCallbacks (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4E805D9: ??? (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4F4AED8: ??? (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4E8049F: ??? (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4F37577: Tcl_FSEvalFileEx (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4F3ED2A: Tcl_MainEx (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x108933: ??? (in /usr/bin/tclsh8.6)
==28436==    by 0x5213F49: (below main) (libc-start.c:308)
==28436==  Address 0x7c is not stack'd, malloc'd or (recently) free'd
==28436== 
==28436== 
==28436== Process terminating with default action of signal 11 (SIGSEGV)
==28436==  Access not within mapped region at address 0x7C
==28436==    at 0x6339041: mdb_txn_renew0 (in /usr/lib64/tcl/lmdb0.3.5/liblmdb0.3.5.so)
==28436==    by 0x633A373: mdb_txn_begin (in /usr/lib64/tcl/lmdb0.3.5/liblmdb0.3.5.so)
==28436==    by 0x6334C71: LMDB_MAIN (in /usr/lib64/tcl/lmdb0.3.5/liblmdb0.3.5.so)
==28436==    by 0x4E7E795: TclNRRunCallbacks (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4E805D9: ??? (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4F4AED8: ??? (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4E8049F: ??? (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4F37577: Tcl_FSEvalFileEx (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x4F3ED2A: Tcl_MainEx (in /usr/lib64/libtcl8.6.so)
==28436==    by 0x108933: ??? (in /usr/bin/tclsh8.6)
==28436==    by 0x5213F49: (below main) (libc-start.c:308)
==28436==  If you believe this happened as a result of a stack
==28436==  overflow in your program's main thread (unlikely but
==28436==  possible), you can try to increase the size of the
==28436==  main thread stack using the --main-stacksize= flag.
==28436==  The main thread stack size used in this run was 8388608.
==28436== 
==28436== HEAP SUMMARY:
==28436==     in use at exit: 1,370,532 bytes in 100 blocks
==28436==   total heap usage: 212 allocs, 112 frees, 2,414,982 bytes allocated
==28436== 
==28436== LEAK SUMMARY:
==28436==    definitely lost: 0 bytes in 0 blocks
==28436==    indirectly lost: 0 bytes in 0 blocks
==28436==      possibly lost: 1,285,280 bytes in 72 blocks
==28436==    still reachable: 85,252 bytes in 28 blocks
==28436==         suppressed: 0 bytes in 0 blocks
==28436== Rerun with --leak-check=full to see details of leaked memory
==28436== 
==28436== For counts of detected and suppressed errors, rerun with: -v
==28436== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
fish: “valgrind tclsh segfault-lmdb.tcl” terminated by signal SIGSEGV (Address boundary error)

debug build against non-debug Tcl weird behaviour

So, I nailed down the environment where I can reproduce the failure to this:

./configure --enable-symbols --with-system-lmdb --with-tcl=/usr/local/lib/tcl8.6/

I see this failure:

# cat test.tcl
puts [info patchlevel]
set dir [file dirname [info script]]
load [file join $dir liblmdb0.3.4.so]
package require lmdb

set myenv [lmdb env]
$myenv open -path $dir -mode 0664

# tclsh8.6 test.tcl
8.6.5
ERROR: No such file or directory
    while executing
"$myenv open -path $dir -mode 0664"
    (file "test.tcl" line 7)
[email protected]:~/github/tcl-lmdb # 

Which I nailed down to this snippet.

1371│       for(i=2; i+1<objc; i+=2){
1372│         zArg = Tcl_GetStringFromObj(objv[i], 0);
1373│         if( strcmp(zArg, "-path")==0 ){
1374│             path = Tcl_GetStringFromObj(objv[i+1], 0);
1375│         } else if( strcmp(zArg, "-mode")==0 ){
1376│             if(Tcl_GetIntFromObj(interp, objv[i+1], (int *)&mode) != TCL_OK) {
1377│                 return TCL_ERROR;
1378│             }                                                                                                                                                  
1379├>        }

The for loop is entered twice, the first time to parse the -path "." arguments pair, the second time to parse the -mode 0664 arguments pair.

path is nice until the call at 1376 returns, then it's junk. Here's an extract from my debugging session:

Breakpoint 5, LMDB_ENV (cd=0x0, interp=0x801c30010, objc=6, objv=0x801c632f0) at ./generic/tclmdb.c:1374                                                             
(gdb) next                                                                                                                                                           
(gdb) p path                                                                                                                                                         
$3 = 0x801ce4810 "."                                                                                                                                                 
(gdb) c                                                                                                                                                              
Continuing.                                                                                                                                                          

Breakpoint 6, LMDB_ENV (cd=0x0, interp=0x801c30010, objc=6, objv=0x801c632f0) at ./generic/tclmdb.c:1376                                                             
(gdb) next                                                                                                                                                           
(gdb) p path                                                                                                                                                         
$4 = 0x801ce0000 "\001\b"   

I am not able to explain this yet.

UTF-8 character stripped

The result of this script

package require lmdb

set key {Linn‘us}
set val {1707-05-23}

# Open
set myenv [lmdb env]
$myenv open -path [file dirname [info script]]
set mydbi [lmdb open -env $myenv]

# Write
set txn [$myenv txn]
$mydbi put $key $val -txn $txn
$txn commit
$txn close

# Read
set txn [$myenv txn -readonly 1]
set cur [$mydbi cursor -txn $txn]
lassign [$cur get -next] rkey rval
$cur close
$txn abort
$txn close

# Close
$mydbi close -env $myenv
$myenv close

# Compare
if {$rkey ne $key || $rval ne $val} {
    puts stderr "Expected |$key, $val|, got |$rkey, $rval|"
}

is

Expected |Linn‘us, 1707-05-23|, got |Linnus, 1707-05-23|

mdb_dump -p says

VERSION=3
format=print
type=btree
mapsize=1048576
mapaddr=0x80064f000
maxreaders=126
db_pagesize=4096
HEADER=END
 Linn\18us
 1707-05-23
DATA=END

Just dumping it here until I figure out whether it's a tcl-lmdb or pure lmdb issue.

[$env open] -fidexmap default

Hi Danilo,

I see that the -fixedmap param to env open defaults to true.
https://github.com/ray2501/tcl-lmdb/blob/master/generic/tclmdb.c#L1363
This is very risky as mmap(2) doesn't guarantee that the address hint is respected.
As you only open the env once - when doing $env open - and copy data in/out via Tcl_Objs, I wonder why you chose to opt for having this default to true.

If there's no compelling reason, I think it would be better to change the default to false.

Thanks!