kim/leveldb-haskell

Haskell bindings to LevelDB (https://github.com/google/leveldb)

Working on it

leveldb-haskell's Issues

Please add support for QuickCheck 2.8

We've updated to QuickCheck 2.8 in NixOS, but unfortunately this means that we can no longer run the leveldb-haskell test suite: http://packdeps.haskellers.com/feed?needle=leveldb-haskell.

Lazy ByteStrings?

I'm still new to Haskell, but I wonder if Lazy Bytestrings can play with the FFI bindings. And if there are any practical blockers preventing such use.

Handle finalizer can run too soon

leveldb-haskell puts a finalizer on the leveldb handle. This makes the pattern open-leveldb-handle >>= \h -> get-leveldb-iterator h >>= \it -> use it work most of the time, except when the GC happens to run before use it completes evaluation.

How about Iterator retains a reference to the handle so the user doesn't need to be aware of this?

iterItems returns items in the reverse order

Hi, currently iterItems is composing a list with (val : acc), which composes items in the reverse order.

It may make actual sense in some cases, may not in other cases. Just giving heads up for anyone who'd stumble as I did.

Broken link

The link http://leveldb.googlecode.com is broken. Perhaps https://github.com/google/leveldb is the up-to-date link?

Enormous heap when iterating over 10million records

I'm getting a very large heap when doing a simple count of all the records in the DB.
I have a little over 10 million records in the DB. Below is the code I'm using:

ldbCount :: DB.DB -> ResourceT IO ()
ldbCount db =
  DB.withIterator db
                  def
                  (\ i -> do
                      let stream = DB.keySlice i DB.AllKeys DB.Desc :: DB.Stream (ResourceT IO) DB.Key
                      totalCount <- DB.foldl' (\c _ -> c + 1) (0::Int) stream
                      liftIO . putStrLn $ show totalCount
                  )

Here is the output when using stack exec -- count db_dir +RTS -s

  6,385,123,872 bytes allocated in the heap
  1,770,246,472 bytes copied during GC
     459,767,960 bytes maximum residency (12 sample(s))
         3,930,216 bytes maximum slop
             845 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0     11952 colls,     0 par    1.856s   1.752s     0.0001s    0.0063s
  Gen  1        12 colls,     0 par    1.408s   2.037s     0.1697s    0.9557s

  INIT    time    0.000s  (  0.000s elapsed)
  MUT     time   15.436s  ( 16.567s elapsed)
  GC      time    3.264s  (  3.789s elapsed)
  EXIT    time    0.012s  (  0.080s elapsed)
  Total   time   18.748s  ( 20.435s elapsed)

  %GC     time      17.4%  (18.5% elapsed)

  Alloc rate    413,651,455 bytes per MUT second

  Productivity  82.6% of total user, 81.5% of total elapsed

The database itself is only 2.5G. Could there be a memory leak of some kind that allows the heap to grow so large or have I made a naive assumption in the implementation?

Any plan to support comparator?

I've only took a brief look, and it seems that comparator api is in the low level C interface, but missing from the Haskell API. Do you guys have any plan?

LevelDB crashes ghci

LevelDB consistently crashes ghci on my system. (Mac OS X 10.6.8 running ghc 7.0.4). Note that LevelDB commands work fine when compiled into a binary by ghc.

$ ghci
GHCi, version 7.0.4: http://www.haskell.org/ghc/ :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package ffi-1.0 ... linking ... done.
package flags have changed, resetting and loading new packages...
Prelude> :m Database.LevelDB
Prelude Database.LevelDB> withLevelDB "/tmp/ldb" [CreateIfMissing] $ \db -> return ()
Loading package bytestring-0.9.1.10 ... linking ... done.
Loading package filepath-1.2.0.0 ... linking ... done.
Loading package leveldb-haskell-0.0.3 ... linking ... done.
Segmentation fault

Hackage Docs missing

Hi

First of all, thanks for the nice package!

I just wanted to report that on the package's hackage homepage, the module listing does not link to any documentation, as previous versions did. It would be good if you could fix that.

Thanks!

Support bindings to CompactRange

It would be very nice of the library had a binding to CompageRange.

See: https://github.com/google/leveldb/blob/803d69203a62faf50f1b77897310a3a1fcae712b/include/leveldb/db.h#L141

Stream merging and grouping

FYI https://gist.github.com/robinp/fa14b51425f010454a5ea5b1c9f220ce has stream merging and grouping operators. I don't have time to make this a proper PR with test, but if anyone is interested feel free to.

Segfault when using an already closed database

The following program terminates with a segfault:

import qualified Database.LevelDB.Base as LevDb
import qualified Data.ByteString.Char8 as BSC

main =
  do db <- LevDb.open "test.db" (LevDb.defaultOptions {
                                            LevDb.createIfMissing = True
                                          })
     LevDb.close db
     let k = BSC.pack "KEY"
         v = BSC.pack "VALUE"
     LevDb.put db LevDb.defaultWriteOptions k v
     putStrLn "done"

I would expect an exception when doing a put on an already closed database.

I could go ahead and fix the problem. My plan is:

• Internal.hs no longer exports the constructors of the DB type
• The DB type contains an additional MVar tracking the state of the connection (open/closed)
• Functions that now use the LevelDbPtr directly instead use an accessor function for getting the DB handle. This accessor function is implemented in Internal.hs

The accessor function is a bit tricky because it has to deal with concurrency. It's not sufficient to allow only one client at a time accessing the connection because leveldb allows concurrent operations on the same connection. But the close function needs exclusive access to the connection and all further operations invoked on the connection should fail afterwards.

I think I can come up with a solution to this problem. But I want to hear your opinions first.

ghci and runghc fails

I'm trying to debug why ghci behaves differently than ghc. Here is a succinct example with the corresponding output https://gist.github.com/57df6f71fa1144ec20cf

• Loading the file within ghci and running main outputs: "Bus error: 10" after a few seconds
• Running the file with runhaskell or runghc silently fails after a few seconds
• Compiling with ghc works dandy.

(I edited the issue to embed the gist contents in case the gist gets removed somehow)

This isn't a huge priority, but it would be very nice to test leveldb code without having to compile the source every time.

-- in LevelDbTest.hs
import Database.LevelDB
import Data.Text as T
import Data.Text.Encoding as E
import Data.ByteString (ByteString)

main :: IO ()
main = do 
  withLevelDB "/tmp/leveldbtest" [CreateIfMissing, CacheSize 1024] $ \db -> do
    put db [] (encode "foo") (encode "bar")
    get db [] (encode "foo") >>= print


encode :: String -> ByteString
encode string = E.encodeUtf8 $ T.pack string

$ ghci
GHCi, version 7.0.4: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package ffi-1.0 ... linking ... done.
Prelude> :load LevelDbTest.hs 
[1 of 1] Compiling Main             ( LevelDbTest.hs, interpreted )
Ok, modules loaded: Main.
*Main> main
Loading package bytestring-0.9.1.10 ... linking ... done.
Loading package array-0.3.0.2 ... linking ... done.
Loading package filepath-1.2.0.0 ... linking ... done.
Loading package leveldb-haskell-0.0.2 ... linking ... done.
Loading package deepseq-1.3.0.0 ... linking ... done.
Loading package text-0.11.1.13 ... linking ... done.
Bus error: 10

$ runhaskell LevelDbTest.hs # same pause as ghci but silently finishes

$ ghc --make LevelDbTest.hs 
[1 of 1] Compiling Main             ( LevelDbTest.hs, LevelDbTest.o )
Linking LevelDbTest ...
ld: warning: could not create compact unwind for .LFB3: non-standard register 5 being saved in prolog

$ ./LevelDbTest # correct output
Just "bar"

RocksDB segfaults on many puts

Pretty much any of the examples can fail when multiple sequential puts are performed (at least on Mac OS X).
For example, adding this fragment to the comparator example and running the compiled code 3-5 times reproduces this issue:

    let write = uncurry $ put db def
    let input = [(bs, bs) | x <- ['a' .. 'z'], let bs = singleton x]
    mapM_ write input

The full example code then would be:

module Main where

import           Control.Monad.IO.Class     (liftIO)
import           Data.Default
import           Database.LevelDB
import qualified Database.LevelDB.Streaming as S
import Data.ByteString.Char8 (singleton)


customComparator :: Comparator
customComparator = Comparator compare

main :: IO ()
main = runResourceT $ do
    db <- open "/tmp/lvlcmptest"
               defaultOptions{ createIfMissing = True
                             , comparator = Just customComparator
                             }

    put db def "zzz" ""
    put db def "yyy" ""
    put db def "xxx" ""

    let write = uncurry $ put db def

    let input = [(bs, bs) | x <- ['a' .. 'z'], let bs = singleton x]
    mapM_ write input

    withIterator db def $ \iter -> liftIO $
            S.toList (S.entrySlice iter S.AllKeys S.Asc)
        >>= print

    return ()

And this is the response I get from running this code (as you can see it star failing after couple of runs):

$ leveldb-example-comparator
[("a","a"),("b","b"),("c","c"),("d","d"),("e","e"),("f","f"),("g","g"),("h","h"),("i","i"),("j","j"),("k","k"),("l","l"),("m","m    "),("n","n"),("o","o"),("p","p"),("q","q"),("r","r"),("s","s"),("t","t"),("u","u"),("v","v"),("w","w"),("x","x"),("xxx",""),("y    ","y"),("yyy",""),("z","z"),("zzz","")]
$ leveldb-example-comparator
[("a","a"),("b","b"),("c","c"),("d","d"),("e","e"),("f","f"),("g","g"),("h","h"),("i","i"),("j","j"),("k","k"),("l","l"),("m","m    "),("n","n"),("o","o"),("p","p"),("q","q"),("r","r"),("s","s"),("t","t"),("u","u"),("v","v"),("w","w"),("x","x"),("xxx",""),("y    ","y"),("yyy",""),("z","z"),("zzz","")]
$ leveldb-example-comparator
[("a","a"),("b","b"),("c","c"),("d","d"),("e","e"),("f","f"),("g","g"),("h","h"),("i","i"),("j","j"),("k","k"),("l","l"),("m","m    "),("n","n"),("o","o"),("p","p"),("q","q"),("r","r"),("s","s"),("t","t"),("u","u"),("v","v"),("w","w"),("x","x"),("xxx",""),("y    ","y"),("yyy",""),("z","z"),("zzz","")]
$ leveldb-example-comparator
[1]    63389 segmentation fault
$ leveldb-example-comparator
leveldb-example-comparator: schedule: re-entered unsafely.
   Perhaps a 'foreign import unsafe' should be 'safe'?
[1]    63400 segmentation fault
$ leveldb-example-comparator
leveldb-example-comparator: schedule: re-entered unsafely.
   Perhaps a 'foreign import unsafe' should be 'safe'?

Custom comparator unsafe re-entry

Working with a custom comparator I get the following error, which also causes a segfault:

comparator: schedule: re-entered unsafely.
   Perhaps a 'foreign import unsafe' should be 'safe'?

I'm using GHC 7.8.4 and leveldb-haskell 0.6.1.

The following code (slightly modified from the example "comparator.hs") is sufficient to display the issue:

{-# LANGUAGE OverloadedStrings #-}

-- | Demo custom comparator

module Main where

import           Control.Monad
import           Control.Monad.IO.Class     (liftIO)
import qualified Data.ByteString.Char8      as B
import           Data.Default
import           Database.LevelDB
import qualified Database.LevelDB.Streaming as S


customComparator :: Comparator
customComparator = Comparator compare

main :: IO ()
main = runResourceT $ do
    db <- open "/tmp/lvlcmptest"
               defaultOptions{ createIfMissing = True
                             , comparator = Just customComparator
                             }

    forM_ [1..1000000::Int] $ \i -> do
        put db def (B.pack $ "zzz" ++ show i) ""

    withIterator db def $ \iter -> liftIO $
            S.toList (S.entrySlice iter S.AllKeys S.Asc)
        >>= print

    return ()

The problem only manifests itself when I issue a large number of puts in rapid succession.

Do you have any idea about what could be causing this?

Build error

http://hydra.cryp.to/build/202371/nixlog/2/raw

Release to Hackage

Would you mind uploading a new release? I'd like to get #32 into the wild.