Encoding problem? utf8 string written and retrieved results in half a string padded with zeroes

[venemo]

Encoding problem? utf8 string written and retrieved results in half a string padded with zeroes

[Pumpapa]

To be complete: I've just reinstalled node and all modules from scratch, currently v0.12.7 (v4.0.0 obviously doesn't work yet). My machine is MacOSX Yosemite v10.10.5

[Venemo]

This is interesting. The output file does contain 26 characters (the correct length of the original string), but somehow every second character is a ^@. The interesting part is that console.log can properly output the second string, so at the moment I don't see a reason why fs.write can't.

But here is where it gets totally weird: if I create a Buffer from both s1 and s2 then console.log tells me that the two buffers contain different bytes.

I also tried closing the transaction only after the file has been written, with the same result.

[Venemo]

Oh, note: I use node v0.12.2 on Fedora 22.

[Venemo]

I checked: the bytes returned from LMDB (and that are wrapped by the CustomExternalStringResource class) are exactly what they should be. My current guess is that either V8 or node does something somewhere internally that messes with external string resources.

Even if I return a dummy string from CustomExternalStringResource::data() node fails to put it into a buffer or write it into a file.

[Venemo]

Okay, I've found the problem:
https://github.com/nodejs/node/blob/v0.12.7/src/string_bytes.cc#L331

In JavaScript, strings are UTF-16 so node-lmdb stores them in UTF-16 as well. The above implementation simply uses memcpy to copy bytes. However copying a number of bytes is not the same as copying a number of UTF-16 characters...

It seems that they've fixed it, at least the madness is not present in the master branch:
https://github.com/nodejs/node/blob/master/src/string_bytes.cc#L382

I'm not sure if they still maintain the 0.x branches, but if they do, this might be worth a bugreport in nodejs.

[Pumpapa]

As a workaround I force a copy, which results in the proper string, e.g. by doing ('*'+s).slice(1), as below. I'm unsure why this works, while directly using lmdb's result doesn't, if both strings internally are represented as UTF-16, but it does work.

    var fs = require("fs"),
        lmdb = require('node-lmdb');

    var env = new lmdb.Env();
    env.open({
        path: "./lmdb",
        mapSize: 2*1024*1024*1024, // maximum database size
        maxDbs: 2
    });
    var dbi = env.openDbi({
        name: "moindb",
        create: true
    })

    var nm='name',
        s1 = "This is an ordinary string";
    fs.writeFileSync('1st', s1, {encoding:'utf8'});
    var txnw = env.beginTxn();
    txnw.putString(dbi,nm,s1);
    txnw.commit();
    var txnr = env.beginTxn({ readOnly: true }),
        s2 = txnr.getString(dbi,nm);
    txnr.commit();
    fs.writeFileSync('2nd',('*'+s2).slice(1), {encoding:'utf8'});

[Venemo]

@Pumpapa Take a look at the link I gave. It has working code for strings on the V8 heap (which uses WriteUtf8) and a buggy special case for external string resources (which uses a simple memcpy).

[Pumpapa]

Yeah. I looked at the link but didn't look deeper into the context. I agree that it's a bug on their side which has been resolved in the latest version and which will automatically flow into node-lmdb as soon as that is on par with v4.0.0. I added my work-around only for people who, like me, need something today. Thanks for looking into it though. Like I said, I'm very happy with node-lmdb, which serves me much better than my previous solution (mongodb, which is buggy, frankly).

[Venemo]

@Pumpapa I'm glad to hear that. Stay tuned for node 4.0 support! :)