Handling of non-BMP Unicode characters

[caligatio]

Handling of non-BMP Unicode characters

[cjerdonek]

By the way, a related question is whether the high and low surrogate code points in the BMP are handled correctly:

The High Surrogates (U+D800–U+DBFF) and Low Surrogate (U+DC00–U+DFFF) codes are reserved for encoding non-BMP characters in UTF-16 by using a pair of 16-bit codes: one High Surrogate and one Low Surrogate. A single surrogate code point will never be assigned a character.

Are there any test cases for these code points?

[cjerdonek]

Okay, here is a test case using a non-BMP Unicode character. The test case is for SHA-256 using the default encoding UTF8. Sample Javascript code looks like the following:

// With 'TEXT', the jsSHA constructor defaults to encoding 'UTF8'.
var shaObj = new jsSHA(s, 'TEXT');
return shaObj.getHash('SHA-256', 'HEX');

When I try the string '\ud83d\ude00', which is the Unicode Character 'GRINNING FACE' (U+1F600), I get the following:

ea645893a930e0872284cf8fbc8d14f66f052aea0bc8d9c4829f08a38219940c

But I believe one should instead get:

f0443a342c5ef54783a111b51ba56c938e474c32324d90c3a60c9c8e3a37e2d9

Here is Python code I used to get the expected value (tested with Python 2.7.6):

import hashlib
import json

def sha(bytes):
    return hashlib.sha256(bytes).hexdigest()

def test(u):
    print(json.dumps(u))
    b = u.encode('utf-8')
    print(repr(b))
    print(sha(b))
    print

# Unicode Character 'GRINNING FACE' (U+1F600)
# http://www.fileformat.info/info/unicode/char/1F600/index.htm
test(u"\U0001F600")
# Gives:
# JSON:    "\ud83d\ude00"
# UTF-8:   '\xf0\x9f\x98\x80'
# SHA-256: f0443a342c5ef54783a111b51ba56c938e474c32324d90c3a60c9c8e3a37e2d9

[cjerdonek]

Any acknowledgment of this issue?

I don't think it would be too hard to fix. The Mozilla page linked in the original comment has code that can probably be used close to as is.

[Caligatio]

Sorry for the delay here; I have had limited computer access for the last
few weeks. I found a code snippet that should fit in nicely with my string
parsing but haven't had a chance to actually try it. If you have a patch
handy, that would speed things up.

On Tuesday, September 23, 2014, Chris Jerdonek [email protected]
wrote:

Any acknowledgment of this issue?

I don't think it would be too hard to fix. The Mozilla page linked in the
original comment has code that can probably be used close to as is.

—
Reply to this email directly or view it on GitHub
#21 (comment).

[cjerdonek]

Thanks, @Caligatio. If you already have something in mind, I'll let you take care of it. There's no rush (at least for me). I just wasn't sure if you had seen the issue.

[Caligatio]

I believe this should be fixed in https://github.com/Caligatio/jsSHA/blob/db68ec213c17f34eb105e9561b55045c0f7cbc65/src/sha_dev.js

Can you try out your tests to see if you come to the same conclusion?

[cjerdonek]

Thanks! But can you make it easier to test? sha_dev.js is ignored in your bower.json, so I don't get the change when I bower install. Or else you could recompile the other files. Thanks.

[Caligatio]

All files should now be synced up.

[cjerdonek]

I just had one test for this case, but it passes now. Thanks!