Behavior in Nativescript

[abdallahkadour]

Behavior in Nativescript

[hildjj]

Every time I have to touch anything related to NativeScript, nothing ever works the way it did a month or two ago. After trying for over an hour, I couldn't get anything to build this time. :(

Can you show me the output of console.log(object), console.log(encoded), and console.log(JSON.stringify(decoded)) as well, please?

If we can't figure this out remotely, I'll probably need some help to create an environment that compiles.

[abdallahkadour]

Of course

CONSOLE LOG:  the object {
"x": "val_x",
"y": "val_y"
}
CONSOLE LOG: the encoded object: �axeval_xayeval_y
CONSOLE LOG: the decoded object: {"¢axeval_xayeval_y": "¢axeval_xayeval_y"}
CONSOLE LOG: the JSON string of decoded:  {"¢axeval_xayeval_y":"¢axeval_xayeval_y"}

[abdallahkadour]

I want to mention that I am only using the decode function. It fails when decoding a cbor which I can parse problemless using other tools.

[hildjj]

It looks like enocde() is return a string, which i quite odd. I wonder what buffer implementation you're getting. What is console.log(encoded.toString('hex'))? How about console.log(encoded[0])?

[abdallahkadour]

CONSOLE LOG: hex a261786576616c5f7861796576616c5f79
CONSOLE LOG: encoded[0] 162

[hildjj]

ok, so encode is working correctly, but that's a really odd default toString routine for your buffer class. Does require.resolve("buffer/") return something like ".../node_modules/buffer/index.js"?

Can you try console.log(await cbor.diagnose(encoded)).

[hildjj]

Oh, I see. That is the default toString for the external buffer module.

Wait... are you sure CONSOLE LOG: the decoded object: {"¢axeval_xayeval_y": "¢axeval_xayeval_y"} is actually the output of cbor.decode? ¢ is U+00A2, and a is U+0061, which matches some sort of decoding of encoded as a string, rather than as CBOR.

[abdallahkadour]

Yes I am sure this is my code:

import process from 'process'
global.process = process
import prereqs from 'cbor-rn-prereqs'
import cbor from 'cbor'

let object = {
      x: 'val_x',
      y: 'val_y'
    }

    let encoded = cbor.encode(object)
    let decoded = cbor.decode(encoded)

    console.log('hex', encoded.toString('hex'))
    console.log('encoded[0]', encoded[0])
    console.log('the object', object)
    console.log('the encoded object', encoded)
    console.log('the decoded object', decoded)
    console.log('JSON string', JSON.stringify(decoded))

[abdallahkadour]

the diagnose log: CONSOLE LOG: diagnose {"¢axeval_xayeval_y": "¢axeval_xayeval_y", "¢axeval_xayeval_y": "¢axeval_xayeval_y"}

[hildjj]

At this point, I probably need to play with it myself in a debugger. Could you create a small github repo that has your code in a working environment for me, please?

[abdallahkadour]

Of course: debug-repo

[hildjj]

OK, I've found the problem and will think about a work-around. in NS, TextDecoder exists now, but it doesn't deal with slices of Buffers correctly, instead decoding the whole original Buffer.

[hildjj]

To see the problem without 3rd party bits:

const td = new TextDecoder('utf8', {fatal: true, ignoreBOM: true})
const buf = new Uint8Array([97, 98, 99, 100]);
const s = new Uint8Array(buf.buffer, buf.byteOffset + 1, 1);
const dec = td.decode(s)
console.log('utf8', buf, s, dec)
// utf8 97,98,99,100 98 abcd

(expected "b" instead of "abcd")

[hildjj]

Here is the error: https://github.com/NativeScript/NativeScript/blob/17c85107ba84953630b0471c1f6f3d68f6d59f76/packages/core/text/text-common.ts#L70

That function as it currently exists would need a lot of work to merely be wrong.

[hildjj]

OK, here is an imperfect workaround. npm i @cto.af/textdecoder. Then at the top, change to:

import TextDecoder  from "@cto.af/textdecoder/polyfill";
global.TextDecoder = TextDecoder;
import {fixes} from "cbor-rn-prereqs";
import * as cbor from "cbor";
import * as utils from "cbor/lib/utils.js"

const td = new TextDecoder('utf8', {fatal: true, ignoreBOM: true})
utils.utf8 = buf => td.decode(buf)

Unfortunately, the top level TextDecoder instance in cbor is resolved at compile time, so replacing the utf8 function is the quickest way to get you up and running. I did a release of @cto.af/textdecoder today that had a full and tested UTF-8 implementation, so that's what you're swapping in.

I would also recommend filing a bug against NativeScript to have them fix their quite broken implementation. It doesn't decode non-ASCII UTF-8, and doesn't deal with slices.

[abdallahkadour]

@hildjj Thanks for your Info. Before I opened this issue, I tried setting the variable as follow global.TextDecoder = util.TextDecoder but TextDecoder is read only. Did I miss something? I will try your approach monday.

[hildjj]

util.TextDecoder only exists in node 10, I think. Setting global.TextDecoder isn't really needed for cbor, but I left it there in case you're using it somewhere else.

I'll see what I can do to fix this in cbor, but short of running a set of unit tests at runtime, I'm not sure how to work around an existing-but-broken TextDecoder implementation. I might be able to make it easier to replace, at least.

[hildjj]

Did we get this solved adequately?

[hildjj]

I think we came to an adequate solution. File more bugs if you continue to have problems with NativeScript being broken.