[Node Host SDK | Host function] the value returned by a host function is truncated if I use emojis

[extism]

[Node Host SDK | Host function] the value returned by a host function is truncated if I use emojis

[bhelx]

I think this is the problem:

> let str = "hello 🌎"
undefined
> str.length
8
> Buffer.byteLength(str)
10

String#length in javascript counts the characters and not the bytes.

I think you need to change this:

  var offs = currentPlugin.memoryAlloc(helloMessage.length)

to this:

  var offs = currentPlugin.memoryAlloc(Buffer.byteLength(helloMessage))

Could you try that?

Did you get this from an example? We probably should change this in other places or start thinking about some abstractions that can help prevent these kinds of footguns.

[k33g]

Buffer.byteLength(helloMessage) does the job 👍 thank you 🙇

My sample is a mix between the doc and a chat in the Discord https://discord.com/channels/1011124058408112148/1050088035691270204/1066403189211017257

I think we can close the issue

[syke99]

I agree with @bhelx Emojis are stored in Unicode code points, and some are comprised of multiple Unicode points. So when working with emojis in strings and setting byte arrays to the length of said string, the only safe way to do so with Buffer.byteLength()

[bhelx]

Right, I think we should always use byteLength regardless of what encoding we think the string is because it works for all encodings. This is an easy mistake to make though, and we made it when writing this library: #82

Next steps would be to agree on a slightly higher level API for host functions to help prevent these kind of footguns. Perhaps we could have a method kind of like this:

class Plugin {
  allocBytes(data: Uint8Array | String): number {
      let offs = this.memoryAlloc(Buffer.byteLength(data))
      this.memory(offs).write(data)
      return offs
  }
}

We'd like to take some time and come up with something consistent for all the SDKs though, so for now best to just try to document this better.