rbehzadan/opennlp-service

RESTful API for Apache OpenNLP Library

opennlp-service's Issues

emoji in text body produces invalid JSON

consider this request:

curl http://localhost -d 'All of my drawing when I was an art major lol smh 😭'

results in this response:

{"sentences":[{"body":"All of my drawing when I was an art major lol smh \uD83D\uDE2D","chunks":[{"body":"All","end":1,"label":"NP","start":0},{"body":"of","end":2,"label":"PP","start":1},{"body":"my drawing","end":4,"label":"NP","start":2},{"body":"when","end":5,"label":"ADVP","start":4},{"body":"I","end":6,"label":"NP","start":5},{"body":"was","end":7,"label":"VP","start":6},{"body":"an art major lol smh \uD83D","end":13,"label":"NP","start":7}],"namedEntities":[],"parses":["(TOP (NP (NP (NP (DT All)) (PP (IN of) (NP (PRP$ my) (NN drawing)))) (SBAR (WHADVP (WRB when)) (S (NP (PRP I)) (VP (VBD was) (NP (DT an) (NN art) (JJ major) (NN lol) (NN smh))))) (. \uD83D\uDE2D)))"],"tokens":[{"body":"All","partOfSpeech":"DT"},{"body":"of","partOfSpeech":"IN"},{"body":"my","partOfSpeech":"PRP$"},{"body":"drawing","partOfSpeech":"NN"},{"body":"when","partOfSpeech":"WRB"},{"body":"I","partOfSpeech":"PRP"},{"body":"was","partOfSpeech":"VBD"},{"body":"an","partOfSpeech":"DT"},{"body":"art","partOfSpeech":"NN"},{"body":"major","partOfSpeech":"JJ"},{"body":"lol","partOfSpeech":"NN"},{"body":"smh","partOfSpeech":"NN"},{"body":"\uD83D","partOfSpeech":"NN"},{"body":"\uDE2D","partOfSpeech":"."}]}]}

which looks good at first. the emoji is correctly encoded in the JSON body attribute as \uD83D\uDE2D, however later on in the chunking process, it breaks the multibyte character in half (maybe because it views it as punctuation or something 🤷‍♂ ) with:

{"body":"an art major lol smh \uD83D","end":13,"label":"NP","start":7}

and so the result that's returned for JSON encoding is just \uD83D which is no longer valid JSON. when trying to parse it with jq or even php's json_decode crashes with error:

parse error: Invalid \uXXXX\uXXXX surrogate pair escape at line 1, column 417

see also related issue: https://stackoverflow.com/a/42461709/4233593