Error in output json

[cdqa-suite]

Error in output json

[fmikaelian]

@mamrou

The issue was due to the fact that JSON.stringify(this.json) (function that is supposed to convert a JSON object to a JSON string) converts special characters like - or " into characters that are not JSON compliant. To avoid this I followed the SO links above to create a computed property named valid_json that post-process the JSON file after JSON.stringify() by replacing non-compliant characters by their unicode equivalents. This makes the JSON file valid:

    valid_json: function () {
      var json = JSON.stringify(this.json).replace(/[\u007F-\uFFFF]/g, function(chr) {
          return "\\u" + ("0000" + chr.charCodeAt(0).toString(16)).substr(-4)
      })
      return json
    }

[fmikaelian]

Hi @mamrou

• What code did you use to load the json file?
• Can you load the original input json file (before annotations)?

[mamrou]

Hi @fmikaelian

• a classic with open('output.json') as json_file: bnp_data = json.load(json_file)
• Yes, there is no issue when loading the original input file before annotations.

[fmikaelian]

It seems it is an encoding error due to the followinf character: http://www.hexadecimaldictionary.com/hexadecimal/0xAC/. Maybe the annotator does not encode the question / answers with the correct format. We could compare squad json and our custom annotated json types to see where we are wrong?

[fmikaelian]

If you load the original json, do nothing and export it directly, can you read the output?

[mamrou]

Same encoding error message as above.

[fmikaelian]

Ok. Could you run a diff between original file and output file (without annotations)?

See https://stackoverflow.com/questions/30139597/visual-studio-code-is-there-a-compare-feature-like-that-plugin-for-notepad

[mamrou]

I run a diff between the two json to spot potential tracks of failure. What I can say is that the encoding of some specific punctuations or characters (like `, ' or €) may be at the origin of the issue.
I will clean the original json from all those special characters and punctuations and keep you posted about the result.

[fmikaelian]

Could we just encode the output json with the same format as input directly in the annotator? Do you know what is the encoding format of the input SQuAD? Maybe we can find this info online?

[fmikaelian]

vscode report the following error for many lines in the output json: Invalid characters in string. Control characters must be escaped.json(262)

[fmikaelian]

• https://stackoverflow.com/questions/31649362/json-stringify-and-unicode-characters
• https://stackoverflow.com/questions/4901133/json-and-escaping-characters