robotframework-thailand/robotframework-jsonlibrary

Robotframework Test Library to manipulate JSON using JSONPath

robotframework-jsonlibrary's Issues

Parse error at 1:3 near token ? (?)

my json exp: $.[?(@.bundle_id=='10')].oper_status
my jsonobject is:

{
       "number_of_lag_in_use": 2,
       "number_of_aggregators": 2,
       "interfaces": {
          "Port-channel10": {
             "name": "Port-channel10",
             "bundle_id": 10,
             "protocol": "lacp",
             "flags": "SU",
             "oper_status": "up",
             "members": {
                "TenGigabitEthernet1/0/39": {
                   "interface": "TenGigabitEthernet1/0/39",
                   "flags": "P",
                   "bundled": true,
                   "port_channel": {
                      "port_channel_member": true,
                      "port_channel_int": "Port-channel10"
                   }
                },
                "TenGigabitEthernet1/0/40": {
                   "interface": "TenGigabitEthernet1/0/40",
                   "flags": "P",
                   "bundled": true,
                   "port_channel": {
                      "port_channel_member": true,
                      "port_channel_int": "Port-channel10"
                   }
                }
             },
             "port_channel": {
                "port_channel_member": true,
                "port_channel_member_intfs": [
                   "TenGigabitEthernet1/0/39",
                   "TenGigabitEthernet1/0/40"
                ]
             }
          },
          "Port-channel34": {
             "name": "Port-channel34",
             "bundle_id": 34,
             "protocol": "lacp",
             "flags": "SD",
             "oper_status": "down",
             "members": {
                "TenGigabitEthernet1/0/3": {
                   "interface": "TenGigabitEthernet1/0/3",
                   "flags": "D",
                   "bundled": false,
                   "port_channel": {
                      "port_channel_member": true,
                      "port_channel_int": "Port-channel34"
                   }
                }
             },
             "port_channel": {
                "port_channel_member": true,
                "port_channel_member_intfs": [
                   "TenGigabitEthernet1/0/3"
                ]
             }
          }
       }
    }   

Validate Schema: Make error messages more descriptive

When Validate Json By Schema throws a jsonschema.ValidationError it is hard to know where or what exactly the Problem is.
example:
Json does not match the schema: {'type': 'object', 'enum': ['SUCCESS', 'ERROR', 'UNDEFINED', 'BUSINESS_ERROR']}
there is no Info what exactly the problem is.

I would propose to change the error message from
except jsonschema.ValidationError as e: fail(f"Json does not match the schema: {e.schema}")
to
except jsonschema.ValidationError as e: fail(f"{e.message}, Schema path: {' > '.join(e.schema_path)}")

So the error in the example would instead read:
'SUCCESS' is not one of ['ERROR', 'UNDEFINED', 'BUSINESS_ERROR'], Schema path: properties > state > enum

which makes more sense to me. What do you think?

Wrong character encoding for UTF-8

Function convert_json_to_string returns wrong encoding if you use UTF-8. I have workaround:

` @staticmethod
def convert_json_to_string(json_object):
"""Convert JSON object to string

    Arguments:
        - json_object: json as a dictionary object.

    Return new json_string

    Examples:
    | ${json_str}=  |  Convert JSON To String | ${json_obj} |
    """
    return json.dumps(json_object, **ensure_ascii=False**)`

Add a license

Could you add a license, please?

Keywords not found by intellibot

Issue with Pycharm, Intellibot plugin and the keyword decorator that have single quoted keyword name.
eg. @Keyword('Add Object To Json')
Intellibot seems to have issues with the decorators as defined in the robotframework-jsonlibrary.
It will only find single quoted keyword names, but not the unquoted keyword names when the @Keyword('keyword name') is used.
At runtime robot only works with the unquoted Keywords.
If I comment out the decorators in my copy of robotframework-jsonlibrary, IntelliBot finds the unquoted keywords fine.

PyCharm 2019.1.2 (Community Edition)
Build #PC-191.7141.48, built on May 7, 2019
JRE: 11.0.2+9-b159.56 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Windows 10 10.0
IntelliBot @SeleniumLibrary Patched v0.10.143.381
robotframework-jsonlibrary 0.3
Python 3.7.3

Travis CI no longer running

It looks like Travis CI is not used any more for validating and delivering this repo.

The company behind Travis CI stop supporting open source projects. See here https://www.reddit.com/r/linux/comments/k92tc0/travis_is_no_longer_free_for_opensource_projects/

A new CI solution will be necessary.. if someone is up to the challenge.

Update Value To Json backwards incompatibile change

Now Update Value To Json is not in place update like it was before, but exposes updated json as retuned value. Rough closed PRs reviews let me think that previous behavior was considered as bug. From my perspective it was designed correctly. Like robot collections library keyword Set To Dictionary is not returning new dictionary, but in-place updates given dict.

Apart from that, such change should be at least marked as backwards incompatible change as has huge impact on already created test scripts.

Using wrong syntax for 'greater than or equal' operator breaks keyword 'Get Value From Json'

Example of misstyped 'greater than or equal' operator (take NOTE of price=>8.99 which is wrong - it should be >= - nevertheless it breaks the keyword)

Get all books with price biger than or equal to 8.99
    [Documentation]    `?()`	applies a filter (script) expression.
    Get Value From Json    ${bookstore_data}    $..book[?(@.price=>8.99)]

error

Parse error at 1:18 near token > (FILTER_OP)	
15:34:05.578	DEBUG	Traceback (most recent call last):
  File "c:\python_virtual_envs\api_requests\lib\site-packages\JSONLibrary\JSONLibraryKeywords.py", line 77, in get_value_from_json
    json_path_expr = parse(json_path)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw_ext\parser.py", line 179, in parse
    return ExtentedJsonPathParser(debug=debug).parse(path)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 32, in parse
    return self.parse_token_stream(lexer.tokenize(string))
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 55, in parse_token_stream
    return new_parser.parse(lexer = IteratorToTokenStream(token_iterator))
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 69, in p_error
    raise Exception('Parse error at %s:%s near token %s (%s)' % (t.lineno, t.col, t.value, t.type))

Same applies in the other direction with misstyped 'less than or equal' operator =<

Parse error at 1:18 near token < (FILTER_OP)

pypi release is broken

robotframework-jsonlibrary release 0.2 is broken on pypi - but looks like relative imports are fixed on github master... can you push the fix out as new release to pypi please? :-)

$ pip install robotframework-jsonlibrary
Collecting robotframework-jsonlibrary
Using cached https://files.pythonhosted.org/packages/e0/3c/609d90b967905c0b5c90f736666531e025756e71f3d538546cabff61ef3e/robotframework-jsonlibrary-0.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-ct5ux1vz/robotframework-jsonlibrary/setup.py", line 5, in
from JSONLibrary.version import VERSION
File "/tmp/pip-build-ct5ux1vz/robotframework-jsonlibrary/JSONLibrary/init.py", line 2, in
from JSONLibraryKeywords import JSONLibraryKeywords
ModuleNotFoundError: No module named 'JSONLibraryKeywords'

----------------------------------------

site with keyword documentation gives 404

from the link in the README.md

Tox == 3.0 dependency

Would it be possible to either remove dependency on specific version of tox, or make it less strict (e.g. >=) ?

robotframework-jsonlibrary/setup.py

Line 17 in b834e5d

'tox==3.0.0',

Now, if newer version of tox is installed in environment, pip install robotframework-jsonlibrary reverts it to this particular version, and this looks confusing and incorrect. Morever, no dependency in python code present for tox.

JSONPath $..book[(@.length-1)] fails - Parse error at 1:8 near token ( (()

PREREQUISITE:
You have installed JSONLibrary
-- > pip install robotframework-jsonlibrary

How to reproduce

1. create file bookstore.robot with following content

*** Settings ***
Library    JSONLibrary
Suite Setup    BEFORE TEST SUITE

*** Test Cases *** 
Get the last book in order
    Get Value From Json    ${bookstore_data}    $..book[(@.length-1)]

*** Keywords ***
BEFORE TEST SUITE
    ${bookstore_data}=    Load JSON From File  ${CURDIR}/bookstore.json
    Set Suite Variable    ${bookstore_data}

2. create file bookstore.json with JSON data:

{ "store": {
    "book": [ 
      { "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      { "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      },
      { "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      },
      { "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

3. execute the test

robot -d results -L TRACE bookstore.robot

Result

Parse error at 1:8 near token ( (()	
14:19:27.818	DEBUG	Traceback (most recent call last):
  File "c:\python_virtual_envs\api_requests\lib\site-packages\JSONLibrary\JSONLibraryKeywords.py", line 77, in get_value_from_json
    json_path_expr = parse(json_path)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw_ext\parser.py", line 179, in parse
    return ExtentedJsonPathParser(debug=debug).parse(path)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 32, in parse
    return self.parse_token_stream(lexer.tokenize(string))
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 55, in parse_token_stream
    return new_parser.parse(lexer = IteratorToTokenStream(token_iterator))
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 69, in p_error
    raise Exception('Parse error at %s:%s near token %s (%s)' % (t.lineno, t.col, t.value, t.type))

Shouldn't the keyword "Get Value From Json" fail if element is not found?

If an element is not present in JSON I would expect Get Value From Json keyword to fail instead of returning an empty list.

How do you think about it?

Cheers
Tset 🤖

AND operator not working in jsonPath

Hi!
I tried to use a multiple condition filter in a jsonPath
like:
$[?(@.invoiceNum='${value1}' && @.issueNum=${value2})].issueDate
but the and operator doesn't work (i guess the or operator is the same).
I get a parse error:
Invalid JSONPath query '$[?(@.invoiceNum='1111111' && @.issueNum=1)].issueDate': Parse error at 1:28 near token & (&)

Request to reverse issue #5

Hi, in release 0.4 issue #5 was included. My testcases are now breaking because the keyword 'Get Value From Json' now fails when the provided element is not present in the JSON. I use this keyword to perform certain cleanup actions, and the cleanup is done in a followup loop, that loops through the resulting list from the 'Get Value From Json' keyword. If the list is empty no cleanup is needed and the loop is simply skipped automatically. In this new version however, the keyword fails and I have to code around it to skip the cleanup-step. Granted, this can be fixed on my end by some extra code like a 'Run Keyword And Ignore Error', but I think it's fundamentally wrong to let the keyword fail when the provided element is not in the JSON. It's not an error, the element is simply not there so the returned list should be empy, like it was.

So I would like to request to reverse the code-change, and maybe start a discussion about this issue. What is the general consensus about how this should work? Does anyone agree with me? Down here an example of my current code:

${ids} Get Value From Json ${zaken} $..id
FOR ${id} IN @{ids}
Delete Zaak With Id ${id}
END

Add Object To Json doesn't add anything unless key already in the json object

Name of this keyword Add Object To Json suggests it will add something.

However, when I have e.g. an empty json object:

{
    
}

and I run:

*** Settings ***
Library    JSONLibrary 

*** Test Cases *** 
Empty JSON
    ${data}    Load JSON From File    empty.json
    Add Object To Json    ${data}    $..name    pavelsaman   
    Log    ${data}    

it will log { }.

name has to exist in order for the keyword to do something.

It's obvious why:

json_path_expr = parse(json_path)
for match in json_path_expr.find(json_object):
    if type(match.value) is dict:
        match.value.update(object_to_add)
    if type(match.value) is list:
        match.value.append(object_to_add)

return json_object

when the json_object is empty, it won't even enter the for loop.

When my initial json object is:

{
    "name": []
}

it will result in:

{'name': [['pavelsaman']]}

which is not desired. Similarly with a dict in the initial json.

It's at least a naming issue when by reading the name and documentation, I'd expect something gets added no matter what the initial state (provided I pass a valid json object to the keyword).

Examples of JSONPath expressions

There is already a link to goessner.net in the README but one need to scroll a bit to find the examples. So here's the direct linkt to the examples section for the lazzy one ;-)

JSONPath Examples

http://goessner.net/articles/JsonPath/index.html#e3
(includes comparison with XPATH examples)

Issues with `Delete Object From Json` KW

I was not able to delete objects from JSON. I wanted to delete everything from rows in example JSON below so that I get an empy list "rows": []. I have tried following jsonpaths

• $.rows.* (nothing happened)
• $..rows.* (nothing happened)
• $.rows[*] IndexError: list assignment index out of range
• $..rows[*] IndexError: list assignment index out of range

{
  "meta": {
    "_type": "RESULTSET",
    "_created": "2019-10-17T19:35:13.372Z",
    "_executed_aql": "SELECT e/ehr_id/value FROM EHR e"
  },
  "q": "SELECT e/ehr_id/value FROM EHR e",
  "columns": [
    {
      "name": "#0",
      "path": "/ehr_id/value"
    }
  ],
  "rows": [
    [
      "__MODIFY_EHR_ID_1__"
    ],
    [
      "__MODIFY_EHR_ID_2__"
    ],
    [
      "__MODIFY_EHR_ID_3__"
    ],
    [
      "__MODIFY_EHR_ID_4__"
    ],
    [
      "__MODIFY_EHR_ID_5__"
    ],
    [
      "__MODIFY_EHR_ID_6__"
    ],
    [
      "__MODIFY_EHR_ID_7__"
    ],
    [
      "__MODIFY_EHR_ID_8__"
    ],
    [
      "__MODIFY_EHR_ID_9__"
    ],
    [
      "__MODIFY_EHR_ID_10__"
    ]
  ]
}

Traceback (most recent call last):
  File "/.../venv/lib/python3.7/site-packages/JSONLibrary/JSONLibraryKeywords.py", line 120, in delete_object_from_json
    del(match.context.value[match.path.index])

Am I doing something wrong?

Pretty print when converting json to string

When using the 'Convert Json To String' keyword, the string is not really readable for bigger jsons.
This could easily be solved by using the indent parameter in the json.dumps method:
return json.dumps(json_object, indent=4)

I don't know if this can cause problems for other use cases than printing a json or dumping it into a file, but if yes, the pretty printing could be done via an additional parameter for the 'Convert Json To String' keyword.

Add jmespath query language as another sepcification for querying json files

How about adding another json query specification language to be handled by this library?
I'm thinking of something like:

    *** Settings ***
    Library    JsonLibrary    # by default it will be jsonpath_ng.ext

or by using jmespath as a backend json query specification

    *** Settings ***
    Library    JsonLibrary    jmespath    # just as an argument passed to initializer so this implementation will be used as a backend for querying json

such implementation won't be so hard, although there is some catch: if jmespath is used there won't be a possibility to manipulate json files (adding, deleting, modifying) as jmespath is a query language (imho more powerfull than jsonquery) and it does not handle any manipulations (...in an easy way, the hard way is waaay to hard - at least for me :)).

How about that?

JSONLibrary keywords are not working

Hi there,

Since yesterday, our automated test cases started failing because of latest version 0.5 and when I looked at source code, it seems keywords are removed @Keyword("Convert String To JSON") from
@staticmethod
def convert_json_to_string(json_object):
"""Convert JSON object to string

    Arguments:
        - json_object: json as a dictionary object.

Convert String to JSON?

This might be a silly question. But is there a way to get a json object from string instead of a file by this library?

jsonpath-ng instead of jsonpath-ext + jsonpath-rw

Background:

https://github.com/h2non/jsonpath-ng is a fork of jsonpath-rw. Its contributors contains also the author of the main package jsonpath-rw. It is represented as a replacement for these two packages and full support for the jsonpath syntax.

Proposal:

Replace jsonpath-rw, jsonpath-ext with jsonpath-ng

Typo in Keyword documentation

There is a little typo in the keyword documentation:

Cannot create response object with empty array or object in body

When I want the mockserver to respond with a body with an empty array [] or object {}, the 'body' parameter is not in the response.

in library.py line 95 a test is done:
if body_type is 'JSON' and body: rsp['body'] = json.dumps(body)

However, when body is [] or {}, bool([]) and bool({}) evaluate to False. So the next line is skipped.
possible fix:
if body_type is 'JSON' and body is not None: rsp['body'] = json.dumps(body)

Keyword to write JSON object to file

Hi,

Would be useful to have an keyword Dump JSON To File in order to serialize the object to file. Mostly in cases where we need to update an value and then write it back to file for configuration purposes.

Relative path for keyword "Load JSON From File"

Why not use relative (instead of absolute) path for the Load JSON From File keyword ?

btw: greate library! ❤️

More return type for Get Value From JSON

I have the idea to add more return type to the keyword Get Value From JSON.
Because what it return is not a JSON, this optonial parameter would let you choose if the keyword must return a json object, a string ?, a array, ...

Support for Python 3.2 +

How to reproduce
Not able to install the library for python 3.2 +.

Error

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/fr/jtjjqpbs1sx0lxh6qbgzgdjw00gftq/T/pip-build-ghem6ueu/robotframework-jsonlibrary/

issue in add_objcet_to_json

Hi, I believe there is a small issue with the function add_object_to_json. In line 132 of jsonlibrary.py, it should be...

match.value.update({child_name: object_to_add_cpy[child_name]})

Currently, the function is adding the whole object as a value.
input: {key1;value1}
output: {key1: {key1;value1}}

can you please check and let me know if it is a bug or I am doing something wrong?

JSONPath $..book[0,1] fails - Parse error at 1:9 near token , (,)

How to reproduce

Use JSON data (bookstore.json) and RF test case (bookstore.robot) from issue #7
but replace test case with this one:

Get the first two books
    Get Value From Json    ${bookstore_data}    $..book[0,1]

Result

Parse error at 1:9 near token , (,)	
14:53:23.052	DEBUG	Traceback (most recent call last):
  File "c:\python_virtual_envs\api_requests\lib\site-packages\JSONLibrary\JSONLibraryKeywords.py", line 77, in get_value_from_json
    json_path_expr = parse(json_path)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw_ext\parser.py", line 179, in parse
    return ExtentedJsonPathParser(debug=debug).parse(path)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 32, in parse
    return self.parse_token_stream(lexer.tokenize(string))
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 55, in parse_token_stream
    return new_parser.parse(lexer = IteratorToTokenStream(token_iterator))
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\ply\yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "c:\python_virtual_envs\api_requests\lib\site-packages\jsonpath_rw\parser.py", line 69, in p_error
    raise Exception('Parse error at %s:%s near token %s (%s)' % (t.lineno, t.col, t.value, t.type))

Missing keywords

Hi there,

I'm a bit confused with the version and available keywords.
After installing latest version via pip (0.3.1) the keyword 'Should Have Value In Json' is not available to me.

Is this because this keyword is only available in a newer release?

Thank for clarification.