Ignore order for nested iterables

[seperman]

Ignore order for nested iterables

[seperman]

I made some adjustments and it works with list containing lists but not lists containing dictionaries.
We will have to fork Pickle and make it consistent in order to get the best results.

[seperman]

It is in 2.1.1

[anatoly-kussul]

Thanks for quick fix.
But it doesn't work if i have nested list in nested list:

>>> a = [1, 2, [3, [4, 5]]]
>>> b = [[[5, 4], 3], 2, 1]
>>> DeepDiff(a, b, ignore_order=True)
2016-08-10 12:24:08,807  WARNING Unable to order data type: <class 'list'>. Ignore order might be giving inaccurate results.
Traceback (most recent call last):
  File "/home/anatoly/Work/unfork-ng/.env/lib/python3.5/site-packages/deepdiff/diff.py", line 81, in order_unordered
    sorted_by_type = sorted(data, key=lambda x: type(x))
TypeError: unorderable types: list() < int()
2016-08-10 12:24:08,807  WARNING Unable to order data type: <class 'list'>. Ignore order might be giving inaccurate results.
Traceback (most recent call last):
  File "/home/anatoly/Work/unfork-ng/.env/lib/python3.5/site-packages/deepdiff/diff.py", line 81, in order_unordered
    sorted_by_type = sorted(data, key=lambda x: type(x))
TypeError: unorderable types: int() < list()
{'iterable_item_added': {'root[0]': [[5, 4], 3]}, 'iterable_item_removed': {'root[2]': [3, [4, 5]]}}

Have some another not working example with nested list in dictionary values:

>>> a = [{1: [2, 3], 4: [5, [6, 7]]}]
>>> b = [{4: [[7, 6], 5], 1: [3, 2]}]
>>> DeepDiff(a, b, ignore_order=True)
2016-08-10 12:21:44,909  WARNING Unable to order data type: <class 'list'>. Ignore order might be giving inaccurate results.
Traceback (most recent call last):
  File "/home/anatoly/Work/unfork-ng/.env/lib/python3.5/site-packages/deepdiff/diff.py", line 81, in order_unordered
    sorted_by_type = sorted(data, key=lambda x: type(x))
TypeError: unorderable types: list() < int()
2016-08-10 12:21:44,910  WARNING Unable to order data type: <class 'list'>. Ignore order might be giving inaccurate results.
Traceback (most recent call last):
  File "/home/anatoly/Work/unfork-ng/.env/lib/python3.5/site-packages/deepdiff/diff.py", line 81, in order_unordered
    sorted_by_type = sorted(data, key=lambda x: type(x))
TypeError: unorderable types: int() < list()
{'iterable_item_added': {'root[0]': {1: [3, 2], 4: [[7, 6], 5]}}, 'iterable_item_removed': {'root[0]': {1: [2, 3], 4: [5, [6, 7]]}}}

[seperman]

I'm gonna have to write another serializer instead of Pickle to make this work right. Will keep you posted.

[anatoly-kussul]

Ok, thanks.

[seperman]

I had some time to finally write a custom content hash for this issue. It passes the sample inputs you have mentioned here. https://github.com/seperman/deepdiff/tree/contenthash
Please check it out and let me know if it works fine.

[anatoly-kussul]

Found another weird case with repetitions:
If repetition is in top level list it's ok:

>>> a = [1, 2, [3, 4]]
>>> b = [[4, 3], 2, 1, 1]
>>> DeepDiff(a, b, ignore_order=True)
{}

>>> a = [1, 2, [3, 4]]
>>> b = [[4, 3], 2, 1, 1]
>>> DeepDiff(a, b, ignore_order=True, report_repetition=True)
{'repetition_change': {'root[0]': {'value': 1, 'new_indexes': [2, 3], 'new_repeat': 2, 'old_indexes': [0], 'old_repeat': 1}}}

But if repetition in nested list, it works another way:

>>> a = [1, 2, [3, 4]]
>>> b = [[4, 3, 3], 2, 1]
>>> DeepDiff(a, b, ignore_order=True)
{'iterable_item_removed': {'root[2]': [3, 4]}, 'iterable_item_added': {'root[0]': [4, 3, 3]}}

For me it's fine, but I think it's weird that logic is different for top-level and nested lists

[seperman]

@anatoly-kussul I merged the PR. It seems it does a great job when ignoring order now. But it does not do it right as far as reporting repetition. The reason is that once it identifies to items as being the same when order is ignored, it won't bother to check what is inside them for repetitions. I have some ideas how to do this properly later.

[seperman]

#39

[anatoly-kussul]

Thanks a lot. 👍