genkey output is not consistent

[DMWMBot]

genkey output is not consistent

[DMWMBot]

gfball: I've added a possible solution, plus a unittest that demonstrates the problem (at least on this machine).

The solution passes the unittest, but is probably significantly more expensive. It recurses through a nested structure, finding all the primitive types, converting them to unicode strings then finally sorts the list of all the resulting strings. The ordered list of strings is then fed into md5 to generate the hash.

As an added side effect, cache, merge and analytics will need to be emptied since the new value is not any of the previous possibilities.

[vkuznet]

valya: Hmm, you're right. I miss that. But now my fear is that your solution may not work in general case. It rely on a fact that dict will have"key" and "value" and I use genkey in different places. How about this way:

{{{
def genkey(query):
if isinstance(query, dict):
squery = str(query)
lquery = list(squery)
lquery.sort()
query = ''.join(lquery)

 keyhash.update(query)
 return keyhash.hexdigest()

}}}

I convert query, if it is a dict, into a list, sort the list and create a new string out of it. Seems to work, but I didnt' test with unicode/base strings mixture.

[DMWMBot]

gfball: Yes, on the way home I realised there were lots of things wrong with it, eg

['a', 'b'] and ['b', 'a']
['abc', 'def'] and ['abcdef']
{'a': 'b'} and {'b': 'a'}
{1: {2: 3}} and 1abac61 2eb6285 2]]

would all compare the same, which is clearly broken.

I think the right way to do this is to do it as genkey() already does (using json.dumps, which deals with unicode issues for us and works fine with everything but dict) but use a subclass of JSONDecoder that represents dict as [(k1, v1), (k2, v2) ...], sorted by key. I think this should have 1) relatively low overhead and 2) doesn't require us to write any recursive code.

[vkuznet]

valya: Sounds good to me. Give it a shot and send a patch.

[DMWMBot]

gfball: Turns out to be really quite simple. If you use JSONEncoder directly, it accepts a kwarg "sort_keys" which resolves this problem without having to even change the dictionary representation. In many cases the qhash will be the same as before.

Had to also make JSONEncoder available in jsonwrapper, since we can only get it from json, not cjson.

[vkuznet]

valya: Patch has been applied, closing the ticket.