jsonpickle/jsonpickle

Python library for serializing any arbitrary object graph into JSON. It can take almost any Python object and turn the object into JSON. Additionally, it can reconstitute the object back into Python.

jsonpickle

jsonpickle is a library for the two-way conversion of complex Python objects and JSON. jsonpickle builds upon existing JSON encoders, such as simplejson, json, and ujson.

Warning

jsonpickle can execute arbitrary Python code.

Please see the Security section for more details.

For complete documentation, please visit the jsonpickle documentation.

Bug reports and merge requests are encouraged at the jsonpickle repository on github.

Why jsonpickle?

Data serialized with python's pickle (or cPickle or dill) is not easily readable outside of python. Using the json format, jsonpickle allows simple data types to be stored in a human-readable format, and more complex data types such as numpy arrays and pandas dataframes, to be machine-readable on any platform that supports json. E.g., unlike pickled data, jsonpickled data stored in an Amazon S3 bucket is indexible by Amazon's Athena.

Security

jsonpickle should be treated the same as the Python stdlib pickle module from a security perspective.

Warning

The jsonpickle module is not secure. Only unpickle data you trust.

It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.

Consider signing data with an HMAC if you need to ensure that it has not been tampered with.

Safer deserialization approaches, such as reading JSON directly, may be more appropriate if you are processing untrusted data.

Install

Install from pip for the latest stable release:

pip install jsonpickle

Install from github for the latest changes:

pip install git+https://github.com/jsonpickle/jsonpickle.git

Numpy/Pandas Support

jsonpickle includes built-in numpy and pandas extensions. If you would like to encode sklearn models, numpy arrays, pandas DataFrames, and other numpy/pandas-based data, then you must enable the numpy and/or pandas extensions by registering their handlers:

>>> import jsonpickle.ext.numpy as jsonpickle_numpy
>>> import jsonpickle.ext.pandas as jsonpickle_pandas
>>> jsonpickle_numpy.register_handlers()
>>> jsonpickle_pandas.register_handlers()

Development

Use make to run the unit tests:

make test

pytest is used to run unit tests internally.

A tox target is provided to run tests using all installed and supported Python versions:

make tox

jsonpickle itself has no dependencies beyond the Python stdlib. tox is required for testing when using the tox test runner only.

The testing requirements are specified in setup.cfg. It is recommended to create a virtualenv and run tests from within the virtualenv.:

python3 -mvenv env3
source env3/bin/activate
pip install --editable '.[dev]'
make test

You can also use a tool such as vx to activate the virtualenv without polluting your shell environment:

python3 -mvenv env3
vx env3 pip install --editable '.[dev]'
vx env3 make test

If you can't use a venv, you can install the testing packages as follows:

pip install .[testing]

jsonpickle supports multiple Python versions, so using a combination of multiple virtualenvs and tox is useful in order to catch compatibility issues when developing.

GPG Signing

Unfortunately, while versions of jsonpickle before 3.0.1 should still be signed, GPG signing support was removed from PyPi (https://blog.pypi.org/posts/2023-05-23-removing-pgp/) back in May 2023.

License

Licensed under the BSD License. See COPYING for details.

jsonpickle's Issues

Tangential comment: It would be awesome if someone built a custom Django field using jsonpickle...

...and compared it to django-picklefield (https://github.com/gintas/django-picklefield), which is rather slow. If i get time, i'll try to do it.

Simple round-trip fails to reconstruct object of locally-defined class

The following unit test, inside a function, triggers the first assertion (or will trigger the second if the 1st is removed)

def test_json_stuff(self):
    import jsonpickle
    class SomeObject(object):
         pass
    object_original = SomeObject()
    object_clone = jsonpickle.decode(jsonpickle.encode(object_original))
    self.assertFalse(isinstance(object_clone, type(dict())))
    self.assert_(isinstance(object_clone, SomeObject))

Traceback (most recent call last):
File "test.py", line 248, in test_json_stuff
self.assertFalse(isinstance(object_clone, type(dict())))
AssertionError

Basically it's returning the plain dict instead of reconstructing the object as expected. However, if the definition of the 'SomeObject' class is moved to file scope, it works.

The worst bit about this is not the limitation, but the silent failure and the returning of the dict instead of raising an exception.

AttributeError flattening an object with string '__slots__'

Python allows __slots__ to be a string, list, or other iterable.

If __slots__ is a string, jsonpickle treats it as an iterable and fails iterating over the characters of the string. Consider this simple test using pymongo:

import jsonpickle
import bson
oid = bson.ObjectId()
jsonpickle.encode(oid)

Produces this traceback:

Traceback (most recent call last):
  File "C:\Users\jaraco\projects\public\jsonpickle\test-oid.py", line 4, in <module>
    jsonpickle.encode(oid)
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\__init__.py", line 135, in encode
    max_depth=max_depth)
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\pickler.py", line 29, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\pickler.py", line 121, in flatten
    return self._flatten(obj)
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\pickler.py", line 125, in _flatten
    return self._pop(self._flatten_obj(obj))
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\pickler.py", line 137, in _flatten_obj
    return flatten_func(obj)
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\pickler.py", line 185, in _ref_obj_instance
    return self._flatten_obj_instance(obj)
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\pickler.py", line 246, in _flatten_obj_instance
    return self._flatten_newstyle_with_slots(obj, data)
  File "C:\Users\jaraco\projects\public\jsonpickle\jsonpickle\pickler.py", line 268, in _flatten_newstyle_with_slots
    self._flatten_key_value_pair(k, getattr(obj, k), data)
AttributeError: 'ObjectId' object has no attribute '_'

Because oid.__slots__ == '__id', when jsonpickle tries to iterate over slots, it first looks for the attribute '_' and fails.

I'll prepare a test and fix for this.

I note also, though there is a harder problem of supporting iterables in general. If the iterable for __slots__ is already consumed, __slots__ is not helpful in ascertaining the slots used by the class.. but that's a separate issue.

Reused time object does not decodes

import jsonpickle, datetime
t = datetime.time(hour=10)
s = jsonpickle.encode({1:t, 2:t})
print s
print jsonpickle.decode(s)

  {"1": {"py/repr": "datetime/datetime.time(10, 0)"}, "2": {"py/id": 0}}

  File "/Library/Python/2.7/site-packages/jsonpickle/__init__.py", line 325, in decode
    return j.restore(json.decode(string))
  File "/Library/Python/2.7/site-packages/jsonpickle/unpickler.py", line 153, in restore
    data[k] = self.restore(v)
  File "/Library/Python/2.7/site-packages/jsonpickle/unpickler.py", line 66, in restore
    return self._pop(self._objs[obj[tags.ID]])
IndexError: list index out of range

decoding serialized datetime with timezone fails

Trying to decode a serialized datetime object with timezone raises an exception. See this example:

>>> import datetime
>>> import jsonpickle
>>> import pytz
>>> dt = jsonpickle.encode(datetime.datetime.now(pytz.timezone('Europe/Berlin')))
>>> dt
'{"py/repr": "datetime/datetime.datetime(2011, 7, 11, 18, 59, 20, 508822, tzinfo=<DstTzInfo \'Europe/Berlin\' CEST+2:00:00 DST>)"}'
>>> now = jsonpickle.decode(dt)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/martin/Work/bo_verdi_internet/eggs/jsonpickle-0.4.0-py2.7.egg/jsonpickle/__init__.py", line 325, in decode
    return j.restore(json.decode(string))
  File "/home/martin/Work/bo_verdi_internet/eggs/jsonpickle-0.4.0-py2.7.egg/jsonpickle/unpickler.py", line 78, in restore
    return self._pop(loadrepr(obj[tags.REPR]))
  File "/home/martin/Work/bo_verdi_internet/eggs/jsonpickle-0.4.0-py2.7.egg/jsonpickle/unpickler.py", line 233, in loadrepr
    module, evalstr = reprstr.split('/')
ValueError: too many values to unpack

jsonpickle requires that classes must implement `__getstate__` and `__setstate__` but pickle doesn't require both

Normal pickle can dump objects that only implement __getstate__ but jsonpickle requires them to implement __setstate__ too even if you set unpickleable=False

In [1]: class Example(object):
   ...:         def __getstate__(self):
   ...:                 return dict(hello="world", foo=100, bar=200)
   ...:

In [2]: import jsonpickle

In [3]: import json

In [4]:

In [4]: obj = Example()

In [5]: assert jsonpickle.encode(obj, unpicklable=False) == json.dumps(obj.__getstate__())
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-5-8be712d0cbcc> in <module>()
----> 1 assert jsonpickle.encode(obj, unpicklable=False) == json.dumps(obj.__getstate__())

AssertionError:

In [6]: assert jsonpickle.encode(obj, unpicklable=False) != '{}'
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-6-70e2f17806dc> in <module>()
----> 1 assert jsonpickle.encode(obj, unpicklable=False) != '{}'

AssertionError:

In [7]: import pickle

In [8]: print pickle.dumps(obj)
ccopy_reg
_reconstructor
p0
(c__main__
Example
p1
c__builtin__
object
p2
Ntp3
Rp4
(dp5
S'foo'
p6
I100
sS'bar'
p7
I200
sS'hello'
p8
S'world'
p9
sb.

error: package directory 'jsonpickle' does not exist

when installed jsonpickle with python 2.7, an error jump out as follows:
error: package directory 'jsonpickle' does not exist

C:\Python27>python F:\2014_Year\01.DevWork\3rdLib\jsonpickle-0.7.0\setup.py install
running install
running bdist_egg
running egg_info
writing jsonpickle.egg-info\PKG-INFO
writing top-level names to jsonpickle.egg-info\top_level.txt
writing dependency_links to jsonpickle.egg-info\dependency_links.txt
warning: manifest_maker: standard file 'setup.py' not found

error: package directory 'jsonpickle' does not exist

Cannot override serialization function when is_repr() is true

This prevents the overriding of the datetime serialization from being customized. When the _flatten_obj_instance is called, the registered handlers are not used if the Class is listed in the NEEDS_REPR array in the utils. If a person wants to override these, then that should be up to them.

Support for os.stat_result

It's seems that object is not supported by default.

Strangely enough, running handlers.register(posix.stat_result, handlers.SimpleReduceHandler) seems to make it work. Is this a bug?

Internal C++ object already deleted.

Apologies if this is not the right place to ask this.
I am trying to use jsonpickle for a list of points defined in PyQt/PySide.
Can you please let me know if this is possible using the module or I'm doing something wrong here?

    from PySide import QtCore
    import jsonpickle

    coords = QtCore.QPointF(3.7, 4)
    a = {"node": coords}
    p = jsonpickle.encode(a)
    u = jsonpickle.decode(p)

    print u

results in:

Internal C++ object (PySide.QtCore.QPointF) already deleted.

py/<type> metadata is still present when unpicklable=True

I'm not sure what the right thing to do is, but intuitively I don't expect py/ metadata to still be present when encoded with unpicklable=True. eg.

>>> encode(('foo', 'bar'), unpicklable=True)
'{"py/tuple": ["foo", "bar"]}'

I would expect:

>>> encode(('foo', 'bar'), unpicklable=True)
'["foo", "bar"]'

As there is no reason to preserve type information for unpickling.

jsonpickle fails to preserve Unicode strings

Any chance of decode("foo") returning a Unicode string? E.g. controlled by a parameter, not to break compatibility.

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from jsonpickle import encode, decode
>>> type(decode(encode(u'foo'))) == type(u'foo')
False

Fails to encode attribute of nested `collections.defaultdict`

I use nested defaultdicts to represent sparse matrices. With default settings (i.e., no directives to the backend) on Python 2.7.5 and Python 3.4.1, the outer default_factory is simply not encoded. The following example shows this:

import jsonpickle
import collections
mydict = collections.defaultdict(lambda: collections.defaultdict(int))
mydict[3][2] = 4
encoded = jsonpickle.encode(mydict)
print(encoded)

This prints:

{"py/object": "collections.defaultdict", "3": {"default_factory": {"py/type": "__builtin__.int"}, "2": 4, "py/object": "collections.defaultdict"}}

This is wrong: there's no outer default_factory at all. (Note this has nothing to do with the use of a lambda function, as far as I can tell: non-anonymous default_factories behave the same way.)

This also doesn't go round-trip:

mydict_new = jsonpickle.decode(encoded)
print(mydict.default_factory) # prints pointer to the lambda function
print(mydict_new.default_factory) # prints None

Decoding cyclical object fails to preserve structure of tree.

I'm attempting to use jsonpickle to encode/decode a tree like data structure. Unfortunately, jsonpickle.decode() isn't working properly. I'm using version 0.3.1 installed via pip.

I've created a minimal code example to show the problem here: https://gist.github.com/712554

The document is structured as such

• Document
  • Section 1
    • Question 1
    • Question 2
  • Section 2
    • Question 3
    • Question 4

and it is serialized correctly to disk. Upon reading it back in, however, Questions 3 and 4 report their parent as being Section 1, rather than Section 2.

With the str() and repr() methods in the Document, Section, and Question classes, printing results in the following before encoding, and after decoding the encoded jsonpickle string.

Original Document Structure: 
Document "My Document"
Section "Section 1", parent: "My Document"
Question "Question 1", parent: "Section 1"
Question "Question 2", parent: "Section 1"
Section "Section 2", parent: "My Document"
Question "Question 3", parent: "Section 2"
Question "Question 4", parent: "Section 2"

Document Structure after jsonpickle.decode()
Document "My Document"
Section "Section 1", parent: "My Document"
Question "Question 1", parent: "Section 1"
Question "Question 2", parent: "Section 1"
Section "Section 2", parent: "My Document"
Question "Question 3", parent: "Section 1"
Question "Question 4", parent: "Section 1"

Question 3 and Question 4 should have "Section 2" as a parent, rather than "Section 1"

SandboxViolation installing jsonpickle

We recently ran into an error installing jsonpickle on our system where simplejson was installed as a zipped egg:

Processing dependencies for gryphon==3.9b3
Searching for jsonpickle
Reading http://cheeseshop.polimetrix.com
Best match: jsonpickle 0.4.0
Downloading http://cheeseshop.polimetrix.com/jsonpickle-0.4.0.tar.gz
Processing jsonpickle-0.4.0.tar.gz
Running jsonpickle-0.4.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-AzSx_I/jsonpickle-0.4.0/egg-dist-tmp-Ft4uq4
error: SandboxViolation: chmod('/var/lib/jenkins/.python-eggs/simplejson-2.1.3-py2.7-linux-x86_64.egg-tmp/simplejson/tmpozJqL9.$extract', 493) {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

It appears the problem lies when the jsonpickle setup script tries to determine its version. To accomplish this task, it first imports jsonpickle, which causes the backends to be loaded, which causes simplejson to be imported, which causes setuptools to expand the zipped egg, which violates the setuptools Sandbox.

It would be much better if jsonpickle were able to determine its version without initializing jsonpickle.

I could imagine several ways this could be done.

1. Store the version in a version.py file, import that version.py in jsonpickle.init and execfile it from setup.py.
2. Store the version in setup.py and depend on setuptools to determine the version in jsonpickle.init (See the eggmonster package for an example of how this might be done).
3. Store the version redundantly in setup.py and init.py.
4. Set a flag (environment variable or similar) that tells jsonpickle.init not to initialize the engine when running setup.py.

I can think of other alternatives as well, but none as attractive as the above. My preference would be for 1, but I'll leave it to the maintainers to find a fix that suits their style best.

If you would like me to implement the fix, just say so (and if you have any preference for the approach), and I'll fix it in a fork.

Regards,
Jason

jsonpickle_test.JSONPickleTestCase.test_ordered_dict() error with Python 3.4

jsonpickle_test.JSONPickleTestCase.test_ordered_dict() triggers error with Python 3.4. This problem does not occur with older versions of Python.

======================================================================
ERROR: test_ordered_dict (jsonpickle_test.JSONPickleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle/tests/jsonpickle_test.py", line 909, in test_ordered_dict
    decoded = jsonpickle.decode(encoded)
  File "/tmp/jsonpickle/jsonpickle/__init__.py", line 153, in decode
    return unpickler.decode(string, backend=backend, keys=keys)
  File "/tmp/jsonpickle/jsonpickle/unpickler.py", line 25, in decode
    return context.restore(backend.decode(string), reset=reset)
  File "/tmp/jsonpickle/jsonpickle/unpickler.py", line 76, in restore
    return self._restore(obj)
  File "/tmp/jsonpickle/jsonpickle/unpickler.py", line 99, in _restore
    return restore(obj)
  File "/tmp/jsonpickle/jsonpickle/unpickler.py", line 126, in _restore_object
    instance = handler(self).restore(obj)
  File "/tmp/jsonpickle/jsonpickle/handlers.py", line 145, in restore
    factory, args = [restore(i, reset=False) for i in obj['__reduce__']]
ValueError: too many values to unpack (expected 2)

======================================================================

specifying explicit backend

I need to be able to explicitly specify the serialize backend via an optional 'backend' arg in the encode/decode functions. I have several separate modules using jsonpickle, but some require different backends. So re:

import jsonpickle as js
s = js.encode(mydata, 'json')
js.decode(mydata, 'json')

I've implemented this functionality locally and I'd like to contribute the change, but I see that this project hasn't seen any activity in quite a while. Will somebody pck up my pull request if I send it?

Also, I'm using Yaml as a backend as well. In my case, I need to do this since the other backends all decode strings to UTF-8, and in my case this causes problems in some host apps that do not support it. Yaml doesn't do this. Would you consider adding Yaml as a new default backend?

thx
A

unpickled = jsonpickle.decode(pickled)) raises an unexpected IndexError: list index out of range

I get an IndexError: list index out of range when I write this kind of script
when decoding a JSON pickled object using
unpickled = jsonpickle.decode(pickled)
I don't know if problem is on my side or on your side...

import datetime
import time
from collections import OrderedDict
import jsonpickle

class TimestampedVariable(object):
    def __init__(self, value=None):
        self._value = value
        self._dt_read = datetime.datetime.utcnow()
        self._dt_write = self._dt_read

    def get(self, default_value=None):
        if self._dt_read == None and self._dt_write == None:
            value = default_value
            self._value = value
            self._dt_write = datetime.datetime.utcnow()
        else:
            value = self._value
        self._dt_read = datetime.datetime.utcnow()        
        return(value)

    def set(self, new_value):
        self._dt_write = datetime.datetime.utcnow()
        self._value = new_value

    def __repr__(self):
        dt_now = datetime.datetime.utcnow()
        td_read = dt_now - self._dt_read
        td_write = dt_now - self._dt_write
        s = '<TimestampedVariable>\n'
        s += '  value: ' + str(self._value) + '\n'
        s += '  dt_read : ' + str(self._dt_read) + ' (%s ago)' % td_read + '\n'
        s += '  dt_write: ' + str(self._dt_write) + ' (%s ago)' % td_write + '\n'
        return(s)


    def erasable(self, td=datetime.timedelta(seconds=1)):
        dt_now = datetime.datetime.utcnow()
        td_read = dt_now - self._dt_read
        td_write = dt_now - self._dt_write
        return( ( td_read > td ) and ( td_write > td ) )


class PersistantVariables(object):
    def __init__(self):
        self._data = dict() #OrderedDict()

    def __getitem__(self, key):
        if key not in self._data:
            self._data[key] = TimestampedVariable(None)

        return self._data[key]

    def __setitem__(self, key, value):
        if key not in self._data:
            self._data[key] = TimestampedVariable(value)

        return self._data[key]

    def __repr__(self):
        return(str(self._data))


pvars = PersistantVariables()
print(pvars['var1'])

#print(pvars['var2']['var2b'])

time.sleep(2)

print(pvars._data)

time.sleep(2)

print(pvars['var2.var2b'])

print(pvars._data)

#pvars['z'] = 2
#pvars['z2']['z3'] = 3

#print(pvars.sorts)


pickled = jsonpickle.encode(pvars)
print(pickled)
unpickled = jsonpickle.decode(pickled)

This code returns this error

IndexError                                Traceback (most recent call last)
<ipython-input-1-b30e13247ae1> in <module>()
     85 pickled = jsonpickle.encode(pvars)
     86 print(pickled)
---> 87 unpickled = jsonpickle.decode(pickled)

/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/jsonpickle/__init__.py in decode(string)
    323     """
    324     j = Unpickler()
--> 325     return j.restore(json.decode(string))

/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/jsonpickle/unpickler.py in restore(self, obj)
    118                 self._namestack.append(k)
    119                 # step into the namespace
--> 120                 value = self.restore(v)
    121                 if (util.is_noncomplex(instance) or
    122                         util.is_dictionary_subclass(instance)):

/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/jsonpickle/unpickler.py in restore(self, obj)
    151             for k, v in sorted(obj.iteritems(), key=operator.itemgetter(0)):
    152                 self._namestack.append(k)
--> 153                 data[k] = self.restore(v)
    154                 self._namestack.pop()
    155             return self._pop(data)

/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/jsonpickle/unpickler.py in restore(self, obj)
    118                 self._namestack.append(k)
    119                 # step into the namespace
--> 120                 value = self.restore(v)
    121                 if (util.is_noncomplex(instance) or
    122                         util.is_dictionary_subclass(instance)):

/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/jsonpickle/unpickler.py in restore(self, obj)
     64 
     65         if has_tag(obj, tags.ID):
---> 66             return self._pop(self._objs[obj[tags.ID]])
     67 
     68         if has_tag(obj, tags.REF):

IndexError: list index out of range

Handle changes to jsonpickle's internal JSON format

jsonpickle's JSON format has changed in the past.

Further enhancements (such as supporting stronger id()-like references) will likely require changing the internal format.

Forwardsd compatibility, that is, having JSON written out by newer versions of jsonpickle readable in older versions of jsonpickle is probably untenable a.t.m.

Backwards compatibility, though, should be a high priority. If we change the JSON format then we should ensure that we can still read JSON files produced by older versions of jsonpickle.

There are several approaches to this issue.

I'll scribble down some thoughts here. In all likelyhood there's probably a design pattern or architectural strategy that we can employ to solve this problem but I don't know it at this moment.

We could namespace the pickling aspects of jsonpickle. The existing implementation could be called "pickler_v1" or something along those lines. This also suggests that we should sprinkle {"py/version": "1"} tags at the top-level of the JSON stream.

Dictionaries with either no py/version tag or with a py/version tag == "1" would correspond to the pickler_v1 implementation. This information is then understood by the unpickler which chooses the right unpickler implementation at runtime.

We don't need to support encoding multiple versions, only decoding them. The thought is that once a pickle is read and re-saved then we'll get the current format.

When a new way of encoding is introduced the py/version field is incremented and a new unpickler is registered to handle the new version.

This has a few implications:

• Future versions of jsonpickle must have a superset of the features provided by older jsonpickles
• We'll have to carry around different unpickler implementations. This is probably the biggest downside.

This isn't necessarily a problem at the moment but is something that we should address before releasing versions of jsonpickle that radically change the underlying JSON structure.

There have already been discussions on the mailing list about improving jsonpickle's referencing support, so this issue is not too far off from manifesting itself.

Are there any other approaches to this problem aside from the py/version-like thing?

We could use heuristics. That'll keep us from having to embed a version number in the flattened dict format. I'm not a huge fan of heuristics, though.

We could have a single unpickler that knows about the various versions. I can imagine it getting really nested and really complicated. jsonpickle already employs a fair amount of Python voodoo so making a single unpickler that deals with it seems like unneeded complexity.

Lastly, we could punt on the issue and simply say that jsonpickle does not make any guarantees about backwards compatibility. This is the easiest option to (not) implement, but it is probably the worst option for users.

IndexError: list index out of range when used with jaraco.modb and unpickling an object with references to multiple Queue objects

I receive the following error when trying to Unpickle an object that contains references to multiple Queue objects:

Traceback (most recent call last):
File "./jtest.py", line 24, in
tobj2= jaraco.modb.decode(result)
File "/usr/local/lib/python2.7/dist-packages/jaraco.modb-3.1-py2.7.egg/jaraco/modb/init.py", line 66, in decode
return Unpickler().restore(value)
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 76, in restore
File "/usr/local/lib/python2.7/dist-packages/jaraco.modb-3.1-py2.7.egg/jaraco/modb/init.py", line 39, in _restore
restored = super(Unpickler, self)._restore(obj)
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 99, in _restore
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 129, in _restore_object
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 155, in _restore_object_instance
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 164, in _restore_object_instance_variables
File "/usr/local/lib/python2.7/dist-packages/jaraco.modb-3.1-py2.7.egg/jaraco/modb/init.py", line 39, in _restore
restored = super(Unpickler, self)._restore(obj)
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 99, in _restore
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 129, in _restore_object
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 155, in _restore_object_instance
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 164, in _restore_object_instance_variables
File "/usr/local/lib/python2.7/dist-packages/jaraco.modb-3.1-py2.7.egg/jaraco/modb/init.py", line 39, in _restore
restored = super(Unpickler, self)._restore(obj)
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 99, in _restore
File "build/bdist.linux-x86_64/egg/jsonpickle/unpickler.py", line 102, in _restore_id
IndexError: list index out of range

Test code to reproduce:
import Queue

class MyTest(object):

   def __init__(self):
       self.queue_obj1 = Queue.Queue()
       self.queue_obj2 = Queue.Queue()

if name == "main":
import Queue
from pymongo import MongoClient
import jaraco.modb
database = "testing"
client = MongoClient()
db = client[database]
col = db["test"]
tobj = MyTest()
json_obj = jaraco.modb.encode(tobj)
obj_id = col.save(json_obj)
result = col.find_one(obj_id)
tobj2= jaraco.modb.decode(result)
print type(tobj)
print type(tobj2)

The error does not occur when only one Queue object is referenced in the MyTest() class.

Cannot decode subclass of list with custom __init__

I've added a failing test case here:

• jaraco/jsonpickle@6068c87

Basically, if a subclass of a list has custom init parameters, it is not decoded. Here's is the output of the failing test:

PS C:\Users\jaraco\projects\public\jsonpickle> py.test tests/jsonpickle_test.py
-k with_init
============================= test session starts =============================
platform win32 -- Python 2.7.1 -- pytest-2.0.2
collected 56 items

tests/jsonpickle_test.py F

================================== FAILURES ===================================
________________ PicklingTestCase.test_list_subclass_with_init ________________

self = <jsonpickle_test.PicklingTestCase testMethod=test_list_subclass_with_init>

    def test_list_subclass_with_init(self):
        obj = ListSubclassWithInit('foo')
        self.assertEqual(obj.attr, 'foo')
        flattened = self.pickler.flatten(obj)
        inflated = self.unpickler.restore(flattened)
>       self.assertEqual(type(inflated), ListSubclassWithInit)
E       AssertionError: <type 'dict'> != <class 'samples.ListSubclassWithInit'>

tests\jsonpickle_test.py:445: AssertionError
===================== 55 tests deselected by 'with_init' ======================
=================== 1 failed, 55 deselected in 0.34 seconds ===================

I originally encountered this error when attempting to serialize a boto.resultset.ResultSet.

Encoding does not work when __getstate__ returns a list.

When the pickled objects implement the getstate and setstate methods, and __getstate__ returns a list (I am assuming anything that is not a primitive type) the encoding does not work well.

The problem lies in the fact that when a list is created it takes a slot in the memory. Then, before another list is created, the previous one is garbage collected, and the new list might get the same exact id as the previous one (the results I am presenting happen around 99%, sometimes it works fine). When make_refs is True the library assumes that the list is the same object (I think its the expected behaviour), but when make_refs is False it still assumes something weird (pickler.py:125) and falls back to repring the value, which in turn yields something funny while decoding.

import jsonpickle

class A(object):
  def __init__(self, x, y):
    self.x = x
    self.y = y
  def __setstate__(self, l):
    self.x = l[0]
    self.y = l[1]
  def __getstate__(self):
    return [self.x, self.y]
  def __repr__(self):
    return 'A: (%s, %s)' % (repr(self.x), repr(self.y))

a1 = A(5, 7)
a2 = A(42, 10)
l = [a1, a2]

print l
refs = jsonpickle.decode(jsonpickle.encode(l))
print refs
no_refs = jsonpickle.decode(jsonpickle.encode(l, make_refs=False))
print no_refs

Results:

[A: (5, 7), A: (42, 10)]
[A: (5, 7), A: (5, 7)]
[A: (5, 7), A: ('[', '4')]

falure when deserializing a tree like data structure

This is a duplicate of Issue 11 that I accidentally closed. I also created a unit test that demonstrates the issue in my forked copy of the repo https://github.com/glimberg/jsonpickle/commit/1b58f717c8e05d7e20ab794d04dbb14f3335c13c

I'm attempting to use jsonpickle to encode/decode a tree like data structure. Unfortunately, jsonpickle.decode() isn't working properly. I'm using version 0.3.1 installed via pip.

I've created a minimal code example to show the problem here: https://gist.github.com/712554

The document is structured as such

• Document
  • Section 1
    • Question 1
    • Question 2
  • Section 2
    • Question 3
    • Question 4

and it is serialized correctly to disk. Upon reading it back in, however, Questions 3 and 4 report their parent as being Section 1, rather than Section 2.

With the str() and repr() methods in the Document, Section, and Question classes, printing results in the following before encoding, and after decoding the encoded jsonpickle string.

Original Document Structure: 
Document "My Document"
Section "Section 1", parent: "My Document"
Question "Question 1", parent: "Section 1"
Question "Question 2", parent: "Section 1"
Section "Section 2", parent: "My Document"
Question "Question 3", parent: "Section 2"
Question "Question 4", parent: "Section 2"

Document Structure after jsonpickle.decode()
Document "My Document"
Section "Section 1", parent: "My Document"
Question "Question 1", parent: "Section 1"
Question "Question 2", parent: "Section 1"
Section "Section 2", parent: "My Document"
Question "Question 3", parent: "Section 1"
Question "Question 4", parent: "Section 1"

Question 3 and Question 4 should have "Section 2" as a parent, rather than "Section 1"

IndexError: list index out of range when custom handler used

I believe I've discovered an issue with custom handlers. If a custom handler is used, during restore, the unpickler._mkref never gets called on the instance. Thus, references to the instance will fail. Given that they're assigned positionally, this probably also could lead to incorrect references.

I haven't yet confirmed the issue, but wanted to report it here to keep tabs on the issue, and also to highlight the potential issue as the datetime support now uses custom handlers.

IndexError: list index out of range with numeric dictionary keys

The reliance on sorted dictionary keys for object reference management will cause an IndexError if the lexical sorting of those keys differs from the numerical sorting and one of the objects contains references to other objects elsewhere in the order.

I've put together a test which fails thus.

docs: mention that you must inherit from "object"

jsonpickle works best with new-style classes. Document this.

AttributeError decoding OrderedDict

The traceback speaks for itself.

Python 2.7.4 (default, Apr  6 2013, 19:55:15) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import jsonpickle
>>> import collections
>>> jsonpickle.decode(jsonpickle.encode(collections.OrderedDict()))
OrderedDict()
>>> jsonpickle.decode(jsonpickle.encode(collections.OrderedDict(a=1)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\python\lib\site-packages\jsonpickle-0.4.1dev-py2.7.egg\jsonpickle\__init__.py", line 325, in decode
    return j.restore(json.decode(string))
  File "c:\python\lib\site-packages\jsonpickle-0.4.1dev-py2.7.egg\jsonpickle\unpickler.py", line 123, in restore
    instance[k] = value
  File "c:\python\lib\collections.py", line 59, in __setitem__
    root = self.__root
AttributeError: 'OrderedDict' object has no attribute '_OrderedDict__root'

jsonpickle fails with latest feedparser

The following test fails using the latest feedparser (r315)
#!/usr/bin/env python
# -- coding: utf-8 --

import jsonpickle
import feedparser
import pprint

data = """<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" version="2.0">
<channel>
    <item>
        <content:encoded><![CDATA[<a href="http://example.com/a.mp3">a</a> é]]></content:encoded>
    </item>
</channel>
</rss>"""

parsed = feedparser.parse(data)
x = parsed
pprint.pprint(x)
jsonpickle.encode(x, unpicklable=False)

An exception is thrown:

  File "/usr/lib/pymodules/python2.6/jsonpickle/util.py", line 40, in is_type
    return type(obj) is types.TypeType or repr(obj).startswith('<class')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 41-42: ordinal not in range(128)

namedtuple fails to decode properly

A namedtuple will decode, but it will be a dict, not the namedtuple you want

https://gist.github.com/abuchanan/5448778

Possibly related to #7

JsonPickle incorrectly encodes Infinity and NaN

According to the JSON spec:

Numeric values that cannot be represented as sequences of digits (such as Infinity and NaN) are not permitted.

Currently, this module passes through these values which results in most other JSON parsers failing to decode the output. Could you add an argument to jsonpickle.encode (similar to the 'allow_nan' option in simplejson) that properly throws an error when it sees these values? Even better would be the option to allow those values silently be converted to null.

TypeError ? How to decode / encode bottlepy FormsDict object ?

I ve tryed to serialize a bottle.FormsDict using last version on pypi (0.6.1).

    >>> from libs import bottle
    >>> d = bottle.FormsDict()
    >>> d
    <libs.bottle.FormsDict object at 0x7f721a476790>
    >>> d['a'] = 1 
    >>> d['test'] = ('test', 2)
    >>> import jsonpickle
    >>> jsonpickle.encode(d)
    '{"py/object": "libs.bottle.FormsDict", "dict": {"a": [1], "test": [{"py/tuple": ["test", 2]}]}, "py/seq": ["a", "test"]}'

    >>> jsonpickle.decode(jsonpickle.encode(d))
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/local/lib/python2.6/dist-packages/jsonpickle/__init__.py", line 152, in decode
    return unpickler.decode(string, backend=backend, keys=keys)
    File "/usr/local/lib/python2.6/dist-packages/jsonpickle/unpickler.py", line 25, in decode
    return context.restore(backend.decode(string), reset=reset)
    File "/usr/local/lib/python2.6/dist-packages/jsonpickle/unpickler.py", line 71, in restore
    return self._restore(obj)
    File "/usr/local/lib/python2.6/dist-packages/jsonpickle/unpickler.py", line 94, in _restore
    return restore(obj)
    File "/usr/local/lib/python2.6/dist-packages/jsonpickle/unpickler.py", line 121, in _restore_object
    return self._restore_object_instance(obj, cls)
    File "/usr/local/lib/python2.6/dist-packages/jsonpickle/unpickler.py", line 147, in _restore_object_instance
    return self._restore_object_instance_variables(obj, instance)
    File "/usr/local/lib/python2.6/dist-packages/jsonpickle/unpickler.py", line 174, in _restore_object_instance_variables
    instance.append(self._restore(v))
    TypeError: append() takes exactly 3 arguments (2 given)

Any advice ? thanks

Bug in encoding Counter objects?

I am i doing something wrong or is the following a bug?

Python 2.7.6 (default, Dec  2 2013, 22:30:26) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import jsonpickle as jp
>>> from collections import Counter
>>> a = Counter({1: 2})
>>> b = jp.encode(a, keys=True); b
'{"py/object": "collections.Counter", "json://1": 2}'
>>> c = jp.decode(b, keys=True); c
Counter({u'json://1': 2})
>>> a == c
False

Unwanted deserialization into Unicode is occuring

Hi,

I have a class structure with various ascii string data contained in it. I serialize it up and out; it writes out an ascii file (I've checked). When I deserialize it back into a class structure, the formerly ascii strings have become unicode strings. This is not acceptable for the consumers of this information.

This is an example:
>>> jsonpickle.decode(jsonpickle.encode("foo"))
u'foo'

This is really problematic, since I will have to manually walk the object graph and ascii-up the data.The documentation seems to suggest this won't happen, c.f., http://jsonpickle.github.com/#module-jsonpickle

I have scanned through jsonpickle to try and figure out what is causing this behavior - I don't see it offhand.

I appreciate the need to decode as Unicode, however, there is also a need to decode as ASCII, so I'm filing this as a bug. My ideal resolution would be a "decode_as_ascii" kwarg in the decode function.

Thank you for your time.

Version information

I pulled down the latest version from the master branch:

commit 186a038cccd549b2aad42eaf8a8fcd38182f7931 
Author: David Aguilar <[email protected]>
Date:   Sat Oct 2 08:13:57 2010 -0700

jsonpickle/__init__.py: Enable sorted keys by default

Enable sorted keys for json/simplejson backends.
We should do this for the other backends too.

Signed-off-by: David Aguilar <[email protected]>

Python 2.7.1 on Ubuntu
Standard library backend.

Further information

This appears to be driven by behavior in Python's native JSON parser, which evidently not only performs this behavior, but only does it occasionally, c.f, http://bugs.python.org/issue10038

Encoding tuples fails

Hi, i just gave jsonpickle a try for the first time, and couldn't get a tuple object to decode properly. It decoded to a string. Did i do something wrong below?

Python 2.7.6 (default, Dec  2 2013, 22:30:26) 
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import jsonpickle as jp

In [2]: d = {(0, 1): 5}

In [3]: d_json = jp.encode(d)

In [4]: d_json
Out[4]: '{"(0, 1)": 5}'

In [5]: e = jp.decode(d_json)

In [6]: e
Out[6]: {u'(0, 1)': 5}

In [7]: e[(0, 1)]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-c5b7b986b1ae> in <module>()
----> 1 e[(0, 1)]

KeyError: (0, 1)

In [8]: 

Bug in decoding OrderedDict objects?

Python 2.7.6 (default, Dec  2 2013, 22:30:26) 
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import jsonpickle as jp

In [2]: a = {1: OrderedDict([(2, 0), (3, 0)]), 4: OrderedDict([(5, 0), (6, 0)])}; a
Out[2]: {1: OrderedDict([(2, 0), (3, 0)]), 4: OrderedDict([(5, 0), (6, 0)])}

In [3]: a_j = jp.encode(a, keys=True); a_j
Out[3]: '{"json://4": {"py/object": "collections.OrderedDict", "__reduce__": [{"py/type": "collections.OrderedDict"}, {"py/tuple": [{"py/id": 2}]}]}, "json://1": {"py/object": "collections.OrderedDict", "__reduce__": [{"py/type": "collections.OrderedDict"}, {"py/tuple": [[[2, 0], [3, 0]]]}]}}'

In [4]: a_jj = jp.decode(a_j, keys=True); a_jj
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-68-43d809249ee1> in <module>()
----> 1 a_jj = jp.decode(a_j, keys=True); a_jj

/usr/local/lib/python2.7/site-packages/jsonpickle/__init__.pyc in decode(string, backend, keys)
    150     """
    151     if backend is None:
--> 152         backend = json
    153     return unpickler.decode(string, backend=backend, keys=keys)

/usr/local/lib/python2.7/site-packages/jsonpickle/unpickler.pyc in decode(string, backend, context, keys, reset)
     23         context = Unpickler(keys=keys, backend=backend)
     24     return context.restore(backend.decode(string), reset=reset)
---> 25 
     26 
     27 def _make_backend(backend):

/usr/local/lib/python2.7/site-packages/jsonpickle/unpickler.pyc in restore(self, obj, reset)
     69         {'key': 'value'}
     70         """
---> 71         if reset:
     72             self.reset()
     73         return self._restore(obj)

/usr/local/lib/python2.7/site-packages/jsonpickle/unpickler.pyc in _restore(self, obj)
     92         elif util.is_dictionary(obj):
     93             restore = self._restore_dict
---> 94         else:
     95             restore = lambda x: x
     96         return restore(obj)

/usr/local/lib/python2.7/site-packages/jsonpickle/unpickler.pyc in _restore_dict(self, obj)
    200         data = {}
    201         for k, v in sorted(obj.items(), key=util.itemgetter):
--> 202             self._namestack.append(k)
    203             if self.keys and k.startswith(tags.JSON_KEY):
    204                 k = decode(k[len(tags.JSON_KEY):],

/usr/local/lib/python2.7/site-packages/jsonpickle/unpickler.pyc in _restore(self, obj)
     92         elif util.is_dictionary(obj):
     93             restore = self._restore_dict
---> 94         else:
     95             restore = lambda x: x
     96         return restore(obj)

/usr/local/lib/python2.7/site-packages/jsonpickle/unpickler.pyc in _restore_object(self, obj)
    116         if cls is None:
    117             return self._mkref(obj)
--> 118         handler = handlers.get(cls)
    119         if handler is not None: # custom handler
    120             instance = handler(self).restore(obj)

/usr/local/lib/python2.7/site-packages/jsonpickle/handlers.pyc in restore(self, obj)
    132     """
    133 
--> 134     def flatten(self, obj, data):
    135         pickler = self.context
    136         if not pickler.unpicklable:

/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.pyc in __init__(self, *args, **kwds)
     50             root[:] = [root, root, None]
     51             self.__map = {}
---> 52         self.__update(*args, **kwds)
     53 
     54     def __setitem__(self, key, value, dict_setitem=dict.__setitem__):

/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.pyc in update(*args, **kwds)
    545                 self[key] = other[key]
    546         else:
--> 547             for key, value in other:
    548                 self[key] = value
    549         for key, value in kwds.items():

TypeError: 'int' object is not iterable

Undefined variable: data

On line143 of pickler.py, the flatten function, PyDev in Eclipse is reporting that data is undefined:

        return self._pop(data)

unpicklable raises RuntimeError: maximum recursion depth exceeded

Using this class:

class Hop(SQLObject):
    BITTERING = 0 
    AROMA = 1
    BOTH = 2
    LEAF = 0
    PELLET = 1
    PLUG = 2
    hop_types = ['Bittering', 'Aroma', 'Both',]
    hop_forms = ['Leaf', 'Pellet', 'Plug',]

    hop_type = IntCol(default=BITTERING)
    hop_form = IntCol(default=LEAF)
    alpha = PercentCol(default=0.0)
    beta = PercentCol(default=0.0)
    stability = PercentCol(default=0.0)
    origin = UnicodeCol(default=None)
    name = UnicodeCol(length=64, default=None)
    description = UnicodeCol(default=None)
    substitutes = RelatedJoin('Hop',
                              joinColumn='master_hop',
                              otherColumn='substitute_hop',
                              addRemoveName="Substitute",
                              intermediateTable="substitute_hops",
                              createRelatedTable=True)
    versions = Versioning()

    def _get_hop_type(self):
        return Hop.hop_types[self._SO_get_hop_type()]

    def _get_hop_form(self):
        return Hop.hop_forms[self._SO_get_hop_form()]    

    def _get_short_description(self):
        if self.description and len(self.description) > 64:
            return self.description[:64]+"..."
        else:
            return self.description

    def _get_inventory(self):
        try:
            inv = list(Inventory.select(Inventory.q.hop == self.id))[0]
        except:
            return 0

        return "%s %s" % (inv.amount, Measure.Measure[inv.amount_units])

>>> import jsonpickle
>>> import beerlog
>>> hop = beerlog.Hop.get(1)
>>> hop
<Hop 1 hop_type='Bittering' hop_form='Leaf' alpha=0 beta=0 stability=0 origin=None name=u'Amarillo' description=None>
>>> jsonpickle.encode(hop)
'{"py/object": "beerlog.brewery.models.Hop", "py/state": {"_SO_val_origin": null, "_SO_validatorState": {"py/object": "sqlobject.sqlbuilder.SQLObjectState", "connection": null, "soObject": {"py/object": "beerlog.brewery.models.Hop", "py/state": {"_SO_val_origin": null, "_SO_validatorState": {"py/id": 1}, "_SO_val_hop_type": 0, "_SO_val_stability": 0, "_SO_val_name": "Amarillo", "id": 1, "_SO_val_alpha": 0, "_SO_val_hop_form": 0, "_SO_val_beta": 0, "_SO_val_description": null, "_SO_createValues": {}}}}, "_SO_val_hop_type": 0, "_SO_val_stability": 0, "_SO_val_name": "Amarillo", "id": 1, "_SO_val_alpha": 0, "_SO_val_hop_form": 0, "_SO_val_beta": 0, "_SO_val_description": null, "_SO_createValues": {}}}'
>>> jsonpickle.decode(jsonpickle.encode(hop))
<Hop 1 hop_type='Bittering' hop_form='Leaf' alpha=0 beta=0 stability=0 origin=None name=u'Amarillo' description=None>
>>> jsonpickle.encode(hop, unpicklable=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/__init__.py", line 313, in encode
    return json.encode(j.flatten(value))
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 197, in _flatten_obj_instance
    state = self.flatten(obj.__getstate__())
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 127, in flatten
    return self._pop(self._flatten_dict_obj(obj, obj.__class__()))
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 136, in flatten
    return self._pop(self._flatten_obj_instance(obj))
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 206, in _flatten_obj_instance
    return self._flatten_dict_obj(obj.__dict__, data)
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 215, in _flatten_dict_obj
    self._flatten_key_value_pair(k, v, data)
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 234, in _flatten_key_value_pair
    data[k] = self.flatten(v)

# keeps going quite a while!!!

  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/pickler.py", line 129, in flatten
    if util.is_type(obj):
  File "/Users/tkennedy/src/beerlog/env/lib/python2.7/site-packages/jsonpickle/util.py", line 40, in is_type
    return type(obj) is types.TypeType or repr(obj).startswith('<class')
  File "/Library/Python/2.7/site-packages/sqlobject/main.py", line 1613, in __repr__
    ' '.join(['%s=%s' % (name, repr(value)) for name, value in self._reprItems()]))
  File "/Library/Python/2.7/site-packages/sqlobject/main.py", line 1632, in _reprItems
    value = getattr(self, col.name)
  File "beerlog/brewery/models.py", line 38, in _get_hop_type
    return Hop.hop_types[self._SO_get_hop_type()]
RuntimeError: maximum recursion depth exceeded
>>> 

Test failures with simplejson not installed

When simplejson is not installed, then test suite fails. I use jsonpickle 0.6.0.

Results with Python 2.7 and simplejson not installed:

$ PYTHONPATH="$(pwd)" python2.7 tests/runtests.py
...
======================================================================
ERROR: test_backend (backends_tests.SimpleJsonTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/backends_tests.py", line 62, in setUp
    self.set_preferred_backend('simplejson')
  File "/tmp/jsonpickle-0.6.0/tests/backends_tests.py", line 27, in set_preferred_backend
    self._is_installed(backend)
  File "/tmp/jsonpickle-0.6.0/tests/backends_tests.py", line 16, in _is_installed
    self.fail('%s not available; please install' % backend)
AssertionError: simplejson not available; please install

----------------------------------------------------------------------
Ran 185 tests in 0.925s

FAILED (errors=1)

Results with Python 3.3 and simplejson not installed:

$ PYTHONPATH="$(pwd)" python3.3 tests/runtests.py
...
======================================================================
ERROR: test_datetime_inside_int_keys_defaults (jsonpickle_test.PicklingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/jsonpickle_test.py", line 388, in test_datetime_inside_int_keys_defaults
    s = jsonpickle.encode({1:t, 2:t})
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 137, in encode
    return self._encoders[name](*encoder_args, **encoder_kwargs)
  File "/usr/lib64/python3.3/json/__init__.py", line 236, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python3.3/json/encoder.py", line 191, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.3/json/encoder.py", line 249, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python3.3/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'CgAAAAAA' is not JSON serializable

======================================================================
ERROR: test_datetime_inside_int_keys_with_keys_enabled (jsonpickle_test.PicklingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/jsonpickle_test.py", line 396, in test_datetime_inside_int_keys_with_keys_enabled
    s = jsonpickle.encode({1:t, 2:t}, keys=True)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 137, in encode
    return self._encoders[name](*encoder_args, **encoder_kwargs)
  File "/usr/lib64/python3.3/json/__init__.py", line 236, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python3.3/json/encoder.py", line 191, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.3/json/encoder.py", line 249, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python3.3/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'CgAAAAAA' is not JSON serializable

======================================================================
ERROR: test_datetime_dict_keys_with_keys_enabled (jsonpickle_test.JSONPickleTestCase)
Test that we handle datetime objects as keys.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/jsonpickle_test.py", line 585, in test_datetime_dict_keys_with_keys_enabled
    pickled = jsonpickle.encode(datetime_dict, keys=True)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 120, in flatten
    return self._flatten(obj)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 133, in _flatten
    return self._pop(flatten_func(obj))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 254, in _flatten_dict_obj
    flatten(k, v, data)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 278, in _flatten_key_value_pair
    make_refs=self.make_refs)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 137, in encode
    return self._encoders[name](*encoder_args, **encoder_kwargs)
  File "/usr/lib64/python3.3/json/__init__.py", line 236, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python3.3/json/encoder.py", line 191, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.3/json/encoder.py", line 249, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python3.3/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'B9gMHwAAAAAAAA==' is not JSON serializable

======================================================================
ERROR: test (thirdparty_tests.FeedParserTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/thirdparty_tests.py", line 65, in setUp
    self.doc = feedparser.parse(RSS_DOC)
  File "/usr/lib64/python3.3/site-packages/feedparser.py", line 3984, in parse
    saxparser.parse(source)
  File "/usr/lib64/python3.3/site-packages/drv_libxml2.py", line 189, in parse
    eltName = (_d(reader.NamespaceUri()),\
  File "/usr/lib64/python3.3/site-packages/drv_libxml2.py", line 70, in _d
    return _decoder(s)[0]
  File "/usr/lib64/python3.3/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
TypeError: 'str' does not support the buffer interface

======================================================================
ERROR: test_date (datetime_tests.DateTimeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 54, in test_date
    self._roundtrip(datetime.datetime.today())
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 40, in _roundtrip
    pickled = jsonpickle.encode(obj)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 134, in encode
    optargs, optkwargs = self._encoder_options[name]
KeyError: 'os.path'

======================================================================
ERROR: test_datetime (datetime_tests.DateTimeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 48, in test_datetime
    self._roundtrip(datetime.datetime.now())
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 40, in _roundtrip
    pickled = jsonpickle.encode(obj)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 134, in encode
    optargs, optkwargs = self._encoder_options[name]
KeyError: 'os.path'

======================================================================
ERROR: test_object_with_datetime (datetime_tests.DateTimeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 87, in test_object_with_datetime
    json = jsonpickle.encode(test_obj)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 134, in encode
    optargs, optkwargs = self._encoder_options[name]
KeyError: 'os.path'

======================================================================
ERROR: test_time (datetime_tests.DateTimeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 60, in test_time
    self._roundtrip(datetime.datetime.now().time())
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 40, in _roundtrip
    pickled = jsonpickle.encode(obj)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 134, in encode
    optargs, optkwargs = self._encoder_options[name]
KeyError: 'os.path'

======================================================================
ERROR: test_utc (datetime_tests.DateTimeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 73, in test_utc
    self._roundtrip(datetime.datetime.utcnow().replace(tzinfo=utc))
  File "/tmp/jsonpickle-0.6.0/tests/datetime_tests.py", line 40, in _roundtrip
    pickled = jsonpickle.encode(obj)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/__init__.py", line 134, in encode
    max_depth=max_depth)
  File "/tmp/jsonpickle-0.6.0/jsonpickle/pickler.py", line 31, in encode
    return backend.encode(context.flatten(value, reset=reset))
  File "/tmp/jsonpickle-0.6.0/jsonpickle/backend.py", line 134, in encode
    optargs, optkwargs = self._encoder_options[name]
KeyError: 'os.path'

======================================================================
FAIL: test_backend (backends_tests.SimpleJsonTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.6.0/tests/backends_tests.py", line 62, in setUp
    self.set_preferred_backend('simplejson')
  File "/tmp/jsonpickle-0.6.0/tests/backends_tests.py", line 27, in set_preferred_backend
    self._is_installed(backend)
  File "/tmp/jsonpickle-0.6.0/tests/backends_tests.py", line 16, in _is_installed
    self.fail('%s not available; please install' % backend)
AssertionError: simplejson not available; please install

----------------------------------------------------------------------
Ran 185 tests in 0.662s

FAILED (failures=1, errors=9)

GAE not finding django.utils.simplejson

Reported by gordonmcdowell http://code.google.com/u/gordonmcdowell/ on Dec 04, 2009

Within app-engine-patch-sample of Google App Engine, JSONResponse should be available in
ragendja.template.

Having jsonpickle present in the app and imported, I'd expect that...

def list_people(request):
    return TextResponse( jsonpickle.encode(Person.all().fetch(100), False, 4) )

...would return some JSON structured results. But I get...

jsonpickle requires at least one of the following: cjson, json (new in python2.6), simplejson,
demjson

...I find by altering jsonpickle/init.py line from...

self.load_backend('simplejson', 'dumps', 'loads', ValueError)

...into...

self.load_backend('django.utils.simplejson', 'dumps', 'loads', ValueError)

...I get expected JSON results. How to do this was not immediately obvious to me, so sharing.

Please add to the docs the requirement that jsonpicke.decode only reconstructs objects with no-parameter constructors

Please add to the docs the requirement that jsonpicke.decode only reconstructs objects with no-parameter constructors.

I had to dig through the source to discover why my object wasn't being reconstructed. Could a sentence please be added to this effect to the Usage section (http://jsonpickle.github.com/#module-jsonpickle)?

__slots__ inheriting __slots__

First -- Amazing work on jsonpickle recently -- last used it at 0.40 and wasn't able to make it do what I needed; with 0.70 it's working on really complex structures extremely well. Kudos!

A little bug that I noticed, unrelated to the other slots issues: Slotted objects can inherit from other Slotted objects without redefining the slots. So in _samples, I took:

class ThingWithSlots(object):

    __slots__ = ('a', 'b')

    def __init__(self, a, b):
        self.a = a
        self.b = b

and added

class ThingWithSlotsInheritingSlots(ThingWithSlots):

    __slots__ = ('c',)

    def __init__(self, a, b, c):
        ThingWithSlots.__init__(self, a, b)
        self.c = c

With jsonpickle.pickler._flatten_newstyle_with_slots only obj.c will be stored and when unpicked, obj.a and obj.b will return an AttributeError. A quick patch is:

    def _flatten_newstyle_with_slots(self, obj, data):
        """Return a json-friendly dict for new-style objects with __slots__.
        """
        nestedSlots = [getattr(cls, '__slots__', tuple() ) for cls in type(obj).__mro__]
        mroSlotsList = [i for sub in nestedSlots for i in sub]
        for k in mroSlotsList:
            self._flatten_key_value_pair(k, getattr(obj, k), data)
        return data

for k in itertools.chain([getattr(cls, '__slots__', tuple() ) for cls in type(obj).__mro__]) would also work, if you want to use itertools.

Since this method ends up creating at least two new lists for every slotted object, it might be more efficient to cache the mroSlotsList for each cls type, but this works pretty fast in my case. It unpickles into an object with the proper __slots__ (just ('c',) ) but .a and .b are also set.

Add option for turning off "py/ref" reference type for non-cyclical objects

Reported by pimlottc http://code.google.com/u/pimlottc/ on Oct 08, 2009

The "py/ref" keys used by jsonpickle to create a reference to another json
node in the same response is non-standard and creates a barrier to
interoperability.

Since references only strictly necessary for cyclical objects, most objects
could be serialized without references by simply repeating the same json
fragment in each place an object is referenced. There should be an option
added to enable this support.

Pickler._flatten_obj_instance returns None for some objects

Reported by george.sakkis http://code.google.com/u/george.sakkis/ Dec 06 (5 days ago)

_flatten_obj_instance() falls through and returns None if an object doesn't
have dict or slots, e.g. file and threading.Lock objects.

Fix is easy, just add a "return data" at the end of the method. A more
general approach is to not rely on dict or slots but on (a subset
of) the attributes returned by dir(obj).

JsonPickle & Mongoengine

I was trying to see how would jsonpickle work with mongoengine.
This is a test case I am trying.

from mongoengine import *
import simplejson
import jsonpickle

connect('carbon-server-playground-mongoengine')

class User(Document):
    email = StringField(required=True)
    first_name = StringField(max_length=50)
    last_name = StringField(max_length=50)

class Comment(EmbeddedDocument):
    content = StringField()
    name = StringField(max_length=120)

class Post(Document):
    title = StringField(max_length=120, required=True)
    author = ReferenceField(User)
    comments = ListField(EmbeddedDocumentField(Comment))

if(len(Post.objects)==0):
    john = User(email='[email protected]', first_name='John', last_name='Doe')
    john.save()

    post1 = Post(title='Fun with MongoEngine', author=john)
    post1.save()

Now you can see two different errors by either adding
print jsonpickle.encode(Post.objects[0])
or
print jsonpickle.encode(Post.objects)

The second one most likely needs the getstate & setstate methods added on the collection objects if I ain't wrong.

As for the first one I have no idea as it breaks in pickler class.
# hack for zope persistent objects; this unghostifies the object
getattr(obj, '_', None)
return self._flatten_dict_obj(obj.dict, data)

I'm a nab in python [php plx] and can't really debug this thing yet.
Anyone interested in this too?

Dict keys

Have a big trouble with dict keys.
Now, only unicode-string keys supported (int keys converts to unicode string and remains unicode string after decode). So, dict's after encode/decode change the type of keys. That erroneously in much applications.

Test failures with Python 3

jsonpickle 0.5.0 claims to support Python 3, but it has 8 failures and 50 errors with Python 3.

A subset of failures/errors:

$ PYTHONPATH="$(pwd)" python3.3 tests/runtests.py
...
======================================================================
ERROR: test_long (util_tests.PrimitiveTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.5.0/tests/util_tests.py", line 41, in test_long
    self.assertTrue(is_primitive(long(3)))
NameError: global name 'long' is not defined

======================================================================
ERROR: test_class (jsonpickle_test.PicklingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.5.0/tests/jsonpickle_test.py", line 225, in test_class
    flattened = self.pickler.flatten(inst)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 121, in flatten
    return self._flatten(obj)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 132, in _flatten
    flatten_func = self._get_flattener(obj)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 164, in _get_flattener
    if util.is_type(obj):
  File "/tmp/jsonpickle-0.5.0/jsonpickle/util.py", line 38, in is_type
    return type(obj) is type or type(obj) is types.ClassType
AttributeError: 'module' object has no attribute 'ClassType'

======================================================================
ERROR: test_set_notunpicklable (jsonpickle_test.PicklingTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.5.0/tests/jsonpickle_test.py", line 370, in test_set_notunpicklable
    self.assertEqual(sorted(flattened), sorted(['one', 2, 3]))
TypeError: unorderable types: int() < str()

======================================================================
ERROR: test_decode (jsonpickle_test.JSONPickleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.5.0/tests/jsonpickle_test.py", line 509, in test_decode
    unpickled = jsonpickle.decode(self.expected_json)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 149, in decode
    return unpickler.decode(string, backend=backend, keys=keys)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/unpickler.py", line 25, in decode
    return context.restore(backend.decode(string), reset=reset)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/unpickler.py", line 71, in restore
    return self._restore(obj)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/unpickler.py", line 94, in _restore
    return restore(obj)
  File "/tmp/jsonpickle-0.5.0/jsonpickle/unpickler.py", line 116, in _restore_object
    handler = handlers.BaseHandler._registry.get(cls)
AttributeError: type object 'BaseHandler' has no attribute '_registry'

======================================================================
ERROR: test_unicode_dict_keys (jsonpickle_test.JSONPickleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.5.0/tests/jsonpickle_test.py", line 522, in test_unicode_dict_keys
    pickled = jsonpickle.encode({'é'.decode('utf-8'): 'é'.decode('utf-8')})
AttributeError: 'str' object has no attribute 'decode'

======================================================================
ERROR: test (thirdparty_tests.FeedParserTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/jsonpickle-0.5.0/tests/thirdparty_tests.py", line 65, in setUp
    self.doc = feedparser.parse(RSS_DOC)
  File "/usr/lib64/python3.3/site-packages/feedparser.py", line 3984, in parse
    saxparser.parse(source)
  File "/usr/lib64/python3.3/site-packages/drv_libxml2.py", line 189, in parse
    eltName = (_d(reader.NamespaceUri()),\
  File "/usr/lib64/python3.3/site-packages/drv_libxml2.py", line 70, in _d
    return _decoder(s)[0]
  File "/usr/lib64/python3.3/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
TypeError: 'str' does not support the buffer interface

======================================================================
FAIL: is_function (jsonpickle.util)
Doctest: jsonpickle.util.is_function
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.3/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for jsonpickle.util.is_function
  File "/tmp/jsonpickle-0.5.0/jsonpickle/util.py", line 159, in is_function

----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/util.py", line 165, in jsonpickle.util.is_function
Failed example:
    is_function(locals)
Expected:
    True
Got:
    False


======================================================================
FAIL: is_function (jsonpickle.util)
Doctest: jsonpickle.util.is_function
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.3/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for jsonpickle.util.is_function
  File "/tmp/jsonpickle-0.5.0/jsonpickle/util.py", line 159, in is_function

----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/util.py", line 165, in jsonpickle.util.is_function
Failed example:
    is_function(locals)
Expected:
    True
Got:
    False


======================================================================
FAIL: flatten (jsonpickle.pickler.Pickler)
Doctest: jsonpickle.pickler.Pickler.flatten
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.3/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for jsonpickle.pickler.Pickler.flatten
  File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 89, in flatten

----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 97, in jsonpickle.pickler.Pickler.flatten
Failed example:
    p.flatten(u'hello world')
Expected:
    u'hello world'
Got:
    'hello world'


======================================================================
FAIL: _getclassdetail (jsonpickle.pickler)
Doctest: jsonpickle.pickler._getclassdetail
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.3/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for jsonpickle.pickler._getclassdetail
  File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 311, in _getclassdetail

----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 317, in jsonpickle.pickler._getclassdetail
Failed example:
    _getclassdetail(25)
Expected:
    ('__builtin__', 'int')
Got:
    ('builtins', 'int')
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 319, in jsonpickle.pickler._getclassdetail
Failed example:
    _getclassdetail(None)
Expected:
    ('__builtin__', 'NoneType')
Got:
    ('builtins', 'NoneType')
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 321, in jsonpickle.pickler._getclassdetail
Failed example:
    _getclassdetail(False)
Expected:
    ('__builtin__', 'bool')
Got:
    ('builtins', 'bool')


======================================================================
FAIL: _mktyperef (jsonpickle.pickler)
Doctest: jsonpickle.pickler._mktyperef
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.3/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for jsonpickle.pickler._mktyperef
  File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 299, in _mktyperef

----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 303, in jsonpickle.pickler._mktyperef
Failed example:
    _mktyperef(AssertionError)[tags.TYPE].rsplit('.', 1)[0]
Expected:
    'exceptions'
Got:
    'builtins'


======================================================================
FAIL: jsonpickle ()
Doctest: jsonpickle
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.3/doctest.py", line 2154, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for jsonpickle
  File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 9, in jsonpickle

----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 22, in jsonpickle
Failed example:
    print obj.name
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[2]>", line 1
        print obj.name
                ^
    SyntaxError: invalid syntax
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 28, in jsonpickle
Failed example:
    pickled = jsonpickle.encode(obj)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[4]>", line 1, in <module>
        pickled = jsonpickle.encode(obj)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 131, in encode
        max_depth=max_depth)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 30, in encode
        return backend.encode(context.flatten(value, reset=reset))
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 121, in flatten
        return self._flatten(obj)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 132, in _flatten
        flatten_func = self._get_flattener(obj)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 164, in _get_flattener
        if util.is_type(obj):
      File "/tmp/jsonpickle-0.5.0/jsonpickle/util.py", line 38, in is_type
        return type(obj) is type or type(obj) is types.ClassType
    AttributeError: 'module' object has no attribute 'ClassType'
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 29, in jsonpickle
Failed example:
    print(pickled)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[5]>", line 1, in <module>
        print(pickled)
    NameError: name 'pickled' is not defined
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 34, in jsonpickle
Failed example:
    unpickled = jsonpickle.decode(pickled)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[6]>", line 1, in <module>
        unpickled = jsonpickle.decode(pickled)
    NameError: name 'pickled' is not defined
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 35, in jsonpickle
Failed example:
    str(unpickled.name)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[7]>", line 1, in <module>
        str(unpickled.name)
    NameError: name 'unpickled' is not defined
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 46, in jsonpickle
Failed example:
    obj == unpickled
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[8]>", line 1, in <module>
        obj == unpickled
    NameError: name 'unpickled' is not defined
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 48, in jsonpickle
Failed example:
    obj.name == unpickled.name
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[9]>", line 1, in <module>
        obj.name == unpickled.name
    NameError: name 'unpickled' is not defined
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 50, in jsonpickle
Failed example:
    type(obj) == type(unpickled)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[10]>", line 1, in <module>
        type(obj) == type(unpickled)
    NameError: name 'unpickled' is not defined
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 57, in jsonpickle
Failed example:
    oneway = jsonpickle.encode(obj, unpicklable=False)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[11]>", line 1, in <module>
        oneway = jsonpickle.encode(obj, unpicklable=False)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 131, in encode
        max_depth=max_depth)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 30, in encode
        return backend.encode(context.flatten(value, reset=reset))
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 121, in flatten
        return self._flatten(obj)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 132, in _flatten
        flatten_func = self._get_flattener(obj)
      File "/tmp/jsonpickle-0.5.0/jsonpickle/pickler.py", line 164, in _get_flattener
        if util.is_type(obj):
      File "/tmp/jsonpickle-0.5.0/jsonpickle/util.py", line 38, in is_type
        return type(obj) is type or type(obj) is types.ClassType
    AttributeError: 'module' object has no attribute 'ClassType'
----------------------------------------------------------------------
File "/tmp/jsonpickle-0.5.0/jsonpickle/__init__.py", line 58, in jsonpickle
Failed example:
    print oneway
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python3.3/doctest.py", line 1287, in __run
        compileflags, 1), test.globs)
      File "<doctest jsonpickle[12]>", line 1
        print oneway
                   ^
    SyntaxError: invalid syntax


----------------------------------------------------------------------
Ran 167 tests in 1.022s

FAILED (failures=8, errors=50)

__getstate__ is ignored for dict subclasses

When jsonpickling subclasses of dict attributes defined on the subclass go missing (which you might consider an unrelated issue). Trying to work around this via the get/setstate protocol does not work either as getstate is not called:

class MyClass( dict ) :
    def __init__( self, new_attr, **kwargs ) :
        self.new_attr = new_attr
        dict.__init__( self, **kwargs )

    def __getstate__( self ) :
        return ( self.new_attr, dict( self.iteritems() ) )

    def __setstate__( self, state ) :
        self.new_attr, vals = state
        self.update( vals )

tbl = MyClass( 'a', b=1, c=2 )

s = jsonpickle.encode( tbl )
print s
r =  jsonpickle.decode(s)
r.new_attr

Fails with:
{ "py/object": "__main__.MyClass", "c": 2, "b": 1 } Traceback (most recent call last): File "C:\Sandboxes\ETRM\bug_reports\jsonpickle_dict_subclasses_ignore__getstate__.py", line 20, in <module> r.new_attr AttributeError: 'MyClass' object has no attribute 'new_attr'

Obviously new_attr already goes missing in the encode step.

Andreas

is_picklable(): Return True if an object cannot be pickled?

The documentation mentions that json.util.is_picklable "Return True if an object cannot be pickled".

Shouldn't it instead be "Return True if an object can be pickled."?

jsonpickle imports fail when not installed in PYTHONPATH

At least on my system.

I wanted to just install jsonpickle in a lib subdirectory of a project (lib is in PYTHONPATH) but all the local import statements in jsonpickle failed. This can be solved easily using relative import paths in the module (as explained in http://docs.python.org/tutorial/modules.html#intra-package-references).

I've created a patch, which I'll attach as soon as I figure out how.

Support for iterable __slots__

I've discovered recently that Python allows __slots__ to be an arbitrary iterable. Consider:

>>> class X:
>>>   __slots__ = iter('abc')
>>> x = X()
>>> print(list(x.__slots__))
[]

ugh.

This outcome is a problem for jsonpickle because it relies on __slots__ to serialize the object.