laymonage/django-jsonfield-backport

Backport of the cross-DB JSONField model and form fields from Django 3.1.

django-jsonfield-backport

Backport of the cross-DB JSONField model and form fields from Django 3.1.

from django.db import models
from django_jsonfield_backport.models import JSONField

class ContactInfo(models.Model):
    data = JSONField()

ContactInfo.objects.create(data={
    'name': 'John',
    'cities': ['London', 'Cambridge'],
    'pets': {'dogs': ['Rufus', 'Meg']},
})
ContactInfo.objects.filter(
    data__name='John',
    data__pets__has_key='dogs',
    data__cities__contains='London',
).delete()

Features

Most features of the JSONField model and form fields from Django 3.1 are supported.

• MariaDB, MySQL, Oracle, PostgreSQL, and SQLite support.
• JSONField lookups and transforms support.
• Custom encoder and decoder support.

Due to limited access to Django's APIs, some features are not supported.

• Introspection is not supported.
• On MariaDB and Oracle, Casting to JSONField must be done using the included JSONCast class.

This package is fully compatible with the JSONField from Django 3.1. That means you just need to change your imports and edit your migrations when you finally upgrade to Django 3.1. If you leave them as they are, this package will use the built-in JSONField and system warnings will be raised.

Requirements

This package supports and is tested against the latest patch versions of:

• Python: 3.5 (Django 2.2 only), 3.6, 3.7, 3.8, 3.9
• Django: 2.2, 3.0, 3.1
• MariaDB: 10.2, 10.3, 10.4, 10.5
• MySQL: 5.7, 8.0
• Oracle: 12.2+ (only tested against 12.2.0.1 SE)
• PostgreSQL: 9.5, 10, 11, 12
• SQLite: 3.9.0+ with the JSON1 extension

All database backends are tested with the latest versions of their drivers. SQLite is also tested on GitHub Actions' latest macOS and Windows virtual environments.

Installation

1. Use pip or your preferred dependency management tool to install the package.

   $ pip install django-jsonfield-backport

2. Add "django_jsonfield_backport" to INSTALLED_APPS in your settings.

   INSTALLED_APPS = [
       ...
       "django_jsonfield_backport",
   ]

Usage

To use the model and form fields, import JSONField from django_jsonfield_backport.models and django_jsonfield_backport.forms, respectively.

Model field example:

from django.db import models
from django_jsonfield_backport.models import JSONField

class ContactInfo(models.Model):
    data = JSONField()

Form field example:

from django import forms
from django_jsonfield_backport.forms import JSONField

class ContactForm(forms.Form):
    data = JSONField()

JSONCast, KeyTransform, and KeyTextTransform classes are also available from django_jsonfield_backport.models.

Documentation

Since this package is a backport, the official Django 3.1 docs for models.JSONField_ and forms.JSONField_ are mostly compatible with this package.

Rationale

As of the creation of this package, JSONField implementations exist in multiple packages on PyPI:

• Django: Before Django 3.1, PostgreSQL-only JSONField exists in the contrib.postgres module.
• jsonfield: 1.1k stars, cross-DB support with no extended querying capabilities.
• django-annoying: 787 stars, has a TextField-based JSONField with no extended querying capabilities.
• Django-MySQL: 364 stars, has a MariaDB/MySQL-only JSONField with extended querying capabilities (not entirely the same as in contrib.postgres).
• django-jsonfallback: 26 stars, uses JSONField from contrib.postgres and Django-MySQL before falling back to TextField-based JSONField.
• django-json-field: 116 stars, TextField-based JSONField with custom encoder and decoder support with no extended querying capabilities (unmaintained).
• django-jsonfield: 21 stars, cross-DB support with no extended querying capabilities.
• django-jsonfield-compat: 8 stars, compatibility layer for contrib.postgres JSONField and django-jsonfield.
• oracle-json-field: 2 stars, Oracle-only JSONField with extended querying capabilities (not entirely the same as in contrib.postgres).

Along with other unmaintained packages such as dj-jsonfield, vlk-django-jsonfield, linaro-django-jsonfield, jsonfield2, django-jsonfield2, django-softmachine, django-simple-jsonfield, easy_jsonfield, and django-jsonbfield.

Why create another one?

Up until the new JSONField in Django 3.1, there had been no implementation of JSONField that supports all the database backends supported by Django with more or less the same functionalities as the contrib.postgres JSONField provides.

Django's release process does not backport new features to previous feature releases. However, the current LTS release is 2.2 which is still supported until April 2022. The next LTS release is Django 3.2 in April 2021 that happens to be the end of extended support for Django 3.1.

Some projects only use LTS releases of Django. There are also incompatibilities between Django 3.0 and 3.1. Therefore, using Django 3.1 may not be an option for some people at the moment.

Since JSONField seems to be in popular demand and that it works well as a standalone package, I decided to create a backport.

Besides, I'm the co-author of the new JSONField. ¯\_(ツ)_/¯

License

This package is licensed under the BSD 3-Clause License.

django-jsonfield-backport's Issues

Database backends compatibility

Some parts of the code in django/django#12392 were split and moved into the internal parts of Django. Taking only the model field's code breaks some of the field's functionalities (notably lookups and transforms) in some database backends. We do not have access to modify parts of Django anymore, so this backport needs to incorporate the split parts back into the field's code.

We need to run the tests on all database backends. GitHub Actions can help us for this.

To-do:

• GitHub Actions configuration for tests
• MariaDB
• MySQL
• Oracle
• PostgreSQL
• SQLite

django-jsonfield-backport 1.0.1 requires Django>2.2, but you'll have django 2.2 which is incompatible.

I'm currently using django 2.2 and faced the bug described here: https://code.djangoproject.com/ticket/30335

This led me to this backport. I'm confused though, every time I try to install it pip install django-jsonfield-backport, it forces an upgrade to django 3.1, which is what I'm trying to avoid. If I force django back to 2.2, I get the error from the title django-jsonfield-backport 1.0.1 requires Django>2.2, but you'll have django 2.2 which is incompatible..

The docs say:

Requirements
This package supports and is tested against the latest patch versions of:

Django: 2.2, 3.0, 3.1

Am I missing something with the installation? Or am I misunderstanding the purpose of this package?

Thanks

feature_classes and django-salesforce

django-salesforce is an ORM integration to the Salesforce API.
It gets set up like any other database source with it's backend identifying as vendor salesforce.

This leads to a KeyError in

django-jsonfield-backport/src/django_jsonfield_backport/features.py

Line 65 in 56679e1

value = feature = getattr(feature_classes[connection.vendor], name)

because the vendor cannot be found in the feature_classes map.

What would be the best way to handle unsupported/unknown database vendors so that I can offer a pull request?

Replace existing JSONField type with new JSONField type from django-jsonfield-backport

Hey, this is not an issue but a question.

Question

Currently, I'm using JSONField from django_mysql I want to replace this field type with the new type from django-jsonfield-backport

Current Code

from django_mysql.models.fields import JSONField

class Logs(models.Model):
    diff = JSONField(
        help_text="Stores the changed fields and their values",
        encoder=simplejson.JSONEncoder(use_decimal=True, allow_nan=False),
        decoder=simplejson.JSONDecoder(parse_float=Decimal, strict=False),
    )

Using django-jsonfield-backport

from django_jsonfield_backport.models import JSONField

class Logs(models.Model):
    diff = JSONField(
        help_text="Stores the changed fields and their values",
        encoder=simplejson.JSONEncoder(use_decimal=True, allow_nan=False),
        decoder=simplejson.JSONDecoder(parse_float=Decimal, strict=False),
    ) 

Will this change affects the existing data in the database?

Support Django >= 3.1

Fallback to the built-in JSONField and raise a system warning, much like the deprecated contrib.postgres JSONField in Django 3.1.

AttributeError: 'DatabaseFeatures' object has no attribute 'has_native_json_field'

python 3.6.8
django 2.2.18

File "\yuemia\TianQing\venv\lib\site-packages\django\db\models\query.py", line 72, in __iter__
  for row in compiler.results_iter(results):
File "\yuemia\TianQing\venv\lib\site-packages\django\db\models\sql\compiler.py", line 1086, in apply_converters
  value = converter(value, expression, connection)
File "\yuemia\TianQing\venv\lib\site-packages\django_jsonfield_backport\models.py", line 111, in from_db_value
  if connection.features.has_native_json_field and self.decoder is None:
AttributeError: 'DatabaseFeatures' object has no attribute 'has_native_json_field'

Signal receiver registration is improperly configured

As of #9, we now implement the feature checking by adding attributes to active database connections via signals. However, since we don't turn this into a Django app, the signal registration doesn't happen unless the user imports django_jsonfield_backport.features before they do anything that creates a database connection (e.g. querying). This results in AttributeErrors that happen every time there's a feature check in the code.

We can either add an app config or document that users need to register the signals themselves.

Fix for has_key lookups with numeric keys on MariaDB, MySQL, Oracle, and SQLite

https://github.com/django/django/pull/15503/files - this fix has to be applied to allow filtering with has_key with numeric keys on MariaDB, MySQL, Oracle, and SQLite

Cannot use callable default property on model ForeignKeys

Whenever I use something like this in my model:

system_currency = models.ForeignKey(Currency, on_delete=models.PROTECT, to_field='code', default=Currency.objects.first().code)

I get AttributeError: 'DatabaseFeatures' object has no attribute 'has_native_json_field' when trying to query a model that contains JSONField. Surprisingly, it doesn't seem to affect admin interface.

Changing the default property to constant value fixes the issue:

system_currency = models.ForeignKey(Currency, on_delete=models.PROTECT, to_field='code', default='eur')

It seems that maybe using queries inside model definitions somehow initializes the db config before this library can inject itself into it. Moving the library into the first place in INSTALLED_APPS doesn't help.

Phew! Took me a while to figure this out!

JSONField does not decode the serialized JSON and returns a string

Hi!

First of all, thank you for this package. It is very much needed as it brings some sanity into django - json world.

I'm running Django 2.2 LTS with psql 12. I've added a JSONField to my model, all works good. I can persist some data and make queries. Nonetheless, when I retrieve an object with the JSONField, the field is an ordinary string and not a Python dict as one would expect:

>>> e.extra_headers = {'foo': 123}
>>> e.save()
>>> e.refresh_from_db()
>>> e.extra_headers
'{"foo": 123}'
>>> type(e.extra_headers)
<class 'str'>

On this line:

django-jsonfield-backport/src/django_jsonfield_backport/models.py

Line 111 in 2072fb3

if connection.features.has_native_json_field and self.decoder is None:

The connection.features.has_native_json_field is True and self.decoder is None.

If I explicitly set decoder to json.JSONDecoder on my field (even though Django docs mention this is the default anyway), everything seems to work as intended.

Any ideas what's wrong and what would be the proper solution to this?

Thanks!

Backport fix for QuerySet.order_by() chained with values() on PostgreSQL with a custom decoder

TR-31956

django/django#13358

This may be a backward-incompatible fix if we're going to consider users who use both this package and the old contrib.postgres implementation.

DeprecationWarning on Django 3.0+

I found this on Django 3.0 when integrating in Django-MySQL:

File ".../django_jsonfield_backport/apps.py", line 10, in JSONFieldConfig
  verbose_name = _("JSONField backport from Django 3.1")
File ".../django/utils/translation/__init__.py", line 142, in ugettext_lazy
  warnings.warn(
django.utils.deprecation.RemovedInDjango40Warning: django.utils.translation.ugettext_lazy() is deprecated in favor of django.utils.translation.gettext_lazy().

I run my library tests with python -W error::DeprecationWarning, to avoid passing warning errors on to users, so this broke my test run.

Database Features not set for Postgres for management commands

Hi,

Yes, I have added django-jsonfield-backport to INSTALLED_APPS.

Might be that the cause of the below is somewhere else / somewhere upstream in django, but I am seeing the following.
I have an application that works find with a json field on a model. I can delete that model just fine from the UI that have build, running with runserver or uwsgi etc. It also runs fine if I try to delete model instances from a management command.

When I run the same application with postgres and runserver or uwsgi, it also works fine. I can delete model instances.
But when I run a management command to delete model instances, I get an exception:

  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/home/valentijn/dd/dojo/management/commands/test_valentijn5.py", line 10, in handle
    engagement.delete()
  File "/home/valentijn/dd/dojo/models.py", line 1042, in delete
    super().delete(*args, **kwargs)
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/base.py", line 921, in delete
    collector.collect([self], keep_parents=keep_parents)
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/deletion.py", line 224, in collect
    field.remote_field.on_delete(self, field, sub_objs, self.using)
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/deletion.py", line 15, in CASCADE
    collector.collect(sub_objs, source=field.remote_field.model,
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/deletion.py", line 223, in collect
    elif sub_objs:
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/query.py", line 278, in __bool__
    self._fetch_all()
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/query.py", line 72, in __iter__
    for row in compiler.results_iter(results):
  File "/home/valentijn/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1086, in apply_converters
    value = converter(value, expression, connection)
  File "/home/valentijn/venv/lib/python3.8/site-packages/django_jsonfield_backport/models.py", line 111, in from_db_value
    if connection.features.has_native_json_field and self.decoder is None:
AttributeError: 'DatabaseFeatures' object has no attribute 'has_native_json_field'

When debugging, it looks like the extend_features is not called in this scenario. Looking at the code of this backport it connects to the connection_created signal. For some reason this doesn't seem to be called when running as a management command against a postgres database. For MySQL it runs fine. As said, might be something upstream, but at least I can share my workaround here. I added the below to my management command to make it work:

from django_jsonfield_backport.features import extend_features
from django.db import connection

class Command(BaseCommand):

    def handle(self, *args, **options):
        extend_features(connection)
     
        .... do_stuff .... 

Backport fix for __contains lookup on key transforms

TR-31829

Backport fix for Unicode characters and read-only admin

TR-31157 (django/django#12308)

TR-32080 (django/django#13512)

Add support for simplejson

We are using Decimal fields in json which is supported in simplejson but not in standard json.

When using django-jsonfield-backport we run into an issue as this uses the standard json library:

  File "python3.8/site-packages/django_jsonfield_backport/models.py", line 149, in get_prep_value
    return json.dumps(value, cls=self.encoder)
  File "python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Decimal is not JSON serializable

Would it be possible to add support for both json and simplejson? Maybe add something like this:

try:
    import simplejson as json
except ImportError:
    import json

Fix postgres tests

Latest psycopg2 breaks on Django 3 (django/django#14547). Let's pin it to latest 2.8.

Connection features with @cached_property are not implemented properly

@cached_property is only evaluated if it's attached to an instance. The current feature checks are implemented by mapping connection.vendor to the [vendor]Features class es. However, since we don't create an instance of the [vendor]Features classes, calling features[connection.vendor].feature_name returns a cached_property object, not the evaluated value.

A better implementation would be to use @receiver(connection_created) and use setattr() to add JSON features to connection.features, so that feature checks would be similar to the built-in JSONField.

It's probably a good idea to also add another test matrix with a database that does not support JSONField.

DatabaseFeatures error with SQLite

I'm trying to use the django-jsonfield-backport package with Sqlite3 version 3.4.0 and Django version 2.2.17. I'm running macOS Catalina which from my understanding comes preinstalled with the SQLite JSON1 extension.

I am able to write json data to the db without any issues, however I am encountering an error when querying the db:

  File "/Users/nick/Development/workspace/django-jsonfield-backport/src/django_jsonfield_backport/models.py", line 112, in from_db_value
    if connection.features.has_native_json_field and self.decoder is None:
AttributeError: 'DatabaseFeatures' object has no attribute 'has_native_json_field'

I cloned the latest version of this project in development mode. Upon debugging it appears that the DatabaseFeatures object only contains a reference to the connection. On line 111 in the from_db_value method of models.py I print the connection.features object. As you can see, it doesn't contain all of the attributes that are expected.

>>>print(connection.features.__dict__)
{'connection': <django.db.backends.sqlite3.base.DatabaseWrapper object at 0x10b4cc128>}

I believe the problem is that Django is using the default DatabaseFeatures backend for SQLite. Printing the features object in the from_db_value method returns the following reference:

>>>print(connection.features)
<django.db.backends.sqlite3.features.DatabaseFeatures>

Any idea how to fix this?

AttributeError: 'DatabaseFeatures' object has no attribute 'has_native_json_field'

I'm currently using Django 2.2.15 and was excited to start using this package while waiting for the next Django LTS release to happen. However, trying to use it, I'm running into this issue:

Traceback (most recent call last):
...
  File ".../lib/python3.6/site-packages/django_jsonfield_backport/models.py", line 111, in from_db_value
    if connection.features.has_native_json_field and self.decoder is None:
AttributeError: 'DatabaseFeatures' object has no attribute 'has_native_json_field'

This attribute was added when JSONField landed in Django for 3.1 (django/django@6789ded), so it's not available to me back in Django 2.2.

This library does not support microsoft as a vendor, can that be made possible

Support Python 3.5

Django 2.2 still supports it.

JSONField.bound_data raises TypeError when data is None

Hi,

When a bound form with no value provided for the JSONField is rendered, it will currently crash.

class JSONForm(Form):
    json_field = JSONField(required=False)
form = JSONForm({})
assert form.as_p()

raises

TypeError: the JSON object must be str, bytes or bytearray, not NoneType

A fix has been created already by @AlexHill here: django/django#13844

support rest_framework serializer?

Backport fix for __in lookup on SQLite, MySQL, MariaDB, and Oracle

TR-31936

django/django#13346