svetlyak40wt/django-fields

Fields pack for django framework.

Introduction

Django-fields is an application which includes different kinds of models fields.

Right now, application contains two fields with encryption support: EncryptedCharField and EncryptedTextField.

This project uses Travis for continuous integration:

Requirements

This application depends on python-crypto, which can be found in many Linux repositories, or downloaded from http://www.dlitz.net/software/pycrypto/.

Under Ubuntu, just do:

sudo apt-get install python-crypto

How to run tests

Examples can be found at the examples directory. Look at the, tests.py. Same project is used to run unittests. To run them, just fire ./run-tests.sh.

Contributors

• zbyte64 — thanks to for his django snippet for encrypted fields. After some fixes, this snippet works as supposed.
• John Morrissey — for fixing bug in PickleField.
• Joe Jasinski — different fixes and new fields for encripted email and US Phone.
• Colin MacDonald — for many encripted fields added.
• Igor Davydenko — PickleField.
• kromem - Added support for specifying block_type on encrypted fields.
• Brooks Travis - new field for encrypted US Social Security Number and other fixes.

django-fields's Issues

EncryptedDateField Still Broken

Similar to (and maybe a result of whatever fix was implemented for) issue 4.

django 1.8
python 2.7

My relevant form code:

class PatientInformationForm(forms.ModelForm):
    date_of_birth = forms.DateField(
         widget=TriSelectDateWidget(required=False, years=input_choices.DOB_YEAR_RANGE))

My relevant model:

class PatientInformation(models.Model):
     date_of_birth = enc_fields.EncryptedDateTimeField()

Upon form submission django_fields produces the error:
AttributeError at /registration/patient/ 'datetime.date' object has no attribute 'split'

If I cast the date_of_birth to a string in my form's clean method I encounter the same error described in issue 4.

I have a temporary work around but I think it should ultimately be resolved by django_fields:

def clean_date_of_birth(self):
        # hackish workaround...
        data = self.cleaned_data['date_of_birth']
        data = str(data)
        data = data.replace('-', ':')
        return data

Update Path for MODE_CBC deprecation warning

Hi all. What's the proper update path to fix the following deprecation warning?

https://github.com/svetlyak40wt/django-fields/blob/master/src/django_fields/fields.py#L41

"Default usage of pycrypto's AES block type defaults has been "
"deprecated and will be removed in 0.3.0 (default will become "
"MODE_CBC). Please specify a secure block_type, such as CBC.",
DeprecationWarning,

I tried changing all of the encrypted fields in my app to add "block_type=BLOCK_TYPE" to the model and resyncing the database, but got the following error. I think that this is because the new block type makes encrypted values larger than the field can store.

  File "/home/jjasinski/Sites/osf/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 350, in get_db_prep_save
 prepared=False)
 File "/home/jjasinski/Sites/osf/local/lib/python2.7/site-packages/django_fields/fields.py", line 164, in get_db_prep_value
 str(len(value)) + " > " + str(self.unencrypted_length)
ValueError: Field value longer than max allowed: 37 > 12

EncryptedCharField isn't decrypted in .values()

Any access of an EncryptedCharField which isn't directly on the model instance is not decrypted. As an example, both QuerySet.values() and QuerySet.values_list() show an encrypted password.

Wrong max_length value when using south

I'm not sure why max_length is calculated this way:
kwargs['max_length'] = max_length * 2 + len(self.prefix)

When I use south I get field length max_length * 2 + len(self.prefix) in my migration file and (max_length * 2 + len(self.prefix))*2 + len(self.prefix) in database.

In my opinion max_length shouldn't be changed if it is in kwargs and another key value like 'unencrypted_max_length' should be used:

        max_length = kwargs.pop('unencrypted_max_length', 40)
        ...
        kwargs['max_length'] = kwargs.get('max_length', max_length * 2 + len(self.prefix))

Encryption password stored in a cleartext

Is there a risk of stealing the password along with the database which defeats the whole idea of encrypted fields?
Should be using python keyring?

Fix ReadMe so that people know what version of Django it works with

Fix ReadMe so that people know what version of Django it works with. In older versions such as (1, 0, 2, 'final', 0) django.forms does not have an attribute 'DateInput'. I looked at (1, 2, 3, 'final', 0) and this does have it and does work.

For consideration: Fields automatically decrypted in admin optional.

For our web app we would like to encrypt the email address and password for our users, but:

• We want to see their email in the admin panel.
• We really don't want to see their password.

It'd be nice to flag a model as admin mode "don't decrypt". I'd be willing to put into some coding to make this a reality but I'd have to be pointed in the right direction.

Use PKCS7 padding.

From what I can see, this project pads with random characters, which is... interesting. Padding should be something like PKCS7 instead.

django 1.10

Django 1.10 is not supported. Is there plan for this?

EncryptedDateField and EncryptedDateTimeField Broken

When using an EncryptedDateField, I get the following error:
invalid literal for int() with base 10:''
I think this is because when the ModelForm first renders an empty form, the to_python function is passed an empty value and therefore breaks when trying to do the split:
date_value = self.date_class(*map(int, date_text.split(':')))
(I think) I was able to solve this by adding a test to check if the value is empty in the to_python function. Also, once I added this check, I was able to enter values that match only the string format YYYY:MM:DD. To solve this, I added a form_class to the formfield defaults.

Not sure how to submit patches or changes on Github (still new to it), but here's my changes so far. I can submit in a better format if needed. Let me know what you think....
Joe

(d11)jjasinski:~/Sites/d11/src/django-fields/src/django_fields$ diff fields.py fields.py.orig 
8d7
< from django.forms import fields
122c121
<         defaults = {'widget': self.form_widget,'form_class':self.form_field}

---
>         defaults = {'widget': self.form_widget}
128,129c127
<    
<         if value in fields.EMPTY_VALUES:

---
>         if isinstance(value, self.date_class):
132,136c130,131
<             if isinstance(value, self.date_class):
<                 date_value = value
<             else:
<                 date_text = super(BaseEncryptedDateField, self).to_python(value)
<                 date_value = self.date_class(*map(int, date_text.split(':')))

---
>             date_text = super(BaseEncryptedDateField, self).to_python(value)
>             date_value = self.date_class(*map(int, date_text.split(':')))
150d144
<     form_field = forms.DateField
160d153
<     form_field = forms.DateTimeField

Insecure block cipher mode

The pycrypto library defaults to ECB mode for AES:

http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Electronic_codebook_.28ECB.29

I'm not sure why they do this, considering even their own documentation says not to use it:

https://www.dlitz.net/software/pycrypto/api/current/Crypto.Cipher.blockalgo-module.html#MODE_ECB

But, as you can see, the default call to AES.new() uses this cipher mode:

https://www.dlitz.net/software/pycrypto/api/current/Crypto.Cipher.AES-module.html#new

The major difference that would need to be made to django-fields to support a stronger and secure block cipher mode (i.e. CBC) would be the addition of a random IV that is appended to the ciphertext. (An example of proper use is documented in the pycrypto AES module page linked above.

However, this change isn't backwards compatible for data previously encrypted.

I'll take a stab at upgrading the library to keep backwards compatible defaults, but enabling a stronger block cipher mode via kwords arguments (like how cipher type is decided). Still, any data using the insecure defaults will have to be dumped prior to upgrading to CBC, and then imported again after the change.

EncryptedCharField not encrypting value

Here's my model:

class Report(models.Model):
    loan_number = EncryptedCharField()

When I save any value in loan_number, for example '12345', the form will save, but the value is not stored encrypted in the database. If I look at the report in Django admin, the value for loan_number will be '12345'.

Am I missing something?

Thanks,
Brandon

Heroku and block_type='MODE_CBC' causes errors

As per the recommendation in the source code we switched to block_type='MODE_CBC' for our encrypted fields, only to get the following error:

AssertionError
PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()

Which is thrown by the Crypto library. After some googling around it seems this problem occurs when that library is used in combination with threading. So for example Celery and Heroku.

It is quite easy to fix, just call the Random.atfork() function before using Random in BaseEncryptedField (fields.py line 55 & 111).

EncryptedCharField raises a traceback in the django admin

Traceback (most recent call last):

 File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 92, in get_response
   response = callback(request, *callback_args, **callback_kwargs)

 File "/usr/lib/python2.5/site-packages/django/contrib/admin/sites.py", line 490, in root
   return self.model_page(request, *url.split('/', 2))

 File "/usr/lib/python2.5/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
   response = view_func(request, *args, **kwargs)

 File "/usr/lib/python2.5/site-packages/django/contrib/admin/sites.py", line 509, in model_page
   return admin_obj(request, rest_of_url)

 File "/usr/lib/python2.5/site-packages/django/contrib/admin/options.py", line 1098, in __call__
   return self.change_view(request, unquote(url))

 File "/usr/lib/python2.5/site-packages/django/db/transaction.py", line 240, in _commit_on_success
   res = func(*args, **kw)

 File "/usr/lib/python2.5/site-packages/django/contrib/admin/options.py", line 873, in change_view
   return self.render_change_form(request, context, change=True, obj=obj)

 File "/usr/lib/python2.5/site-packages/django/contrib/admin/options.py", line 590, in render_change_form
   ], context, context_instance=context_instance)

 File "/usr/lib/python2.5/site-packages/django/shortcuts/__init__.py", line 20, in render_to_response
   return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)

 File "/usr/lib/python2.5/site-packages/django/template/loader.py", line 108, in render_to_string
   return t.render(context_instance)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 178, in render
   return self.nodelist.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 779, in render
   bits.append(self.render_node(node, context))

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 792, in render_node
   return node.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/loader_tags.py", line 97, in render
   return compiled_parent.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 178, in render
   return self.nodelist.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 779, in render
   bits.append(self.render_node(node, context))

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 792, in render_node
   return node.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/loader_tags.py", line 97, in render
   return compiled_parent.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 178, in render
   return self.nodelist.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 779, in render
   bits.append(self.render_node(node, context))

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 792, in render_node
   return node.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/loader_tags.py", line 24, in render
   result = self.nodelist.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 779, in render
   bits.append(self.render_node(node, context))

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 792, in render_node
   return node.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/defaulttags.py", line 155, in render
   nodelist.append(node.render(context))

 File "/usr/lib/python2.5/site-packages/django/template/loader_tags.py", line 111, in render
   return self.template.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 178, in render
   return self.nodelist.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 779, in render
   bits.append(self.render_node(node, context))

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 792, in render_node
   return node.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/defaulttags.py", line 155, in render
   nodelist.append(node.render(context))

 File "/usr/lib/python2.5/site-packages/django/template/defaulttags.py", line 155, in render
   nodelist.append(node.render(context))

 File "/usr/lib/python2.5/site-packages/django/template/defaulttags.py", line 244, in render
   return self.nodelist_false.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 779, in render
   bits.append(self.render_node(node, context))

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 792, in render_node
   return node.render(context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 831, in render
   return _render_value_in_context(output, context)

 File "/usr/lib/python2.5/site-packages/django/template/__init__.py", line 811, in _render_value_in_context
   value = force_unicode(value)

 File "/usr/lib/python2.5/site-packages/django/utils/encoding.py", line 92, in force_unicode
   raise DjangoUnicodeDecodeError(s, *e.args)

DjangoUnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid data. You passed in <django.forms.forms.BoundField object at 0xb9e308c> (<class 'django.forms.forms.BoundField'>)

global name 'EncryptedPhoneNumberField' is not defined

https://github.com/svetlyak40wt/django-fields/blob/master/src/django_fields/fields.py#L272

django-fields is not compatible with the django Sites framework (multiple settings.py files).

I use the django sites framework to run several sites from the same application. Since each setting file has its own SECRET_KEY, the sites cannot share encrypted data. django-fields should accept a parameter or look for a var in settings.py to override this behavior with a custom key for the cipher. If it doesn't exist, then fall back on settings.SECRET_KEY.

I think this may be related to issue #1.

Ticket #1 should be re-opened.

The issues in ticket #1 persist. It should be reopened. Additional comments have been added to ticket #1.

Deprecation warning with django 1.3b4

Not sure when this started but the following patch fixes the deprecation warnings.

Initial Update

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

Filtering not supported?

I was getting some strange behavior on my site and traced the problem back to this:

In [1]: from customers.models import *
In [2]: foo = KYCUser.objects.filter(ss_num__isnull=False).last()
In [3]: print foo.ss_num
Out[3]: u'123456789'
In [4]: KYCUser.objects.filter(ss_num='123456789')
Out[4]: []

ss_num looks like this in the model defintion:

ss_num = EncryptedCharField(max_length=10, blank=True, null=True, db_index=True)   

Does this library not support filtering? That'd be understandable, but if so it should make that clear in the docs and not fail silently.

Thanks for putting this out there!

EncryptedCharField doesn't validate length

Instead of raising a validationerror at the appropriate time, a ValueError is raised at save time:

Traceback:
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/core/handlers/base.py" in get_response
  92.                 response = callback(request, *callback_args, **callback_kwargs)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/contrib/admin/options.py" in wrapper
  226.                 return self.admin_site.admin_view(view)(*args, **kwargs)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/contrib/admin/sites.py" in inner
  186.             return view(request, *args, **kwargs)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/db/transaction.py" in _commit_on_success
  240.                 res = func(*args, **kw)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/contrib/admin/options.py" in change_view
  833.                     self.save_formset(request, form, formset, change=True)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/contrib/admin/options.py" in save_formset
  563.         formset.save()
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/forms/models.py" in save
  522.         return self.save_existing_objects(commit) + self.save_new_objects(commit)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/forms/models.py" in save_existing_objects
  640.                 saved_instances.append(self.save_existing(form, obj, commit=commit))
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/forms/models.py" in save_existing
  510.         return form.save(commit=commit)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/forms/models.py" in save
  407.                              fail_message, commit, exclude=self._meta.exclude)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/forms/models.py" in save_instance
  78.         instance.save()
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/db/models/base.py" in save
  410.         self.save_base(force_insert=force_insert, force_update=force_update)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/db/models/base.py" in save_base
  474.                         rows = manager.filter(pk=pk_val)._update(values)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/db/models/query.py" in _update
  442.         query.add_update_fields(values)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/db/models/sql/subqueries.py" in add_update_fields
  245.                 val = field.get_db_prep_save(val)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django/db/models/fields/__init__.py" in get_db_prep_save
  192.         return self.get_db_prep_value(value)
File "/iscape/sites/ahcdeploy/lib/python2.5/site-packages/django_fields/fields.py" in get_db_prep_value
  106.                     str(len(value)) + " > " + str(self.unencrypted_length))

Exception Type: ValueError at /admin/cne/registration/1/
Exception Value: Field value longer than max allowed: 89 > 64

class EncryptedCharField(BaseEncryptedField):
    __metaclass__ = models.SubfieldBase

    def get_internal_type(self):
        return "CharField"

    def formfield(self, **kwargs):
        defaults = {'max_length': self.max_length}
        defaults.update(kwargs)
        return super(EncryptedCharField, self).formfield(**defaults)

    def get_db_prep_value(self, value, connection=None, prepared=False):
        if value is not None and not self._is_encrypted(value):
            if len(value) > self.unencrypted_length:
                raise ValueError("Field value longer than max allowed: " +
                    str(len(value)) + " > " + str(self.unencrypted_length))
        return super(EncryptedCharField, self).get_db_prep_value(
            value,
            connection=connection,
            prepared=prepared,
        )

https://github.com/svetlyak40wt/django-fields/blob/master/src/django_fields/fields.py#L106

EncryptedDateTimeField Causes "can't compare offset-naive and offset-aware datetimes" Error

Using the EncryptedDateTimeField class with Django version >= 1.4 causes a TypeError whenever you try to update the value of the field in the the Django Admin. Django Admin expects timezone-aware datetime objects. Until this is fixed, this field isn't usable in anything newer than 1.3.

Any plans for python3 or Django1.8 support?

Thanks!

Encrypted fields cannot be searched in the admin

Opening this pre-emptively. I'm not sure why but my immediate assumption is because it uses an SQL query to do the search which compares a plaintext vs an encrypted value. Not sure how one would get around this...