iamteem/redisco

A Python Library for Simple Models and Containers Persisted in Redis

redisco's Issues

Some architecture questions

I was looking at redisco models code, to adapt my patch to Models. I become many questions about its architecture.

What is doing Manager class ? It is just a alias to ModelSet ?

What is doing key.py, validation.py, utils.py ?

For what is need ModelBase ? Why ModelBase init was splited to many functions ? Why not methods ?

Uhm. Thanks :)

Saving twice causes model to become non-unique.

Given a redisco model:

class User(models.Model):
    name = models.Attribute(required=True, unique=True)
    password = models.Attribute()

the following sequence fails:

u = User(name='foo')
u.password = 'secret'
u.save()
u.save()

Now, maybe this is reasonable, but the big problem is the return value from the second save call is:

[('name', 'not unique')]

which is extremely misleading.

Missing "zrevrangebyscore" function on SortedSet

The SortedSet container does not expose the zrevrangebyscore function (listed here http://redis.io/commands/zrevrangebyscore)

Is a ListField of ReferenceFields possible?

Scenario: I have a "Game" class that I would like to be able to refer to one or more "Player" objects. If I'm okay with storing the player data in each game, I can do:

class Game(models.Model):
    players = models.ListField(Player)

But I'd like it to just be a reference, not contain copies of each player. For one player, this would work:

class Game(models.Model):
    players = models.ReferenceField("Player", related_name="game")

What I really want (or what I think I want) is a ListField of ReferenceFields. Of course, if I try this:

class Game(models.Model):
    players = models.ListField(models.ReferenceField("Player", related_name="game"))

It's not going to work, since it's going to instantiate the ReferenceField instead of it being a class. Is there any way to accomplish what I want?

Dumping pickled data

class Request(models.Model):
    data = models.Attribute()

req = Request()
req.data = cPickle.dumps(data, -1)
req.save()

Not sure why unicode conversion is done while saving and retrieving.
It work perfectly for me if I by patch the code where patch catches the exception and returns the value without unicode cast.

Add a `unique_together` option

It would be great if redisco could have a unique_together option (in the Meta class)

Sometimes a field only is not enough to manage unicity.

Any hint on how to add this ?

PS : https://docs.djangoproject.com/en/dev/ref/models/options/#unique-together

get_by_id should be aware of previous filters

Considering this case :

class A(models.Model):
    user = models.ReferenceField(B)

class B(models.Model):
    att = models.Attribute()


b1 = B(att='test)
b1.save()

b2 = B(att='test')
b2.save()

a = A(user=b1)
a.save()

self.assertEqual(a.user_set.get_by_id(b1.id), b1)
self.assertEqual(a.user_set.get_by_id(b2.id), None)

The second test will fail as get_by_id doesn't take into account any filter/zfilter/excludes.

Self-reference (recursive) ManyToOne/OneToMany relationships available?

This is what I meant by self-reference.

class Employee(models.model):
    manager = models.ForeignKey('Employee')

This is how Django implements recursive relationships.

To create a recursive relationship – an object that has a many-to-one relationship with itself – use models.ForeignKey('self', on_delete=models.CASCADE).

Is this available to use?

any example is appreciated.

filter and chained zfilter prevent filter to be applied

This code:

obj.filter(obj_ref_id=1).zfilter(count__in=(1, 20)) is not executed properly. Only the zfilter is apply, leaving the filter on the object reference out.

The code documentation states that only one zfilter will be applied due to performances issue but the previous filter is not applied and should.

Redisco should be TimeZone Aware

Redisco should support Timezones or at least, store the DateTimeFields in UTC instead of trusting the naive objects.

ModelSet.filter / zfilter methods should cleanup their mess

They should either delete the stored sets/sorted set immediately after getting the result or expire the keys to the set.

DateTimeField save wrong value

At: https://github.com/kiddouk/redisco/blob/master/redisco/models/attributes.py#L216
I believe the correct microsecond part should be: value.microsecond / 1000000.0

Thank!

DateTimeField getting corrupted on save if microsecond part has less then 6 positions

Script below raises AssertionError while it should run successfuly.

from redisco import models
import datetime as dt
import redisco

redisco.connection.flushdb()

class A(models.Model):
    when = models.DateTimeField()

when = dt.datetime(2011, 1, 1, 0, 0, 0, 1)
A.objects.create(when=when)
obj = A.objects.get_by_id(1)

assert obj.when.microsecond == when.microsecond

Deferred model references broken

When you create a model reference with a string as the target type, redisco fails to create an <origin>_set attribute on the target.
>>> class User(models.Model):
... name = models.CharField()
... address = models.ReferenceField('Address')
...
>>> class Address(models.Model):
... street_address = models.CharField()
...
>>> a=Address()
>>> a.user_set
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'Address' object has no attribute 'user_set'
>>> a.address_set # Exists and raises redisco.models.exceptions.MissingID

Modified test case and fix are here

redisco.models.attributes.ReferenceField.__get__ does not read from instance

given app/models.py as:

from redisco import models

class Department(models.Model):
    name = models.Attribute(required=True)

class Person(models.Model):
    name = models.Attribute(required=True)
    #manager = models.ReferenceField('Person', related_name='underlings')
    department = models.ReferenceField(Department)

You can see that ReferenceField is correctly handling the _manager_id in set, but the get is caching the referenced model on the ReferenceField instance instead of the Model instance:

>>> from app import models
>>> d1 = models.Department(name='Accounting')
>>> d1.save()
True
>>> d2 = models.Department(name='Billing')
>>> d2.save()
True
>>> p1 = models.Person(name='Joe', department=d1)
>>> p1.save()
True
>>> p1
<Person:1 {'department': <Department:1 {'name': 'Accounting'}>, 'name': 'Joe', 'department_id': '1'}>
>>> p2 = models.Person(name='Jack', department=d2)
>>> p2.save()
True
>>> p2
<Person:2 {'department': <Department:1 {'name': 'Accounting'}>, 'name': 'Jack', 'department_id': '2'}>

N.B.

>>> p2.department_id
'2'
>>> p2.department.id
'1'

Operations like ZADD and PUSH should return redis returned value

It is usually useful to check the value returned by redis when doing a zadd. It helps knowing if the elements has been added or just updated. Redisco should reflect that.

Pipelining or Bulk_Create?

I had a look around and didn't seem to find pipelining/"bulk_create" options.

This is the only thing stopping me from using redisco, and even testing it in production.

ImportError

I am getting the following ImportError. I am using Python 3.3.3 - before with Python 2.7.6 everything worked.

Traceback (most recent call last):
  File "./run.py", line 3, in <module>
    from app import app
  File "/Users/govindaf/workspace/commander/commander/app/__init__.py", line 8, in <module>
    from app import views
  File "/Users/govindaf/workspace/commander/commander/app/views.py", line 7, in <module>
    from redisco import models, connection_setup
  File "/Users/govindaf/workspace/commander/commander/venv/lib/python3.3/site-packages/redisco/models/__init__.py", line 1, in <module>
    from base import *
ImportError: No module named 'base'

What is wrong here?

Default port ?

If I am starting redis-server at 6379 port, I am getting 74 unit-tests connection errors.

If I am starting redis-server at 6380 port, I am getting error in tests.connection unit-test and error if I use models.

?!

operations on ListField are not atomic

This may actually be a feature request. But, I had inferred from the documentation of Ohm that append and pop operations on a ListField would be immediately persisted and atomic (see "Persistence strategy" in http://github.com/soveran/ohm/blob/master/README.markdown) - but this doesn't seem to be the case...

with Book and Author models from tests.models.ListFieldTestCase.test_list_of_reference_fields

>>> b1 = Book.objects.create(title='Book1', date_published=date.today())
>>> b2 = Book.objects.create(title='Book2', date_published=date.today())
>>> a1 = Author.objects.create(name='Author1', books=[b1, b2])
>>> # then in another thread
... 
>>> a2 = Author.objects.get_by_id(1)
>>> b3 = Book.objects.create(title='Book3', date_published=date.today())
>>> a2.books.append(b3)
>>> a2.save()
True
>>> len(a2.books)
3
>>> # meanwhile back in the first request
... 
>>> a1.save()
True
>>> len(Author.objects.get_by_id(1).books)
2

Atomic operations on redis lists (and sets?) are one of the killer features, and I'd like redisco to abstract them for me - e.g. Counter! The "Last Write Wins" strategy makes a lot of sense in some places, but an atomic list field would probably be preferable in many cases.

I could see reason to disabled set on ListField entirely in favor of append and pop (lpush, rpop?) on the mode base a la Counters' incr and decr? But for backwards compat maybe it would better to create a new AtomicListField (don't love the name...)

More interesting ideas might include allowing get to return the counters.List instance directly instead of cast members, or at least some sort of proxy/wrapper around it - or for _redisco_model, maybe something similar to a ModelSet?

I wonder if you've considered this case, or what approach you would find more amiable?

Using Mixins to add common fields to a model doesn't work

I tried using a mixin with some common fields, but it doesn't work:

class CeleryTaskMixin:
    celery_status = models.CharField()  # last known status, call update_status
    celery_id = models.CharField()      # celery uuid
    f_name = models.CharField(indexed=True)


class RunningTask(CeleryTaskMixin, models.Model):
    def __init__(self, t=None, *args, **kwargs):
        CeleryTaskMixin.__init__(self)
        models.Model.__init__(self)

    task_type = models.CharField()
    start_time = models.DateTimeField(auto_now=True)
    end_time = models.DateTimeField()

Then when I try and do

models.RunningTask.objects.filter(f_name="blah")

It says

  File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 52, in __iter__
    for id in self._set:
  File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 294, in _set
    s = self._add_set_filter(s)
  File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 317, in _add_set_filter
    (k, self.model_class.__name__))
redisco.models.exceptions.AttributeNotIndexed: Attribute f_name is not indexed in RunningTask class.

This seems like something that should work... (?)

ReferenceField not indexable

The ReferenceField should be indexable, thus allowing to filter by such a field.

Recursion issue with ListField of ReferenceField

I'm trying to use ListField of ReferenceField
to maintain a list of user channels of my User model
as well as a list of users in my Channel model.

Model:

# Module:   models
# Date:     16th August 2014
# Author:   James Mills, prologic at shortcircuit dot net dot au


"""Data Models"""


from circuits.protocols.irc import joinprefix

from redisco.models import Model
from redisco.models import (
    Attribute, BooleanField, DateTimeField,
    IntegerField, ListField, ReferenceField
)


class User(Model):

    host = Attribute(default="")
    port = IntegerField(default=0)

    nick = Attribute(default=None)
    away = BooleanField(default=False)

    channels = ListField("Channel")
    userinfo = ReferenceField("UserInfo")

    registered = BooleanField(default=False)
    signon = DateTimeField(auto_now_add=True)

    @property
    def prefix(self):
        userinfo = self.userinfo
        return joinprefix(self.nick, userinfo.user, userinfo.host)

    class Meta:
        indices = ("id", "nick",)


class UserInfo(Model):

    user = Attribute(default=None)
    host = Attribute(default=None)
    server = Attribute(default=None)
    name = Attribute(default=None)

    def __nonzero__(self):
        return all(x is not None for x in (self.user, self.host, self.name))


class Channel(Model):

    name = Attribute(required=True, unique=True)
    users = ListField("User")

    class Meta:
        indices = ("id", "name",)

This seems to work fine when there is one User and one Channel
object(s) in the database, but as soon as there are two different users
that are members of the same channel I get this error:

2014-08-16 22:02:51,218 - charla.main - ERROR - ERROR <handler[*.join] (Commands.join)> (<join[commands] (<socket._socketobject object at 0x7f956fb54c20>, ('', None, None), '#circuits' )>) (<type 'exceptions.RuntimeError'>): RuntimeError('maximum recursion depth exceeded while calling a Python object',)
  File "/home/prologic/work/circuits/circuits/core/manager.py", line 603, in _dispatcher
    value = handler(*eargs, **ekwargs)
  File "/home/prologic/charla/charla/plugins/core.py", line 111, in join
    user.save()
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 202, in save
    self._write(_new)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 345, in _write
    self._update_indices(pipeline)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 413, in _update_indices
    self._add_to_indices(pipeline)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 418, in _add_to_indices
    self._add_to_index(att, pipeline=pipeline)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 426, in _add_to_index
    index = self._index_key_for(att)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 474, in _index_key_for
    return self._tuple_for_index_key_attr_list(att, value)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 492, in _tuple_for_index_key_attr_list
    return ('list', [self._index_key_for_attr_val(att, e) for e in val])
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 499, in _index_key_for_attr_val
    return self._key[att][_encode_key(val)]
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/utils.py", line 5, in _encode_key
    return base64.b64encode(str(s)).replace("\n", "")
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 516, in __repr__
    return "<%s %s>" % (self.key(), self.attributes_dict)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 516, in __repr__
    return "<%s %s>" % (self.key(), self.attributes_dict)

... repeated lines delete

  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 516, in __repr__
    return "<%s %s>" % (self.key(), self.attributes_dict)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 247, in attributes_dict
    h[k] = getattr(self, k)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/attributes.py", line 273, in __get__
    val = List(key).members
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 34, in __getattribute__
    return object.__getattribute__(self, att)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 217, in all
    return self.lrange(0, -1)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 32, in __getattribute__
    return partial(getattr(object.__getattribute__(self, 'db'), att), self.key)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 39, in db
    if self.pipeline:
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 31, in __getattribute__
    if att in object.__getattribute__(self, 'DELEGATEABLE_METHODS'):

I believe Issue #26 is somewhat related?

Storing utc datetime values for auto_now_add

I went through your code, and found that when auto_now_add is set to True you store datetime.now() .
This shall fail in timezone oriented app. Though mostly applications are hosted on cloud servers and they are configured for utc timezone. This library won't fail when server is in utc timezone. This library will definitely fail in development environment if development machine is not set to use utc timezone.

Though your code can store timezone aware datetimes, but auto_now_add will be useless in timezone oriented app.

Hope you would fix this.

Thanks for brilliant hard work.

Indexing for DateTimeField and auto_now_add

Hi Tim,

DateTimeField is ignored/filtered while initializing indices.

I have written a small patch and would send a pull request in few moments. Although it does not solve problem for datetime field with auto_now_add as at points redisco checks if value is passed and the only creates the index.

Cheers,
Shekhar

Attribute can't handle Unicode

If you store unicode in an Attribute, you won't get back unicode, but an utf-8 encoded str. That's because redis-py (at least by default) encodes all unicode values to utf-8, before executing a command, but cannot decode the read values. Redisco should therefore not pass unicode objects to redis-py but do utf-8 conversion itself.

AttributeError: has no attribute '_errors'

The errors property on models.base should check for _errors first, and if not exists call .is_valid for you

I can:
x = MyModel.objects.create()
x.errors # == []

but not:
x = MyModel()
x.errors # raises AttributeError!

Float precision issue in ranged queries

The current implementation of ranged queries does not work correctly in some cases due to the way redis-py handles floats.

The implementations of lt, gt, etc. in SortedSet, all call redis-py's zrangebyscore with a float as one of the arguments (_min_score or _max_score). redis-py calls convert on these to convert them to strings, using str(), which results in a loss of float precision. This means, that ranged queries like FooModel.objects.zfilter(some_attr__lt=10.009) return wrong results. I'd suggest converting float arguments to a string using "%f" % value before passing them to a redis-py query.