iamteem/redisco

A Python Library for Simple Models and Containers Persisted in Redis

This fork is no longer actively maintained. Go to https://github.com/kiddouk/redisco instead.

Redisco

Python Containers and Simple Models for Redis

Description

Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm and its design and code are loosely based on Ohm and the Django ORM. It is built on top of redis-py. It includes container classes that allow easier access to Redis sets, lists, and sorted sets.

Installation

Redisco requires redis-py 2.0.0 so get it first.

pip install redis

Then install redisco.

pip install redisco

Models

from redisco import models
class Person(models.Model):
    name = models.Attribute(required=True)
    created_at = models.DateTimeField(auto_now_add=True)
    fave_colors = models.ListField(str)

>>> person = Person(name="Conchita")
>>> person.is_valid()
True
>>> person.save()
True
>>> conchita = Person.objects.filter(name='Conchita')[0]
>>> conchita.name
'Conchita'
>>> conchita.created_at
datetime.datetime(2010, 5, 24, 16, 0, 31, 954704)

Model Attributes

Attribute

Stores unicode strings. If used for large bodies of text, turn indexing of this field off by setting indexed=True.

IntegerField

Stores an int. Ints are stringified using unicode() before saving to Redis.

Counter

An IntegerField that can only be accessed via Model.incr and Model.decr.

DateTimeField

Can store a DateTime object. Saved in the Redis store as a float.

DateField

Can store a Date object. Saved in Redis as a float.

FloatField

Can store floats.

BooleanField

Can store bools. Saved in Redis as 1's and 0's.

ReferenceField

Can reference other redisco model.

ListField

Can store a list of unicode, int, float, as well as other redisco models.

Attribute Options

required

If True, the attirbute cannot be None or empty. Strings are stripped to check if they are empty. Default is False.

default

Sets the default value of the attribute. Default is None.

indexed

If True, redisco will create index entries for the attribute. Indexes are used in filtering and ordering results of queries. For large bodies of strings, this should be set to False. Default is True.

validator

Set this to a callable that accepts two arguments -- the field name and the value of the attribute. The callable should return a list of tuples with the first item is the field name, and the second item is the error.

unique

The field must be unique. Default is False.

DateField and DateTimeField Options

auto_now_add

Automatically set the datetime/date field to now/today when the object is first created. Default is False.

auto_now

Automatically set the datetime/date field to now/today everytime the object is saved. Default is False.

Saving and Validating

To save an object, call its save method. This returns True on success (i.e. when the object is valid) and False otherwise.

Calling Model.is_valid will validate the attributes and lists. Model.is_valid is called when the instance is being saved. When there are invalid fields, Model.errors will hold the list of tuples containing the invalid fields and the reason for its invalidity. E.g. [('name', 'required'),('name', 'too short')]

Fields can be validated using the validator argument of the attribute. Just pass a callable that accepts two arguments -- the field name and the value of the attribute. The callable should return a list of errors.

Model.validate will also be called before saving the instance. Override it to validate instances not related to attributes.

def not_me(field_name, value):
    if value == 'Me':
        return ((field_name, 'it is me'),)

class Person(models.Model):
    name = models.Attribute(required=True, validator=not_me)
    age = models.IntegerField()

    def validate(self):
        if self.age and self.age < 21:
            self._errors.append(('age', 'below 21'))

>>> person = Person(name='Me')
>>> person.is_valid()
False
>>> person.errors
[('name', 'it is me')]

Queries

Queries are executed using a manager, accessed via the objects class attribute.

Person.objects.all()
Person.objects.filter(name='Conchita')
Person.objects.filter(name='Conchita').first()
Person.objects.all().order('name')
Person.objects.filter(fave_colors='Red')

Ranged Queries

Redisco has a limited support for queries involving ranges -- it can only filter fields that are numeric, i.e. DateField, DateTimeField, IntegerField, and FloatField. The zfilter method of the manager is used for these queries.

Person.objects.zfilter(created_at__lt=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__gte=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__in=(datetime(2010, 4, 20, 5, 2, 0), datetime(2010, 5, 1)))

Containers

Redisco has three containers that roughly match Redis's supported data structures: lists, sets, sorted set. Anything done to the container is persisted to Redis.

Sets

>>> from redisco.containers import Set >>> s = Set('myset') >>> s.add('apple') >>> s.add('orange') >>> s.members set(['orange', 'apple']) >>> t = Set('nset') >>> t.add('kiwi') >>> t.add('guava') >>> t.members set(['kiwi', 'guava']) >>> s.update(t) >>> s.members set(['kiwi', 'orange', 'guava', 'apple'])

Lists

>>> import redis >>> from redisco.containers import List >>> l = List('alpha') >>> l.append('a') >>> l.append('b') >>> l.append('c') >>> 'a' in l True >>> 'd' in l False >>> len(l) 3 >>> l.index('b') 1 >>> l.members ['a', 'b', 'c']

Sorted Sets

>>> zset = SortedSet('zset') >>> zset.members ['d', 'a', 'b', 'c'] >>> 'e' in zset False >>> 'a' in zset True >>> zset.rank('d') 0 >>> zset.rank('b') 2 >>> zset[1] 'a' >>> zset.add('f', 200) >>> zset.members ['d', 'a', 'b', 'c', 'f'] >>> zset.add('d', 99) >>> zset.members ['a', 'b', 'c', 'd', 'f']

Dicts/Hashes

>>> h = cont.Hash('hkey') >>> len(h) 0 >>> h['name'] = "Richard Cypher" >>> h['real_name'] = "Richard Rahl" >>> h <Hash 'hkey' {'name': 'Richard Cypher', 'real_name': 'Richard Rahl'}> >>> h.dict {'name': 'Richard Cypher', 'real_name': 'Richard Rahl'}

Additional Info on Containers

Some methods of the Redis client that require the key as the first argument can be accessed from the container itself.

>>> l = List('mylist') >>> l.lrange(0, -1) 0 >>> l.rpush('b') >>> l.rpush('c') >>> l.lpush('a') >>> l.lrange(0, -1) ['a', 'b', 'c'] >>> h = Hash('hkey') >>> h.hset('name', 'Richard Rahl') >>> h <Hash 'hkey' {'name': 'Richard Rahl'}>

Connecting to Redis

All models and containers use a global Redis client object to interact with the key-value storage. By default, it connects to localhost:6379, selecting db 0. If you wish to specify settings:

import redisco
redisco.connection_setup(host='localhost', port=6380, db=10)

The arguments to connect are simply passed to the redis.Redis init method.

For the containers, you can specify a second argument as the Redis client. That client object will be used instead of the default.

>>> import redis >>> r = redis.Redis(host='localhost', port=6381) >>> Set('someset', r)

Credits

Most of the concepts are taken from Soveran's Redis related Ruby libraries. cyx for sharing his expertise in indexing in Redis. Django, of course, for the popular model API.

redisco's Issues

Dumping pickled data

class Request(models.Model):
    data = models.Attribute()

req = Request()
req.data = cPickle.dumps(data, -1)
req.save()

Not sure why unicode conversion is done while saving and retrieving.
It work perfectly for me if I by patch the code where patch catches the exception and returns the value without unicode cast.

Default port ?

If I am starting redis-server at 6379 port, I am getting 74 unit-tests connection errors.

If I am starting redis-server at 6380 port, I am getting error in tests.connection unit-test and error if I use models.

?!

redisco.models.attributes.ReferenceField.__get__ does not read from instance

given app/models.py as:

from redisco import models

class Department(models.Model):
    name = models.Attribute(required=True)

class Person(models.Model):
    name = models.Attribute(required=True)
    #manager = models.ReferenceField('Person', related_name='underlings')
    department = models.ReferenceField(Department)

You can see that ReferenceField is correctly handling the _manager_id in set, but the get is caching the referenced model on the ReferenceField instance instead of the Model instance:

>>> from app import models
>>> d1 = models.Department(name='Accounting')
>>> d1.save()
True
>>> d2 = models.Department(name='Billing')
>>> d2.save()
True
>>> p1 = models.Person(name='Joe', department=d1)
>>> p1.save()
True
>>> p1
<Person:1 {'department': <Department:1 {'name': 'Accounting'}>, 'name': 'Joe', 'department_id': '1'}>
>>> p2 = models.Person(name='Jack', department=d2)
>>> p2.save()
True
>>> p2
<Person:2 {'department': <Department:1 {'name': 'Accounting'}>, 'name': 'Jack', 'department_id': '2'}>

N.B.

>>> p2.department_id
'2'
>>> p2.department.id
'1'

Self-reference (recursive) ManyToOne/OneToMany relationships available?

This is what I meant by self-reference.

class Employee(models.model):
    manager = models.ForeignKey('Employee')

This is how Django implements recursive relationships.

To create a recursive relationship – an object that has a many-to-one relationship with itself – use models.ForeignKey('self', on_delete=models.CASCADE).

Is this available to use?

any example is appreciated.

DateTimeField getting corrupted on save if microsecond part has less then 6 positions

Script below raises AssertionError while it should run successfuly.

from redisco import models
import datetime as dt
import redisco

redisco.connection.flushdb()

class A(models.Model):
    when = models.DateTimeField()

when = dt.datetime(2011, 1, 1, 0, 0, 0, 1)
A.objects.create(when=when)
obj = A.objects.get_by_id(1)

assert obj.when.microsecond == when.microsecond

ModelSet.filter / zfilter methods should cleanup their mess

They should either delete the stored sets/sorted set immediately after getting the result or expire the keys to the set.

Missing "zrevrangebyscore" function on SortedSet

The SortedSet container does not expose the zrevrangebyscore function (listed here http://redis.io/commands/zrevrangebyscore)

ImportError

I am getting the following ImportError. I am using Python 3.3.3 - before with Python 2.7.6 everything worked.

Traceback (most recent call last):
  File "./run.py", line 3, in <module>
    from app import app
  File "/Users/govindaf/workspace/commander/commander/app/__init__.py", line 8, in <module>
    from app import views
  File "/Users/govindaf/workspace/commander/commander/app/views.py", line 7, in <module>
    from redisco import models, connection_setup
  File "/Users/govindaf/workspace/commander/commander/venv/lib/python3.3/site-packages/redisco/models/__init__.py", line 1, in <module>
    from base import *
ImportError: No module named 'base'

What is wrong here?

filter and chained zfilter prevent filter to be applied

This code:

obj.filter(obj_ref_id=1).zfilter(count__in=(1, 20)) is not executed properly. Only the zfilter is apply, leaving the filter on the object reference out.

The code documentation states that only one zfilter will be applied due to performances issue but the previous filter is not applied and should.

AttributeError: has no attribute '_errors'

The errors property on models.base should check for _errors first, and if not exists call .is_valid for you

I can:
x = MyModel.objects.create()
x.errors # == []

but not:
x = MyModel()
x.errors # raises AttributeError!

DateTimeField save wrong value

At: https://github.com/kiddouk/redisco/blob/master/redisco/models/attributes.py#L216
I believe the correct microsecond part should be: value.microsecond / 1000000.0

Thank!

Redisco should be TimeZone Aware

Redisco should support Timezones or at least, store the DateTimeFields in UTC instead of trusting the naive objects.

Pipelining or Bulk_Create?

I had a look around and didn't seem to find pipelining/"bulk_create" options.

This is the only thing stopping me from using redisco, and even testing it in production.

Using Mixins to add common fields to a model doesn't work

I tried using a mixin with some common fields, but it doesn't work:

class CeleryTaskMixin:
    celery_status = models.CharField()  # last known status, call update_status
    celery_id = models.CharField()      # celery uuid
    f_name = models.CharField(indexed=True)


class RunningTask(CeleryTaskMixin, models.Model):
    def __init__(self, t=None, *args, **kwargs):
        CeleryTaskMixin.__init__(self)
        models.Model.__init__(self)

    task_type = models.CharField()
    start_time = models.DateTimeField(auto_now=True)
    end_time = models.DateTimeField()

Then when I try and do

models.RunningTask.objects.filter(f_name="blah")

It says

  File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 52, in __iter__
    for id in self._set:
  File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 294, in _set
    s = self._add_set_filter(s)
  File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 317, in _add_set_filter
    (k, self.model_class.__name__))
redisco.models.exceptions.AttributeNotIndexed: Attribute f_name is not indexed in RunningTask class.

This seems like something that should work... (?)

Saving twice causes model to become non-unique.

Given a redisco model:

class User(models.Model):
    name = models.Attribute(required=True, unique=True)
    password = models.Attribute()

the following sequence fails:

u = User(name='foo')
u.password = 'secret'
u.save()
u.save()

Now, maybe this is reasonable, but the big problem is the return value from the second save call is:

[('name', 'not unique')]

which is extremely misleading.

Storing utc datetime values for auto_now_add

I went through your code, and found that when auto_now_add is set to True you store datetime.now() .
This shall fail in timezone oriented app. Though mostly applications are hosted on cloud servers and they are configured for utc timezone. This library won't fail when server is in utc timezone. This library will definitely fail in development environment if development machine is not set to use utc timezone.

Though your code can store timezone aware datetimes, but auto_now_add will be useless in timezone oriented app.

Hope you would fix this.

Thanks for brilliant hard work.

Indexing for DateTimeField and auto_now_add

Hi Tim,

DateTimeField is ignored/filtered while initializing indices.

I have written a small patch and would send a pull request in few moments. Although it does not solve problem for datetime field with auto_now_add as at points redisco checks if value is passed and the only creates the index.

Cheers,
Shekhar

Add a `unique_together` option

It would be great if redisco could have a unique_together option (in the Meta class)

Sometimes a field only is not enough to manage unicity.

Any hint on how to add this ?

PS : https://docs.djangoproject.com/en/dev/ref/models/options/#unique-together

get_by_id should be aware of previous filters

Considering this case :

class A(models.Model):
    user = models.ReferenceField(B)

class B(models.Model):
    att = models.Attribute()


b1 = B(att='test)
b1.save()

b2 = B(att='test')
b2.save()

a = A(user=b1)
a.save()

self.assertEqual(a.user_set.get_by_id(b1.id), b1)
self.assertEqual(a.user_set.get_by_id(b2.id), None)

The second test will fail as get_by_id doesn't take into account any filter/zfilter/excludes.

operations on ListField are not atomic

This may actually be a feature request. But, I had inferred from the documentation of Ohm that append and pop operations on a ListField would be immediately persisted and atomic (see "Persistence strategy" in http://github.com/soveran/ohm/blob/master/README.markdown) - but this doesn't seem to be the case...

with Book and Author models from tests.models.ListFieldTestCase.test_list_of_reference_fields

>>> b1 = Book.objects.create(title='Book1', date_published=date.today())
>>> b2 = Book.objects.create(title='Book2', date_published=date.today())
>>> a1 = Author.objects.create(name='Author1', books=[b1, b2])
>>> # then in another thread
... 
>>> a2 = Author.objects.get_by_id(1)
>>> b3 = Book.objects.create(title='Book3', date_published=date.today())
>>> a2.books.append(b3)
>>> a2.save()
True
>>> len(a2.books)
3
>>> # meanwhile back in the first request
... 
>>> a1.save()
True
>>> len(Author.objects.get_by_id(1).books)
2

Atomic operations on redis lists (and sets?) are one of the killer features, and I'd like redisco to abstract them for me - e.g. Counter! The "Last Write Wins" strategy makes a lot of sense in some places, but an atomic list field would probably be preferable in many cases.

I could see reason to disabled set on ListField entirely in favor of append and pop (lpush, rpop?) on the mode base a la Counters' incr and decr? But for backwards compat maybe it would better to create a new AtomicListField (don't love the name...)

More interesting ideas might include allowing get to return the counters.List instance directly instead of cast members, or at least some sort of proxy/wrapper around it - or for _redisco_model, maybe something similar to a ModelSet?

I wonder if you've considered this case, or what approach you would find more amiable?

Attribute can't handle Unicode

If you store unicode in an Attribute, you won't get back unicode, but an utf-8 encoded str. That's because redis-py (at least by default) encodes all unicode values to utf-8, before executing a command, but cannot decode the read values. Redisco should therefore not pass unicode objects to redis-py but do utf-8 conversion itself.

Some architecture questions

I was looking at redisco models code, to adapt my patch to Models. I become many questions about its architecture.

What is doing Manager class ? It is just a alias to ModelSet ?

What is doing key.py, validation.py, utils.py ?

For what is need ModelBase ? Why ModelBase init was splited to many functions ? Why not methods ?

Uhm. Thanks :)

ReferenceField not indexable

The ReferenceField should be indexable, thus allowing to filter by such a field.

Deferred model references broken

When you create a model reference with a string as the target type, redisco fails to create an <origin>_set attribute on the target.
>>> class User(models.Model):
... name = models.CharField()
... address = models.ReferenceField('Address')
...
>>> class Address(models.Model):
... street_address = models.CharField()
...
>>> a=Address()
>>> a.user_set
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'Address' object has no attribute 'user_set'
>>> a.address_set # Exists and raises redisco.models.exceptions.MissingID

Modified test case and fix are here

Float precision issue in ranged queries

The current implementation of ranged queries does not work correctly in some cases due to the way redis-py handles floats.

The implementations of lt, gt, etc. in SortedSet, all call redis-py's zrangebyscore with a float as one of the arguments (_min_score or _max_score). redis-py calls convert on these to convert them to strings, using str(), which results in a loss of float precision. This means, that ranged queries like FooModel.objects.zfilter(some_attr__lt=10.009) return wrong results. I'd suggest converting float arguments to a string using "%f" % value before passing them to a redis-py query.

Operations like ZADD and PUSH should return redis returned value

It is usually useful to check the value returned by redis when doing a zadd. It helps knowing if the elements has been added or just updated. Redisco should reflect that.

Recursion issue with ListField of ReferenceField

I'm trying to use ListField of ReferenceField
to maintain a list of user channels of my User model
as well as a list of users in my Channel model.

Model:

# Module:   models
# Date:     16th August 2014
# Author:   James Mills, prologic at shortcircuit dot net dot au


"""Data Models"""


from circuits.protocols.irc import joinprefix

from redisco.models import Model
from redisco.models import (
    Attribute, BooleanField, DateTimeField,
    IntegerField, ListField, ReferenceField
)


class User(Model):

    host = Attribute(default="")
    port = IntegerField(default=0)

    nick = Attribute(default=None)
    away = BooleanField(default=False)

    channels = ListField("Channel")
    userinfo = ReferenceField("UserInfo")

    registered = BooleanField(default=False)
    signon = DateTimeField(auto_now_add=True)

    @property
    def prefix(self):
        userinfo = self.userinfo
        return joinprefix(self.nick, userinfo.user, userinfo.host)

    class Meta:
        indices = ("id", "nick",)


class UserInfo(Model):

    user = Attribute(default=None)
    host = Attribute(default=None)
    server = Attribute(default=None)
    name = Attribute(default=None)

    def __nonzero__(self):
        return all(x is not None for x in (self.user, self.host, self.name))


class Channel(Model):

    name = Attribute(required=True, unique=True)
    users = ListField("User")

    class Meta:
        indices = ("id", "name",)

This seems to work fine when there is one User and one Channel
object(s) in the database, but as soon as there are two different users
that are members of the same channel I get this error:

2014-08-16 22:02:51,218 - charla.main - ERROR - ERROR <handler[*.join] (Commands.join)> (<join[commands] (<socket._socketobject object at 0x7f956fb54c20>, ('', None, None), '#circuits' )>) (<type 'exceptions.RuntimeError'>): RuntimeError('maximum recursion depth exceeded while calling a Python object',)
  File "/home/prologic/work/circuits/circuits/core/manager.py", line 603, in _dispatcher
    value = handler(*eargs, **ekwargs)
  File "/home/prologic/charla/charla/plugins/core.py", line 111, in join
    user.save()
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 202, in save
    self._write(_new)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 345, in _write
    self._update_indices(pipeline)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 413, in _update_indices
    self._add_to_indices(pipeline)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 418, in _add_to_indices
    self._add_to_index(att, pipeline=pipeline)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 426, in _add_to_index
    index = self._index_key_for(att)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 474, in _index_key_for
    return self._tuple_for_index_key_attr_list(att, value)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 492, in _tuple_for_index_key_attr_list
    return ('list', [self._index_key_for_attr_val(att, e) for e in val])
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 499, in _index_key_for_attr_val
    return self._key[att][_encode_key(val)]
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/utils.py", line 5, in _encode_key
    return base64.b64encode(str(s)).replace("\n", "")
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 516, in __repr__
    return "<%s %s>" % (self.key(), self.attributes_dict)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 516, in __repr__
    return "<%s %s>" % (self.key(), self.attributes_dict)

... repeated lines delete

  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 516, in __repr__
    return "<%s %s>" % (self.key(), self.attributes_dict)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/base.py", line 247, in attributes_dict
    h[k] = getattr(self, k)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/models/attributes.py", line 273, in __get__
    val = List(key).members
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 34, in __getattribute__
    return object.__getattribute__(self, att)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 217, in all
    return self.lrange(0, -1)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 32, in __getattribute__
    return partial(getattr(object.__getattribute__(self, 'db'), att), self.key)
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 39, in db
    if self.pipeline:
  File "/home/prologic/.virtualenvs/charla/lib/python2.7/site-packages/redisco/containers.py", line 31, in __getattribute__
    if att in object.__getattribute__(self, 'DELEGATEABLE_METHODS'):

I believe Issue #26 is somewhat related?

Is a ListField of ReferenceFields possible?

Scenario: I have a "Game" class that I would like to be able to refer to one or more "Player" objects. If I'm okay with storing the player data in each game, I can do:

class Game(models.Model):
    players = models.ListField(Player)

But I'd like it to just be a reference, not contain copies of each player. For one player, this would work:

class Game(models.Model):
    players = models.ReferenceField("Player", related_name="game")

What I really want (or what I think I want) is a ListField of ReferenceFields. Of course, if I try this:

class Game(models.Model):
    players = models.ListField(models.ReferenceField("Player", related_name="game"))

It's not going to work, since it's going to instantiate the ReferenceField instead of it being a class. Is there any way to accomplish what I want?