sebleier/django-redis-cache

A Redis cache backend for django

django-redis-cache's Issues

tests don't run (no such module as redis.server)

I'm trying to run the tests so I can add and test a new test for a bug
I want to submit a pull request for.

It appears that a non-existant piece of the redis module is being called:

Traceback (most recent call last):
  File "./run_tests.py", line 11, in <module>
    from redis.server import server
ImportError: No module named server

I have tried redis-2.4.12 and redis-2.4.13 (the latest), and done a Google
search and do not see the 'server' module mentioned anywhere. Also,
the README does not give any instructions for running the tests.

I'm attempting to run the tests by running the run_all_tests.sh script.

Can you provide any information that'll help me get the tests running?

Thanks,
Shawn

Untitled

Strings with numeric values get cast to ints.

>>> from django.core.cache import cache
>>> cache
<redis_cache.cache.RedisCache object at 0x1abaa50>
>>> cache.set("foo", "1")
True
>>> cache.get("foo")
1
>>> cache.get("foo") == "1"
False

It's hard to say if this is a bug or not. #27 seems to indicate it's intentional. There's also a unit test for floats that explicitly tests that this behaviour occurs.

If it's intentional a great big warning at the start of the README would probably be useful. Happy to write one if you'd like.

Use Redis' EXISTS for `has_key`

Should save some time, especially for large cache values.

float automatically type casted to int

https://github.com/sebleier/django-redis-cache/blob/master/redis_cache/cache.py#L205

The first thing that cache.set does is try to convert to int. So if I set a key's value to a float its stored as int and I get back an int.
Whereas, if I set it as a string I am able to preserve the precision, though I get it back as string (as stored)

In [1]: from django.core.cache import cache

In [2]: cache.set('ffff', "2.5")
Out[2]: True

In [3]: cache.get('ffff')
Out[3]: '2.5'

In [4]: cache.set('ffff', 2.5)
Out[4]: True

In [5]: cache.get('ffff')
Out[5]: 2

Isn't this auto typecasting a bug?

Process hangs when the remote server is running but doesn't allow you to access.

I'm using redis_cache something like this

from redis_cache import get_redis_connection
from rq import Queue

try:
    queue = Queue(connection=get_redis_connection('db'))
    queue.enqueue(some_action, some_data)
except Exception:
    do_something_else(some_data)

I was expecting a ConnectionError or some other exceptions when the remote server refused me to access, but the program just didn't respond anything.

Could someone give a clue about this?

Issue With Expires Header - Possible config problem ?

Hi,

I seem to be having an issue with my expires / last modified header values not updating after they are set initially.

HTTP/1.1 200 OK
Cache-Control: max-age=120
Content-Type: text/html; charset=utf-8
Date: Sat, 18 Apr 2015 09:52:44 GMT
Expires: Sat, 18 Apr 2015 07:31:30 GMT
Last-Modified: Sat, 18 Apr 2015 07:29:30 GMT
Server: nginx/1.2.1
Vary: Cookie
Connection: keep-alive

Note that the time of the request is well after the expires value, but it does not update and still serves up the old version of the page.

My settings configs are as follows:

CACHES = {
'default': {
    'BACKEND': 'redis_cache.RedisCache',
    'LOCATION': 'my-redis-server:6379',
    },
}

MIDDLEWARE_CLASSES = (
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware'
)

And I've added the @cache_page decorator to my view:

@cache_page(120)
def home(request):
    ...

Can anyone tell me what the issue is ?

Relative imports on __init__.py ?

Hello!

Is there any reason why not to use relative imports in __init__.py ?

_Example_:
from .cache import RedisCache

It is a suggestion, not that behaves the compatibility with Python <2.6.

Thank you.
Andrei

Separate redis backends do not work

I've configured two entirely separate redis instances on different machines for different purposes, and it looks like it is not obeying which backend is being invoked and choosing to write to whichever random one it wants...

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': 'memcache01',
        'KEY_FUNCTION': 'commons.caching.make_key',
    },
    'redis': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': 'redis01:6379',
        'KEY_FUNCTION': 'commons.caching.make_key_plain',
        'OPTIONS': {
            'DB': 0,
        }
    },
    'otherredis': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': 'redis02:6379',
        'KEY_FUNCTION': 'commons.caching.make_key_plain',
        'OPTIONS': {
            'DB': 0,
        }
    },
}

I'm explicitly using get_cache('redis') in one place, and get_cache('otherredis') elsewhere, and watching both redis instances by telnetting and running MONITOR. I am seeing it write to both redis instances...

I'm not too sure on how the backends work in django so I haven't dug into where this is happening yet, but it seems that the code isn't built to handle using separate redis instances.

Support for redis 3.0 Clusters?

Hi,

Would it be possible to have redis-3.0 cluster support either via redis-py-cluster or otherwise?

Add an official dependency on redis-py

django-redis-cache's setup.py should officially require redis-py. I can't think of a reason why you'd install redis-cache without wanting to install the redis bindings, so not having the dependency just makes installtion more complicated than it needs to be.

Changelog needs updating.

The Changelog in the README only has entries up to version 1.50, but the current version is 1.6.3.

handle ConnectionError and skip it

memcached backend didnt raise ConnectionError, so why redis-cache do?

maybe add setting like 'SKIP_CONNECTION_ERRORS'?

TypeError: unhashable type: 'list' error when giving more than one CACHE LOCATION

Hi all,

I've installed django-redis-cache and configured per the instructions on this github index page. However, I get the error from Exhibit 1. I get the error when LOCATION is defined as a list (See SYNTAX 1). If I set the LOCATION to a string, it works ok (See SYNTAX 2). The Django Docs indicate that both syntaxes should work (at least in the memcached example) and the example django-redis-cache configuration on github supports that notion as well. Anyway, just bringing this up as it may prevent people configuring multiple redis cache servers.

Thanks for reading,
Joe

Background: I'm using Django 1.4 and have installed this app and all the deps (pip install django-redis-cache). Redis is configured and I can connect to it via the redis-cli. I can also connect to redis-py by directly connecting to it via the django shell, so I'm not getting any connection issues. (See Exhibit 2)

SYNTAX 1: CAUSE OF ERROR

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': ['127.0.0.1:6379',],
        'OPTIONS': {
            'DB': 1,
            'PARSER_CLASS': 'redis.connection.HiredisParser',
            'PICKLE_VERSION': 2,
        },
    },
}

SYNTAX 2: WORKAROUND

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': '127.0.0.1:6379',
        'OPTIONS': {
            'DB': 1,
            'PARSER_CLASS': 'redis.connection.HiredisParser',
            'PICKLE_VERSION': 2,
        },
    },
}

EXHIBIT 1

myapp_new)myuser@mycomp:~/Sites/myapp_new/proj/myapp$ ./manage.py runserver 
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/management/__init__.py", line 261, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/management/__init__.py", line 69, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/runserver.py", line 4, in <module>
    from django.core.management.commands.runserver import BaseRunserverCommand
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 8, in <module>
    from django.core.servers.basehttp import AdminMediaHandler, run, WSGIServerException, get_internal_wsgi_application
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 28, in <module>
    from django.contrib.staticfiles import handlers
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 8, in <module>
    from django.contrib.staticfiles.views import serve
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/contrib/staticfiles/views.py", line 15, in <module>
    from django.contrib.staticfiles import finders
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/contrib/staticfiles/finders.py", line 11, in <module>
    from django.contrib.staticfiles.storage import AppStaticStorage
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 10, in <module>
    from django.core.cache import (get_cache, InvalidCacheBackendError,
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/cache/__init__.py", line 187, in <module>
    cache = get_cache(DEFAULT_CACHE_ALIAS)
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/django/core/cache/__init__.py", line 179, in get_cache
    cache = backend_cls(location, params)
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/redis_cache/cache.py", line 77, in __init__
    self._init(server, params)
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/redis_cache/cache.py", line 104, in _init
    **kwargs
  File "/home/myuser/Sites/myapp_new/lib/python2.7/site-packages/redis_cache/cache.py", line 50, in get_connection_pool
    if not self._connection_pools.get(connection_identifier):
TypeError: unhashable type: 'list'

EXHIBIT 2

Python 2.7.3 (default, Apr 20 2012, 22:44:07) 
[GCC 4.6.3] on linux2 
Type "help", "copyright", "credits" or "license" for more information.
>>> import redis
>>> r = redis.StrictRedis(host="localhost", port=6379, db=1)
>>> r
<redis.client.StrictRedis object at 0xb6fa984c>
>>> r.set('test','test')
True
>>> r.get('test')
'test'

Getting InvalidCacheBackendError

I have a new project where is installed redis-cache from scratch using pip, so i have version 0.9.2. my redis-py install is also fresh; 2.4.11, django is 1.3.1. I have setup Redis cache as follows:

INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admin',
'django.contrib.admindocs',

# tools
'south',
'epio_commands',
'raven.contrib.django',
'debug_toolbar',
'redis_cache',
'feincms',
'feincms.module.page',
'feincms.module.medialibrary',

# main app
'core',
)

CACHES = {
'default': {
    'BACKEND': 'redis_cache.RedisCache',
    'LOCATION': '{host}:{port}'.format(
            host=config['redis']['host'],
            port=int(config['redis']['port'])
    ),
    'OPTIONS': {
        'PASSWORD': config['redis']['password'],
        'MAX_ENTRIES': 10000
    },
    'VERSION': config['core']['version'],
    'TIMEOUT': 3600*8 # 8 hours
},
}

The weird thing is that while this setup works perfectly in other projects in this project i get the following error:

File "/Users/tijs/.virtualenvs/swov/lib/python2.7/site-packages/django/core/cache/__init__.py", line 179, in get_cache
"Could not find backend '%s': %s" % (backend, e))
django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'redis_cache.RedisCache': 'module' object has no attribute 'RedisCache'

The full stacktrace is here: http://pastebin.com/butH6Jup

timeout not accept None

Hello

Django documentation say:

The timeout argument is optional and defaults to the timeout argument of the appropriate backend in the CACHES setting (explained above). It’s the number of seconds the value should be stored in the cache. Passing in None for timeout will cache the value forever.

But django-redis-cache don't accept None for timeout:

>>> from django.core.cache import cache
>>> cache.set('sdfdsfsdf','cxvxcvxcv', timeout=None)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/xxx/python/xxxx/lib/python2.7/site-packages/redis_cache/cache.py", line 237, in set
    result = self._set(key, pickle.dumps(value), int(timeout), client, _add_only)
TypeError: int() argument must be a string or a number, not 'NoneType'

RemovedInDjango19Warning

I'm getting the following warning:

/Users/xxxx/.virtualenvs/xxxxx/lib/python2.7/site-packages/redis_cache/cache.py:3: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
  from django.utils import importlib

Redis Pipelining Support

I'm using this extension on my application, I just wanted to know if it's possible to use Redis in pipeline mode with it or it already has it built in.

Error Python3.2 django 1.8.2

Django Version: 1.8.2
Exception Type: ImproperlyConfigured
Exception Value:
Port value must be an integer
Exception Location: /usr/local/lib/python3.2/dist-packages/redis_cache/backends/base.py in create_client, line 164
Python Executable: /usr/bin/python3.2

Install Release 1.1.1

As of 2.4.5, redis-py no longer has a connection method in the Redis object, breaking django-redis-cache

As title. redis-py now uses ConnectionPools by default, and there doesn't appear to be any kind of disconnect() command in the Redis class. This breaks django-redis-cache.

Note that I've reported this issue after a cursory read of the source code to both this project and redis-py, so I don't guarantee I have the best grasp of the matter here.

issue when using with multi-cache backends

since Django 1.3 start supporting multi-cache backends. There's a issue in current code.

<settings.py>
CACHES = {
'default': {
'BACKEND': 'redis_cache.RedisCache',
'LOCATION': '192.168.32.139:6379',
'OPTIONS': {
'DB': 2,
#'PARSER_CLASS': 'redis.connection.HiredisParser',
}
},

'2ndcache': {
    'BACKEND': 'redis_cache.RedisCache',
    'LOCATION': '192.168.32.139:6379',
    'OPTIONS': {
        'DB': 3,
        #'PARSER_CLASS': 'redis.connection.HiredisParser',
    },
}    

}

<views.py>
...
cache.set('x',1) #default cache object, using db 2
get_cache('2ndcache').set('x',2) # another cache object, should use db3 according to <settings.py>. But it didn't worked as I expected.

because, in <django-redis-cache / redis_cache / cache.py>
line 46, a new pool will be created when _connection_pool is None. But since one pool works for one db.
so, the when we call
get_cache('2ndcache').set('x',2)
the CacheConnectionPool.get_connection_pool still return the connection pool for db 2, not 3.

MY MODIFICATION FOR THE CODE:
use a dict, to indicate every pair of db:pool

class CacheConnectionPool(object):
_connection_pools = {} #id: reis.ConnectionPool

def get_connection_pool(self, host='127.0.0.1', port=6379, db=1,
    password=None, parser_class=None,
    unix_socket_path=None):
    if self._connection_pools.get(db) is None:
        connection_class = (
            unix_socket_path and UnixDomainSocketConnection or Connection
        )
        kwargs = {
            'db': db,
            'password': password,
            'connection_class': connection_class,
            'parser_class': parser_class,
        }
        if unix_socket_path is None:
            kwargs.update({
                'host': host,
                'port': port,
            })
        else:
            kwargs['path'] = unix_socket_path
        self._connection_pools[db] = redis.ConnectionPool(**kwargs)
    return self._connection_pools.get(db)

Thanks ~

Master/Slave KeyError: 'unix_socket_path'

I'm suddenly getting an error when running master/slave setup.

web_1 | [Wed Jul 22 19:53:18.569499 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/sonofatailorapp/config/urls.py", line 22, in <module>
web_1 | [Wed Jul 22 19:53:18.569539 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     url(r'^giftcard/(?P<slug>[\\w-]+)?', cache_page(60 * 15)(GiftcardView.as_view())),
web_1 | [Wed Jul 22 19:53:18.569543 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/.whiskey/python/lib/python2.7/site-packages/django/views/decorators/cache.py", line 35, in cache_page
web_1 | [Wed Jul 22 19:53:18.569556 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     cache_timeout=cache_timeout, cache_alias=cache_alias, key_prefix=key_prefix
web_1 | [Wed Jul 22 19:53:18.569559 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/.whiskey/python/lib/python2.7/site-packages/django/utils/decorators.py", line 96, in _make_decorator
web_1 | [Wed Jul 22 19:53:18.569569 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     middleware = middleware_class(*m_args, **m_kwargs)
web_1 | [Wed Jul 22 19:53:18.569572 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/.whiskey/python/lib/python2.7/site-packages/django/middleware/cache.py", line 181, in __init__
web_1 | [Wed Jul 22 19:53:18.569581 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     self.cache = caches[self.cache_alias]
web_1 | [Wed Jul 22 19:53:18.569584 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/.whiskey/python/lib/python2.7/site-packages/django/core/cache/__init__.py", line 113, in __getitem__
web_1 | [Wed Jul 22 19:53:18.569594 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     cache = _create_cache(alias)
web_1 | [Wed Jul 22 19:53:18.569597 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/.whiskey/python/lib/python2.7/site-packages/django/core/cache/__init__.py", line 88, in _create_cache
web_1 | [Wed Jul 22 19:53:18.569605 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     return backend_cls(location, params)
web_1 | [Wed Jul 22 19:53:18.569608 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/.whiskey/python/lib/python2.7/site-packages/redis_cache/backends/single.py", line 23, in __init__
web_1 | [Wed Jul 22 19:53:18.569619 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     self.master_client = self.get_master_client()
web_1 | [Wed Jul 22 19:53:18.569621 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]   File "/app/.whiskey/python/lib/python2.7/site-packages/redis_cache/backends/base.py", line 148, in get_master_client
web_1 | [Wed Jul 22 19:53:18.569631 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040]     kwargs['unix_socket_path'],
web_1 | [Wed Jul 22 19:53:18.569640 2015] [wsgi:error] [pid 9:tid 140107097552640] [remote 172.17.42.1:9040] KeyError: 'unix_socket_path'

My config looks like this for docker-compose

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': [
            'redis://redis-slave'
        ],
        'OPTIONS': {
            'DB': 1,
            'PARSER_CLASS': 'redis.connection.HiredisParser',
            'PICKLE_VERSION': 2,
            'MASTER_CACHE': 'redis://redis-master',
        },
    },
}

and like this in kubernetes

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': [
            'redis-slave:6379'
        ],
        'OPTIONS': {
            'DB': 1,
            'PARSER_CLASS': 'redis.connection.HiredisParser',
            'PICKLE_VERSION': 2,
            'MASTER_CACHE': 'redis-master:6379',
        },
    },
}

It yields the same error, so i'm suspecting, somethings changed either in this library or in one of it's dependencies.

I'm using django 1.8 and and this config works fine

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': [
            'redis://redis-master'
        ],
        'OPTIONS': {
            'DB': 1,
            'PARSER_CLASS': 'redis.connection.HiredisParser',
            'CONNECTION_POOL_CLASS': 'redis.BlockingConnectionPool',
            'CONNECTION_POOL_CLASS_KWARGS': {
                'max_connections': 50,
                'timeout': 20,
            },
            'MAX_CONNECTIONS': 1000,
            'PICKLE_VERSION': -1,
        },
    },
}

when i remove cache_page from my urls.py file it also seems to work again. It all worked, but now suddenly it doesn't. Have anyone had the same experience ?

delete...

wrong project...

get_or_set not using timeout

When using get_or_set with a timeout specified, the timeout is not respected and simply set as None/-1
The lock key is also set to -1. Don't know if that's intended.

https://github.com/sebleier/django-redis-cache/blob/master/redis_cache/backends/base.py#L400
None is sent to the _set method.

To me it also seems like the lock key is set twice? First with a timeout of None and then later with the actual timeout, but since it's already set it's ignore.

support for read vs write replicas

first off, thanks for django-redis-cache, we love it!

we have a feature request: we'd love to use a redis replication group with both read only and read/write replicas. any chance you could support that in the future? i'm guessing we'd either provide multiple django cache backends with well-known names, or we'd specify the read vs write endpoints in OPTIONS, and you'd pick the right one for each cache operation. or however you want!

(related to #3 and #31, but not the same.)

thanks in advance!

No module named 'redis_cache.backends'

Hi,
since 1.0, my settings cache is

    'default': 
    {
        'TIMEOUT': 3600,
        "BACKEND": "redis_cache.RedisCache",
        "LOCATION": "127.0.0.1:6379",
        "OPTIONS": {
            "DB": 1,
            'PARSER_CLASS': 'redis.connection.HiredisParser'
        }
    }

and now, we get this errors :

django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'redis_cache.RedisCache': No module named 'redis_cache.backends'

but previously, with version 0.13.x my settings were :

    'default':
    {
        'TIMEOUT': 3600,
        "BACKEND": "redis_cache.cache.RedisCache",
        "LOCATION": "127.0.0.1:6379",
        "OPTIONS": {
            "DB": 1,
            "CLIENT_CLASS": "redis_cache.client.DefaultClient",
        }
    },

the backends package seems to be missing from the installation, doesn't it ?

 ll lib/python3.4/site-packages/redis_cache/
total 28
-rw-r--r-- 1 foxmask foxmask  199 juin  24 10:25 cache.py
-rw-r--r-- 1 foxmask foxmask 1084 juin  24 10:25 compat.py
-rw-r--r-- 1 foxmask foxmask 1885 juin  24 10:25 connection.py
-rw-r--r-- 1 foxmask foxmask  111 juin  24 10:25 __init__.py
drwxr-xr-x 2 foxmask foxmask 4096 juin  24 10:25 __pycache__
-rw-r--r-- 1 foxmask foxmask 1367 juin  24 10:25 sharder.py
-rw-r--r-- 1 foxmask foxmask  529 juin  24 10:25 utils.py

No way to specify socket_timeout

Looks like there is no way to specify the socket_timeout in the OPTIONS, and the default socket_timeout in the underlying redis-py is None. It would be prudent to add the ability to set this so things don't hang if a connection can't be made.

Support incr/decr for Django >= 1.3

Since redis has incrby, it seems crazy to let Django's default get-inc-set get in the way.

def incr(self, key, delta=1, version=None):
    key = self.make_key(key, version)
    return self._cache.incrby(key, delta)

or possibly with the return value wrapped in self.unpickle.

The downside would be that behaviour wrt unset keys is changed (and can't be emulated perfectly without losing the atomicity of incrby). I for one would happily pay that price though, since the majority of use of incr wants a default of 0 anyway.

Use hash_ring to provide access to multiple redis nodes

http://amix.dk/blog/viewEntry/19367
Consistent hashing for an arbitrary pool; An application of it for redis here, courtesy ericflo:
http://dpaste.de/R27r/

[set timeout=Noen] not take effect, because django version is lower than 1.6

my django version is 1.5.5,
and this code below may cause bug;

in file C:\Python27\Lib\site-packages\redis_cache\compat.py

if django.VERSION[:2] >= (1, 6):
from django.core.cache.backends.base import DEFAULT_TIMEOUT as DJANGO_DEFAULT_TIMEOUT
DEFAULT_TIMEOUT = DJANGO_DEFAULT_TIMEOUT
else:
DEFAULT_TIMEOUT = None

SyntaxError: invalid syntax

Stumbled upon this syntax error using Python 3.4, probably because I didn't install hiredis.

Traceback (most recent call last):
  File "src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/djcelery/management/commands/celery.py", line 23, in run_from_argv
    ['{0[0]} {0[1]}'.format(argv)] + argv[2:],
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/bin/celery.py", line 761, in handle_argv
    return self.execute(command, argv)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/bin/celery.py", line 693, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/worker/__init__.py", line 95, in __init__
    self.app.loader.init_worker()
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/celery/loaders/base.py", line 129, in init_worker
    self.on_worker_init()
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/djcelery/loaders.py", line 132, in on_worker_init
    self.close_cache()
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/djcelery/loaders.py", line 99, in close_cache
    cache.cache.close()
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/django/core/cache/__init__.py", line 131, in __getattr__
    return getattr(caches[DEFAULT_CACHE_ALIAS], name)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/django/core/cache/__init__.py", line 113, in __getitem__
    cache = _create_cache(alias)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/django/core/cache/__init__.py", line 84, in _create_cache
    backend_cls = import_string(backend)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/django/utils/module_loading.py", line 26, in import_string
    module = import_module(module_path)
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/redis_cache/__init__.py", line 1, in <module>
    from redis_cache.backends.single import RedisCache
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/redis_cache/backends/single.py", line 7, in <module>
    from redis_cache.backends.base import BaseRedisCache
  File "/Users/antonagestam/.virtualenvs/5m-ia/lib/python3.4/site-packages/redis_cache/backends/base.py", line 108
    except ImportError, e:
                      ^
SyntaxError: invalid syntax

Can not found hiredis(But installed actually)

wiht django 1.3.1, redis 2.4.10, hiredis 0.1.0

Traceback (most recent call last):
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/sentry/client/base.py", line 43, in _wrapped
return func(_args, *_kwargs)
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/sentry/client/base.py", line 90, in set_last_message_id
cache.set(cache_key, message_id, settings.THRASHING_LIMIT + 5)
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis_cache/cache.py", line 210, in set
result = self._set(key, pickle.dumps(value), int(timeout), client)
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis_cache/cache.py", line 191, in _set
return client.setex(key, value, int(timeout))
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis/client.py", line 1140, in setex
return self.execute_command('SETEX', name, time, value)
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis/client.py", line 280, in execute_command
connection.send_command(_args)
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis/connection.py", line 242, in send_command
self.send_packed_command(self.pack_command(_args))
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis/connection.py", line 225, in send_packed_command
self.connect()
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis/connection.py", line 176, in connect
self.on_connect()
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis/connection.py", line 197, in on_connect
self._parser.on_connect(self)
File "/home/kuno/utopia/sogoke/lib/python2.7/site-packages/redis/connection.py", line 111, in on_connect
self._reader = hiredis.Reader(

NameError: global name 'hiredis' is not defined

Release version with py3 fixes

Hello @sebleier

Can you please make a new release with Python3 fixes you made recently. I would love to use the version from PyPI, but it's current version 1.1.0 is broken in Python 3. Thank you!

Accessing to raw client

Why not adding a function/method to access to the raw redis client ? For example to have access to the more advanced function of redis.

Limit connection pool size

There should be a way to limit size of Redis connection pool with the redis-py max_connections parameter.

Deployments with hard limits on number of connections (i.e. using third-party Redis services) fail when connections are dropped due to these limits.

Is close() necessary?

Perhaps I'm doing it wrong, but Django appears to tear down the redis cache backend at the end of every request: I'm watching the output of ./redis-server and it has a whole bunch of Accepted xxx and Client closed connection while I'm running an ab test.

django/core/cache/init.py states at the bottom:
# Some caches -- pythont-memcached in particular -- need to do a cleanup at the
# end of a request cycle. If the cache provides a close() method, wire it up
# here.

Confirming this, I checked the django memcache backend and yes, they do include a close. However, if I comment out the close() function for django-redis-cache things keep working as normal. A limited number of connections stick around (based on my wsgi num. of process & threads) and only go away if I stop the dev server. Also, I get a slight performance boost in terms of reqs/sec.

Is there a known issue with idling connections or some other issue with thread-safety? Otherwise, losing close() will net a bit of a performance boost.

Support for redis.from_url

For the Heroku folks, it's really nice to be able to pass a redis URL from an environment variable straight into config. For instance, https://github.com/ui/django-rq supports a URL config allowing something like the following:

RQ_QUEUES = {
    'high': {
        'URL': os.getenv('REDISTOGO_URL', 'redis://localhost:6379'), # If you're on Heroku
        'DB': 0,
        'DEFAULT_TIMEOUT': 500,
    }
}

The redis-py module comes with a method for doing this: http://redis-py.readthedocs.org/en/latest/#redis.StrictRedis.from_url

Handle wildcard deletion of keys

Redis allows us to find glob-style wildcard-matching keys with the use of the KEYS command[0]

assuming keys 'apple', 'banana', 'boat':

redis 127.0.0.1:6379> keys *
1) "apple"
2) "banana"
3) "boat"
redis 127.0.0.1:6379> keys a*
1) "apple"

It would be interesting to be able to do something like

cache.delete_wildcard("a*")

and other similar operations.

Cheers!

[0] http://redis.io/commands/keys

make_key does prepends odd values

I wanted to retrieve the key 'GOOG' from my redis instance:

In [51]: cache.make_key('GOOG')
Out[51]: u':1:GOOG'

why does it do this?

'module' object has no attribute 'simple_class_factory'

Unable to specify pickle protocol version

I have just finished porting our Django codebase to Python 3. Our plan is to run both Python 2 and Python 3 interpreters in separate virtualenvs on the same webserver and load balance traffice between the two of them for testing purposes.

Unfortunately, it seems they can't both use the same Redis cache. The default pickle protocol for Python 3 is version 3. This version is incompatible with all versions of Python 2. Therefore, anything created by Django while running under the Python 3 interpreter will cause an error when retrieved under Python 2.

It would be awesome if there was an extra parameter to get/set or even a configuration setting that provided a means of manually overriding the protocol version used.

RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.

Seeing this warning on django-redis-cache 1.5.3:

.../site-packages/redis_cache/utils.py:4: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
 from django.utils.importlib import import_module

UnboundLocalError: local variable 'parser_class' referenced before assignment

Hey,

I just found a minor Issue with a try/except-block in cache.py. From the following diff you can see that a failing import_module() would prevent 'parser_class' from being defined, which in turn would cause problems when raising the error in the except clause. I suggested a fix for that in the following diff.

all the best,

• John

https://gist.github.com/1345126

django.core.exceptions.ImproperlyConfigured: "server" must be an iterable or string

When I pass a value of "server" that is type unicode instead of str I get:

  File "/Users/Mike_Edwards/.virtualenvs/emojicatch/lib/python2.7/site-packages/redis_cache/backends/base.py", line 84, in get_servers
    '"server" must be an iterable or string'
django.core.exceptions.ImproperlyConfigured: "server" must be an iterable or string

More specifically, the django-environ app is what's passing the value in as unicode but it seems like that should work either way.

It looks as though a recent change to get_servers uses isinstance narrowly and only allows values that are type str:
f88925a#diff-8fc3ae3ff1cb89b2f4ba473a41661d77R78

I'm using:
python 2.7.6
Django 1.8.2
django-environ 0.3.0

UnpicklingError: unpickling stack underflow raised in get_many method

Version: 0.8.1

The get_many method is attempting to unpickle values stored as plain integers. This issue isn't triggered by using get on its own.

data = cache.get_many(['first_stage', 'second_stage', 'third_stage'])

The error is raised in redis_cache/cache.py, lines 162-165:

for key, value in zip(new_keys, results):
    if value is None:
        continue
    value = self.unpickle(value)

Local vars from the traceback:

              <tr> 
                <td>new_keys</td> 
                <td class="code"><pre>[&lt;redis_cache.cache.CacheKey object at 0x26a5590&gt;,

<redis_cache.cache.CacheKey object at 0x26a5290>,
<redis_cache.cache.CacheKey object at 0x26a5a10>]

              <tr> 
                <td>keys</td> 
                <td class="code"><pre>[&#39;first_stage&#39;, &#39;second_stage&#39;, &#39;third_stage&#39;]</pre></td> 
              </tr> 
            
              <tr> 
                <td>self</td> 
                <td class="code"><pre>&lt;redis_cache.cache.RedisCache object at 0x15ccb10&gt;</pre></td> 
              </tr> 
            
              <tr> 
                <td>results</td> 
                <td class="code"><pre>[&#39;20&#39;, &#39;40&#39;, &#39;60&#39;]</pre></td> 
              </tr> 
            
              <tr> 
                <td>value</td> 
                <td class="code"><pre>&#39;20&#39;</pre></td> 
              </tr> 
            
              <tr> 
                <td>version</td> 
                <td class="code"><pre>None</pre></td> 
              </tr> 
            
              <tr> 
                <td>recovered_data</td> 
                <td class="code"><pre>{}</pre></td> 
              </tr> 
            
              <tr> 
                <td>key</td> 
                <td class="code"><pre>&lt;redis_cache.cache.CacheKey object at 0x26a5590&gt;</pre></td> 
              </tr> 
            
              <tr> 
                <td>map_keys</td> 
                <td class="code"><pre>{&lt;redis_cache.cache.CacheKey object at 0x26a5290&gt;: &#39;second_stage&#39;,

<redis_cache.cache.CacheKey object at 0x26a5590>: 'first_stage',
<redis_cache.cache.CacheKey object at 0x26a5a10>: 'third_stage'}

          </tbody> 
        </table> 

Variable	Value

cyclic import in __init__.py

whenever redis_cache is configured as CACHE back end creates cyclic import.

Documentation: default values for configuration options

In the documentation configuration options are described, but one thing is missing: default values for each option, which are used, when option is not set.

For example it's not clear, whether any connection pool is used by default and which one? I can open sources and investigate this, but it's much more convenient to have this described in the documentation.

do not work properly with last redis-py (2.4.5)

Make it possible to connect to redis via unix socket

It's 2 times faster according to my benchmarks.

Version mismatch between PyPI and your master

I noticed the code here on github contains the cache.delete_pattern code and the code on PyPI doesn't, but both have version 0.9.5. Not a huge deal but had me confused for a while when i tried to "pip freeze > dependances" and "pip install -r dependancies" on my server.

Would you mind bumping the version number on master?