Default settings leaking TCP/IP connections?

[jazzband]

Default settings leaking TCP/IP connections?

[miohtama]

I just confirmed restarting UWSGI indeed killed the connections and Redis connections fell to < 20.

[niwinz]

Hi @miohtama

Can you provide more details like, versions, uwsgi start params, how you obtain a cache instance?

Also, if you are using the last django-redis version (3.6.x) , try with same environment but downgrading django-redis to 3.5.2. In 3.6.x branch new connection management is introduced and can contain some regressions in this part. Tomorrow I'll try reproduce this behavior and fix it.

Thanks for report this error.

[miohtama]

Thanks. I'll try to diagnose the problem further tonight.

Here is already some further information:

• If I set max_connections in CACHES settings, django-redis will hit this settings quite fast, starting to give out errors

-> It is not properly reusing the connection

-> Maybe there is something in UWSGI or the application which prevents the connection reusing

[niwinz]

Try disable threads on uwsgi, the default connection pool seems not thread
safe.

Andrey.

Sent from my Nexus 4
On Apr 24, 2014 9:11 AM, "Mikko Ohtamaa" [email protected] wrote:

Thanks. I'll try to diagnose the problem further tonight.

Here is already some further information:

• If I set max_connections in CACHES settings, django-redis will hit
  this settings quite fast, starting to give out errors

-> It is not properly reusing the connection

-> Maybe there is something in UWSGI or the application which prevents the
connection reusing

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/76#issuecomment-41248890
.

[niwinz]

Hi again!

I have created test project, with simple view, that obtains a value from cache and increments it and sets the value. Additionally returns a value in a response.

from django.core.cache import cache                                                

def handler1(request):                                                             
    value = cache.get("foo1", 1)                                                   
    value = value + 1                                                              
    cache.set("foo1", value, timeout=None)                                         
    return HttpResponse("Hello World {}".format(value))                            

I have started uwsgi with threads and processes, and with last version of django-redis (3.6.1) with default configuration.

This is a uwsgi start command that I have used:

uwsgi -H /home/niwi/.virtualenvs/djredis-pool --http-socket 127.0.0.1:8080 --wsgi-file redis_pool/wsgi.py --master --processes 4 --threads 4

This is a command used for obtain a number of connections:

redis-cli info |grep connected_clients

And I have used ab tool for simulate multiple concurrent access to the url: ab -n 9000 -c 100 http://127.0.0.1:8080/handler1

The results are as expected: in high load, with 4 threads and 4 processes, I found 16 open connections in the pool.

At this momment I need much more information like python environment versions (result from pip freeze) , python version, uwsgi version, and cache instance usage examples for try reproduce the error. Because, with small load tests and default config using django 1.6 and uwsgi the last current version, the connection management seems working properly.

[miohtama]

I think I located whas was special in this configuration: it is using Redis pubsub, using the client connection pool from django-redis (cache.raw_client). Apparently the issue might be little bit deeper in redis-py, outside django-redis scope.

However when I get at the bottom of it, I'll drop few warning words on the documentation.

[miohtama]

And thanks for the all help!

[niwinz]

No, thanks you for investigate! ;)