Configurable tcp socket keepalive on Redis connection - improved resiliency

[edevil]

Configurable tcp socket keepalive on Redis connection - improved resiliency

[edevil]

Upon further investigation the asgi_redis module is using 5s timeouts on BLPOP messages, which should keep the connection healthy, however I've seen cases where the connection is dropped and the BLPOP response is never received. In this case the worker would wait forever. Using the socket_timeout option to the redis-py connection solves this issue, so I don't know if anything else is needed...

[andrewgodwin]

Yes, the pops are refreshed quite often, but the underlying socket connection is held open, and that might be what's dying. How did you add socket_timeout - is it something we should support?

[edevil]

In the layer config, I used a connection URL to configure a host. Here we can pass query parameters such as socket_timeout=10 to redis-py. I don't think anything else needs to be done since this is backend specific.

[andrewgodwin]

Alright, lovely! Hopefully someone else can find that info useful too.

[edevil]

I've been running more tests and, although the socket_timeout solves the issue of the worker waiting forever, it seems that being able to configure the socket_keepalive_options parameter of redis-py would be helpful. Otherwise when an idle connection is dropped there's still a noticeable lag while waiting for the timeout on the next call, whereas if the dropped connection was detected earlier the experience would be better.

[andrewgodwin]

Alright - do you want to open a new ticket with the details of what kind of configuration options would be useful? I have some time tomorrow I can use to work those in (it would be nice to expose socket_timeout more directly too, potentially, and document it)