Error when using with redis cluster

[docker-library]

Error when using with redis cluster

[jchanam]

In order to add you info, what we get through logs is the following:

[WARNING] 214/091626 (1) : Server redis/redis_01 is DOWN, reason: Layer7 timeout, info: " at step 4 of tcp-check (expect string 'role:master')", check duration: 1001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 214/091626 (1) : Server redis/redis_02 is DOWN, reason: Layer7 timeout, info: " at step 4 of tcp-check (expect string 'role:master')", check duration: 1001ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.

ps. The redis_03 is the actual master.

Thanks!

[ebuildy]

Can you explain a bit more your infrastructure? Such as number of Docker hosts, private/public networks ....

[jchanam]

All the dockers are running inside the same VM's. Docker is only used on the VM_X (drawn below). The PHP-FPM connects to the HAProxy in order to connect to the redis master.

We've got:

VM_X
+---------------------------------------------+    VM_Y[1|2|3]
| +---------+    +-----------+    +---------+ |    +---------------+
| | Varnish | -> | Nginx+PHP | -> | HAProxy | | -> | Redis Cluster |
| +---------+    +-----------+    +---------+ |    +---------------+
+---------------------------------------------+

What we've found out, is that the dockers maintain the connections open, and, because of there are "different" machines, does not find out the HAProxy has been restarted.

Is there any option to make the dockers restart on cascade?
Is there any way to make the dockers know that other one has been restarted?

Thanks!

[yosifkit]

So the HAProxy container is restarted, getting a new IP, and the PHP container has a --link my-haproxy:custom-haproxy to it? Currently there is not a simple way to cascade restarts, but, as long as it is the same container (ie just a docker stop and start), the PHP container would get an updated entry in /etc/hosts with the new IP. So now the PHP container can hit custom-haproxy or whatever the link name is.

I am not sure I am even debugging in the right area, but maybe this helps 😕.

[jchanam]

Hi!

Yes, I've got the containers linked as you've said.

What I'm trying to do is a script that, via docker api, checks the updated time and, based on setted dependencies, restart the dockers.

I don't know if there's an easier way of doing this.

Apart from this, what is strange, is that when the haproxy is restarted, he php can get to redis, but other than the master node. I haven't been able to found out why by now...

Thank you for all your efforts!

[freiit]

with docker >=1.9 there is an easier way. Put all containers with 'docker network ...' in a local network, too, so they can find each other always with their names. If you replace a container, just give it the same name and you don't need to replace cascading containers.

[tianon]

I agree -- I think the new Docker Networks feature should help mitigate this particular issue. Regardless, this appears to be an issue specific to Docker's routing, not something inherent to the image. If that's not the case, I think a new issue with more details explaining what the image is doing wrong and hopefully clues for how we can fix it would be appreciated! 👍