[🐛 BUG]: rabbitmq_redial error causes redial/retry logic to prematurely exit

[haydenm315]

[🐛 BUG]: rabbitmq_redial error causes redial/retry logic to prematurely exit

[haydenm315]

Thanks for the recommendations

[rustatian]

Hey @haydenm315 👋 Thanks for the report 👍
I forgot to ask you about the RR version. Could you please try the latest one (2023.3.9)?

[haydenm315]

Just tried with 2023.3.10 and there's messages which indicate an attempt to recover, however I still don't see the queue listed when doing ./rr workers, after bouncing the rabbitmq service to simulate an upgrade or service interruption. Looks like there's an attempt to re-establish the connection now.

Upon making rabbitmq unavailable, roadrunner logs the following.

roadrunner-1  | {"level":"DEBUG","ts":"2024-02-07T18:14:21+0000","logger":"amqp        ","msg":"delivery channel was closed, leaving the rabbit listener"}
roadrunner-1  | {"level":"DEBUG","ts":"2024-02-07T18:14:21+0000","logger":"amqp        ","msg":"number of listeners","listeners":0}
roadrunner-1  | {"level":"DEBUG","ts":"2024-02-07T18:14:21+0000","logger":"amqp        ","msg":"exited from redialer"}
roadrunner-1  | {"level":"ERROR","ts":"2024-02-07T18:14:21+0000","logger":"amqp        ","msg":"amqp connection closed","error":"Exception (504) Reason: \"channel/connection is not open\""}
roadrunner-1  | {"level":"ERROR","ts":"2024-02-07T18:14:21+0000","logger":"amqp        ","msg":"pipeline connection was closed, redialing","error":"Exception (320) Reason: \"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'\"","pipeline":"snapshot","driver":"amqp","start":"2024-02-07T18:14:21+0000"}

After making rabbitmq available again, roadrunner logs the following

roadrunner-1  | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"rabbitmq dial was succeed. trying to redeclare queues and subscribers"}
roadrunner-1  | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"queues and subscribers was redeclared successfully"}
roadrunner-1  | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"connection was successfully restored","pipeline":"snapshot","driver":"amqp","start":"2024-02-07T18:14:21+0000","elapsed":"27.519989644s"}
roadrunner-1  | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"redialer restarted"}
roadrunner-1  | {"level":"DEBUG","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"exited from redialer"}
roadrunner-1  | {"level":"ERROR","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"pipeline connection was closed, redialing","error":"Exception (320) Reason: \"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'\"","pipeline":"snapshot","driver":"amqp","start":"2024-02-07T18:14:49+0000"}
roadrunner-1  | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"rabbitmq dial was succeed. trying to redeclare queues and subscribers"}
roadrunner-1  | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp        ","msg":"rabbitmq dial was succeed. trying to redeclare queues and subscribers"}

[rustatian]

Yeah, so, that means that RR successfully redialed.

@haydenm315 Is that what you wanted to see, am I right?

[haydenm315]

Seems closer than the older release we were using. Still requires a restart of roadrunner to get it to accept jobs and have a ready status. Unless I restart roadrunner after rabbitmq comes back, I don't see jobs like below in the rr workers output.

Jobs of [jobs]:
+--------+----------+--------+---------+--------+---------+----------+
| STATUS | PIPELINE | DRIVER |  QUEUE  | ACTIVE | DELAYED | RESERVED |
+--------+----------+--------+---------+--------+---------+----------+
| READY  | snapshot | amqp   | default | 0      | 0       | 0        |
+--------+----------+--------+---------+--------+---------+----------+

Maybe something's not quite right with "trying to redeclare queues and subscribers".

[rustatian]

I'm not sure, because I have a test especially for the case when rabbitmq is down. Have you tried to push the job after redial?

[rustatian]

@haydenm315 In your configuration I also see, that you're consuming pipeline with the name widgets, but you don't have such a pipeline in your pipelines configuration.

[haydenm315]

I renamed the queue in the previous example. It's not a queue naming issue. Jobs are getting pushed because when I restart the roadrunner server the number of Execs goes up.

[rustatian]

But, it's not a queue, it's a pipeline. Could you please share the latest configuration you have?

[haydenm315]

Here the .rr.yaml file These are the env var values in the cfg file

AMQP_REDIAL_TIMEOUT=99999999
ROADRUNNER_NUM_POLLERS=200
ROADRUNNER_NUM_WORKERS=50
ROADRUNNER_PREFETCH=200

rr.yml.txt

[rustatian]

Thanks 👍
Few notes regarding your configuration:

1. The number of pollers should not be significantly bigger than the number of workers. Poller is a thread inside RR which tries to obtain a job from the internal priority queue. More pollers -> more memory will be used for the thread stack, etc. Try 60 pollers and compare with the current value (200).
2. There is no reload plugin anymore since 2023.1.0 version.

[rustatian]

Hey @haydenm315 👋

I double-checked that behavior. Let me summarize what I did:

1. Started RR, with jobs configuration and 1 pipeline.
2. Shut down RabbitMQ docker. RR started the redial process with exponential backoff.
3. I used a huge redial timeout. Waited for a minute.
4. Then I started RabbitMQ docker.
5. RoadRunner successfully reconnected to it.

Keep in mind, that RR uses an exponential backoff mechanism to redial. It won't reconnect immediately, but after some exponential step. For example: first redial will happen after 1sec after loosing a connection, next - 2s -> 5s -> 15s -> 25s ...

[rustatian]

I also noticed, that your amqp configuration contains some options from the other drivers, e.g.: reserve_timeout is from the beanstalk driver.

[rustatian]

Closing as answered. You are welcome to comment here if you still have questions 😃