odolbeau/rabbit-mq-admin-toolkit

Manage a RabbitMQ cluster easily.

RabbitMQ Admin Toolkit

Installation

The recommended way to install RabbitMQ Tools is through Composer. Require the odolbeau/rabbit-mq-admin-toolkit package:

$ composer require odolbeau/rabbit-mq-admin-toolkit

Usage

You can create / update vhosts with the following command:

vendor/bin/rabbit vhost:mapping:create conf/vhost/events.yml

You can change all connection informations with options. Launch ./console vhost:create -h to have more informations.

You can launch the vhost creation even if the vhost already exist. Nothing will be deleted (and it will not impact workers).

Configuration

You can use the followings parameters for configuring an exchange:

• with dl: if set to true, all queues in the current vhost will be automatically configured to have a dl (with name: {queueName}_dl). Of course, the exchange dl will be created.
• with_unroutable: is set to true, an unroutable exchange will be created and all others ones will be configured to move unroutable messages to this one. The unroutable exchange is a fanout exchange and a unroutable queue is bind on it.

Example

my_vhost_name:

    permissions:
        my_user:
            configure: amq\.gen.*
            read: .*
            write: .*
            
    parameters:
        with_dl: true # If true, all queues will have a dl and the corresponding mapping with the exchange "dl"
        with_unroutable: true # If true, all exchange will be declared with an unroutable config

    exchanges:
        my_exchange:
            type: direct
            durable: true
            with_unroutable: true #if true, unroutable exchange will be created (if not already set as global parameter)

        my_exchange_headers:
            type: headers
            durable: true

    queues:
        my_queue:
            durable: true
            delay: 5000 #create delayed message queue (value is in milliseconds)
            bindings:
                - 
                    exchange: my_exchange
                    routing_key: my_routing_key
                - 
                    exchange: my_exchange
                    routing_key: other_routing_key
                    
        another_queue:
            durable: true
            with_dl: false
            retries: [25, 125, 625]
            bindings:
                - 
                    exchange: my_exchange_headers
                    x-match: all
                    matches: {header_name: value, other_header_name: some_value}

License

This project is released under the MIT License. See the bundled LICENSE file for details.

Changelog

BC breaks between 1.x and 2.0.x

• Short binding syntax is no more supported.

  # old syntax
  queues:
    my_queue:
        bindings:
            - my_exchange:my_routing_key

must be replaced by

  # new syntax
  queues:
    my_queue:
        bindings:
            - 
                exchange: my_exchange
                routing_key: my_routing_key

rabbit-mq-admin-toolkit's Issues

Blocks installation of swarrot 4.x

Hello,

v5.0 version of the package prevents installation of swarrot/swarrot ^4.0 , maybe you could update version requirements to something like swarrot/swarrot ^3.0|^4.0 ?

Warning when no queues are defined

rabbit-mq-admin-toolkit/src/Bab/RabbitMq/Specification/DeadLetterExchangeCanBeCreated.php

Line 13 in beeb950

if (!isset($config['queues']) || 0 === count($config['queues'])) {

In DeadLetterExchangeCanBeCreated.php line 13:
                                                                                       
  Warning: count(): Parameter must be an array or an object that implements Countable

Naming pattern

Hi

On the beginning, I want to thanks for this great library.

I would like to know your opinion about what you think about adding to the configuration the possibility of creating name patterns for the queue dl and retry
curently thay are hardcoded to:

$retryQueueName = $name.'_retry_'.$retries[$i];
$this->createQueue($name.'_dl', array(
$this->createQueue($name.'_delay_'.$delay, array(

maybe we can create config value that can be overridden in a config file for this hardcoded string?
for example

parameters:
      dl_queue_patern: '%%queue_name%%_dl'
      retry_queue_patern: '%%queue_name%%_retry_%%retries%%'
      retry_queue_patern: '%%queue_name%%_delay_%%delay%%'

what do you think about that?

update composer dependecy

I cannot use it with latest version of swarrot.

Change the way retries works

Currently when you specify the property retries for a given queue, it will create 1 retry queue per entries in the array.
For example, if you have:

queues:
  my_queue:
    durable: true
    retries: [30, 100, 100]
    bindings: my_exchange:my_routing_key

You will end with the following queues:

• my_queue
• my_queue_retry_1
• my_queue_retry_2
• my_queue_retry_3

Both queues my_queue_retry_2 & my_queue_retry_3 will have the same ttl.
If you want to have several retries with all the same values, that's a lot of useless queues.

That's why I propose the following change:

• all retry queues will be suffixed by the ttl used instead of the retry attempt number (in our case: my_queue_retry_30, my_queue_retry_100). This is a BC break but this will reduce the number of queues created in some use cases.
• we still have only one retry exchange
• we still have the same amount of bindings (in our case: retry:my_queue_retry_1 => my_queue_retry_30, retry:my_queue_retry_2 => my_queue_retry_100, retry:my_queue_retry_3 => my_queue_retry_100)

WDYT @stof @Shine-neko @antoox @iamluc @joelwurtz @Domisys ?

Please give me your opinion as this BC break could lead to a lot of manual operations during migrations (delete old bindings).

Allow to configure the vhost in the connection settings too

Currently, most commands expect a connection name and a vhost. It would be great to be able to specify the vhost in the connection config file too (making the command-line argument required only when the config file does not configure it). This is especially useful when your connection credentials give you access to only 1 vhost (this is what happens for hosted RabbitMQ providers), where using a different vhost requires using a different connection anyway.

This may require a BC break to switch the vhost from an argument to an option though.

Releases should have a changelog

This would make it easier to know what is changed between versions

Messages are "retried" only once despite my configuration (retries: 3)

Hi !

It seems there is (or I have) a problem with retry (more than once).

Config

I use these libs and versions:

"swarrot/swarrot-bundle": "1.4.1",
"php-amqplib/php-amqplib": "^2.6",
"odolbeau/rabbit-mq-admin-toolkit": "^3.2"

I have this configuration for odolbeau/rabbit-mq-admin-toolkit:

'/':
    parameters:
        with_dl: false
        with_unroutable: false

    exchanges:
        benchmark_test_1:
            type: direct
            durable: true

    queues:
        benchmark_test_1:
            durable: true
            retries: [5, 25, 100]
            bindings:
                - exchange: benchmark_test_1
                  routing_key: benchmark_test_1

It gives me this output

With DL: false
With Unroutable: false
Create exchange benchmark_test_1
Create exchange dl
Create queue benchmark_test_1
Create queue benchmark_test_1_dl
Create binding between exchange dl and queue benchmark_test_1_dl (with routing_key: benchmark_test_1)
Create exchange retry
Create queue benchmark_test_1_retry_1
Create binding between exchange retry and queue benchmark_test_1_retry_1 (with routing_key: benchmark_test_1_retry_1)
Create binding between exchange retry and queue benchmark_test_1 (with routing_key: benchmark_test_1)
Create exchange retry
Create queue benchmark_test_1_retry_2
Create binding between exchange retry and queue benchmark_test_1_retry_2 (with routing_key: benchmark_test_1_retry_2)
Create binding between exchange retry and queue benchmark_test_1 (with routing_key: benchmark_test_1)
Create exchange retry
Create queue benchmark_test_1_retry_3
Create binding between exchange retry and queue benchmark_test_1_retry_3 (with routing_key: benchmark_test_1_retry_3)
Create binding between exchange retry and queue benchmark_test_1 (with routing_key: benchmark_test_1)
Create binding between exchange benchmark_test_1 and queue benchmark_test_1 (with routing_key: benchmark_test_1)

I my swarrot bundle configuration I have:

swarrot:
    provider: amqp_lib
    default_connection: rabbitmq
    default_command: swarrot.command.base
    connections:
        rabbitmq:
            host: rabbitmq
            port: 5672 # 15672
            login: rbu
            password: rbp
            vhost: '/'
    consumers:
        test_consume_quickly: # name of the consumer
            processor: processor.test_consume_quickly # name of the service
            extras:
                poll_interval: 500000
                requeue_on_error: false
            middleware_stack:
                - configurator: swarrot.processor.signal_handler
                - configurator: swarrot.processor.max_messages
                  extras:
                    max_messages: 10
                - configurator: swarrot.processor.max_execution_time
                  extras:
                    max_execution_time: 30
                - configurator: swarrot.processor.memory_limit
                  extras:
                      memory_limit: 50
                - configurator: swarrot.processor.exception_catcher
                - configurator: swarrot.processor.ack
                - configurator: swarrot.processor.retry
                  extras:
                    retry_exchange: 'retry'
                    retry_attempts: 3
                    retry_routing_key_pattern: 'benchmark_test_1_retry_%%attempt%%'


    messages_types:
        my_publisher:
            connection: rabbitmq # use the default connection by default
            exchange: benchmark_test_1
            routing_key: benchmark_test_1

Problem

My consumer raise an expcetion.
The message goes to queue benchmark_test_1_retry_1
The message is sent back into benchmark_test_1
My consumer raise an expcetion.
And..... the message goes to queue benchmark_test_1_dl

But should go into queue benchmark_test_1_retry_2 then benchmark_test_1_retry_3

I don't understand why...
Any idea what going on ?

Thanks !

Exchange name

Hello, just a question :
Can we rename the retry and dl exchange in the yaml options ?

Url encoding of vhost

I met an issue with the command to create queues. I have two vhosts in my application : / and /module. Then I made the configuration file with these two vhosts, here a sample :

/:
    exchanges:
        direct:
            type: direct
            durable: true
    queues:
        jobs_image:
            durable: true
            bindings:
                -
                    exchange: direct
                    routing_key: jobs_image

/module:
    exchanges:
        direct:
            type: direct
            durable: true
    queues:
        jobs_image:
            durable: true
            bindings:
                -
                    exchange: direct
                    routing_key: jobs_image

I realized the command is not able to manage 2 vhosts in the same file (unfortunately...), anyway, I split this configuration into two files.

But the execution of the queues creation command with the vhost /module didn't work because of the leading slash. I was able to make work the command naming vhost %2fmodule. But there is no need to url rewrite the default vhost thanks to this code : https://github.com/odolbeau/rabbit-mq-admin-toolkit/blob/master/src/Bab/RabbitMq/VhostManager.php#L21

Although leading slash is not required in the rabbitmq vhost name, many usage include it. So I ask why there is no url rewriting on every vhost name or is it intentional as the code seems to mean ?

Thanks for your great shared work.