squigg/azure-queue-laravel

PHP Laravel Queue Driver package for Microsoft Azure Storage Queues

azure-queue-laravel

PHP Laravel Queue Driver package to support Microsoft Azure Storage Queues

Prerequisites

• Laravel 5.2 - 8.x (not tested on previous versions)
• PHP 5.6+ for Laravel 5.2+
• PHP 7+ for Laravel 5.5+
• PHP 7.1+ for Laravel 5.6+
• PHP 7.2+ for Laravel 6+
• PHP 7.3+ for Laravel 8+
• PHP 8.0+ for Laravel 9+
• PHP 8.1+ for Laravel 10+
• PHP 8.1+ for Laravel 11+
• Microsoft Azure Storage Account and Storage Account Key
• Queue container created through Azure Portal or via Azure CLI or PowerShell

Installation

Install using composer

You can find this library on Packagist.

Require this package in your composer.json. The version numbers will follow Laravel.

Laravel 11.x

"squigg/azure-queue-laravel": "^11.0"
composer require squigg/azure-queue-laravel:^11.0

Laravel 10.x

"squigg/azure-queue-laravel": "^10.0"
composer require squigg/azure-queue-laravel:^10.0

Laravel 9.x

"squigg/azure-queue-laravel": "^9.0"
composer require squigg/azure-queue-laravel:^9.0

Laravel 8.x

"squigg/azure-queue-laravel": "^8.0"
composer require squigg/azure-queue-laravel:^8.0

Laravel 7.x

"squigg/azure-queue-laravel": "^7.0"
composer require squigg/azure-queue-laravel:^7.0

Laravel 6.x

"squigg/azure-queue-laravel": "^6.0"
composer require squigg/azure-queue-laravel:^6.0

Laravel 5.8.x

"squigg/azure-queue-laravel": "5.8.*"
composer require squigg/azure-queue-laravel:5.8.*

Laravel 5.7.x

"squigg/azure-queue-laravel": "5.7.*"
composer require squigg/azure-queue-laravel:5.7.*

Laravel 5.6.x

"squigg/azure-queue-laravel": "5.6.*"
composer require squigg/azure-queue-laravel:5.6.*

Laravel 5.5.x

"squigg/azure-queue-laravel": "5.5.*"
composer require squigg/azure-queue-laravel:5.5.*

Laravel 5.4.x

"squigg/azure-queue-laravel": "5.4.*"
composer require squigg/azure-queue-laravel:5.4.*

Laravel 5.3.x

"squigg/azure-queue-laravel": "5.3.*"
composer require squigg/azure-queue-laravel:5.3.*

Laravel 5.2.x

"squigg/azure-queue-laravel": "5.2.*"
composer require squigg/azure-queue-laravel:5.2.* 

For versions 5.3 and older only

Add the following pear repository in your composer.json file required for the Microsoft Azure SDK (v5.4+ uses the microsoft/azure-storage package instead, and v5.6+ uses microsoft/azure-storage-queue):

"repositories": [
    {
        "type": "pear",
        "url": "http://pear.php.net"
    }
],

Update Composer dependencies

composer update

Configuration

Add Provider

If you are not using Laravel auto package discovery, add the ServiceProvider to your providers array in config/app.php:

'Squigg\AzureQueueLaravel\AzureQueueServiceProvider',

For Lumen (5.x) you will need to add the provider to bootstrap/app.php:

    $app->register(Squigg\AzureQueueLaravel\AzureQueueServiceProvider::class);

Add Azure queue configuration

Add the following to the connections array in config/queue.php, and fill out your own connection data from the Azure Management portal:

'azure' => [
    'driver'        => 'azure',                             // Leave this as-is
    'protocol'      => 'https',                             // https or http
    'accountname'   => env('AZURE_QUEUE_STORAGE_NAME'),     // Azure storage account name
    'key'           => env('AZURE_QUEUE_KEY'),              // Access key for storage account
    'queue'         => env('AZURE_QUEUE_NAME'),             // Queue container name
    'timeout'       => 60,                                  // Seconds before a job is released back to the queue
    'endpoint'      => env('AZURE_QUEUE_ENDPOINTSUFFIX'),   // Optional endpoint suffix if different from core.windows.net
    'queue_endpoint'=> env('AZURE_QUEUE_ENDPOINT'),         // Optional endpoint for custom addresses like http://localhost/my_storage_name
],

Add environment variables into your .env file to set the above configuration parameters:

AZURE_QUEUE_STORAGE_NAME=xxx
AZURE_QUEUE_KEY=xxx
AZURE_QUEUE_NAME=xxx
AZURE_QUEUE_ENDPOINTSUFFIX=xxx
AZURE_QUEUE_ENDPOINT=xxx

Set the default Laravel queue

Update the default queue used by Laravel by setting the QUEUE_CONNECTION value in your .env file to azure.

QUEUE_CONNECTION=azure

This setting is QUEUE_DRIVER in older versions of Laravel.

Usage

Use the normal Laravel Queue functionality as per the documentation.

Remember to update the default queue by setting the QUEUE_DRIVER value in your .env file to azure.

Changelog

2023-03-30 - V11.0 - Support for Laravel 11.x

2023-03-17 - V10.0 - Support for Laravel 10.x

2022-03-17 - V9.0 - Support for Laravel 9.x

2021-10-16 - V8.1 - Support for PHP 8

2020-09-19 - V8.0 - Support for Laravel 8.x (composer dependency and test refactoring only)

2020-06-04 - V7.0 - Support for Laravel 7.x (composer dependency and test refactoring only)

2020-06-04 - V6.0 - Support for Laravel 6.x (composer dependency changes only)

2019-07-13 - V5.8 - Support for Laravel 5.8 (composer dependency and test changes only)

2019-07-13 - V5.7.1 - Fix invalid signature on call to base Laravel Queue method

2018-09-04 - V5.7 - Support for Laravel 5.7 (composer dependency changes only)

2018-02-07 - V5.6 - Switch to GA version of Microsoft Azure Storage PHP API. Support Laravel 5.6 (composer.json changes only). Update dev dependencies to latest versions.

2017-09-11 - V5.5 - Support Laravel 5.5 and PHP7+ only. Update Azure Storage API to 0.18

2017-09-11 - V5.4 - Update Azure Storage API to 0.15 (no breaking changes)

License

Released under the MIT License. Based on Alex Bouma's Laravel 4 package, updated for Laravel 5.

azure-queue-laravel's Issues

Unable to install on Laravel 5.5.*

Hi!,

I'm having problems when installing your great package on Laravel 5.5.25.

Here's the error message I get when trying to install the package:

composer require "squigg/azure-queue-laravel:5.5.*"
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - squigg/azure-queue-laravel v5.5.1 requires microsoft/azure-storage ~0.18.0 -> satisfiable by microsoft/azure-storage[v0.18.0] but these conflict with your requirements or minimum-stability.
    - squigg/azure-queue-laravel v5.5.0 requires microsoft/azure-storage ~0.18.0 -> satisfiable by microsoft/azure-storage[v0.18.0] but these conflict with your requirements or minimum-stability.
    - Installation request for squigg/azure-queue-laravel 5.5.* -> satisfiable by squigg/azure-queue-laravel[v5.5.0, v5.5.1].


Installation failed, reverting ./composer.json to its original content.

Here's my queue.php configuration file contents:

<?php

return [
    /*
    |--------------------------------------------------------------------------
    | Default Queue Driver
    |--------------------------------------------------------------------------
    |
    | Laravel's queue API supports an assortment of back-ends via a single
    | API, giving you convenient access to each back-end using the same
    | syntax for each one. Here you may set the default queue driver.
    |
    | Supported: "sync", "database", "beanstalkd", "sqs", "redis", "null"
    |
    */

    'default' => env('QUEUE_DRIVER', 'sync'),

    /*
    |--------------------------------------------------------------------------
    | Queue Connections
    |--------------------------------------------------------------------------
    |
    | Here you may configure the connection information for each server that
    | is used by your application. A default configuration has been added
    | for each back-end shipped with Laravel. You are free to add more.
    |
    */

    'connections' => [
        'sync' => [
            'driver' => 'sync',
        ],

        'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'beanstalkd' => [
            'driver' => 'beanstalkd',
            'host' => 'localhost',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'sqs' => [
            'driver' => 'sqs',
            'key' => 'your-public-key',
            'secret' => 'your-secret-key',
            'prefix' => 'https://sqs.us-east-1.amazonaws.com/your-account-id',
            'queue' => 'your-queue-name',
            'region' => 'us-east-1',
        ],

        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'azure' => [
            'driver' => 'azure',
            'protocol' => 'https',
            'accountname' => env('AZURE_STORAGE_NAME'),
            'key' => env('AZURE_STORAGE_KEY'),
            'queue' => env('AZURE_QUEUE_NAME', 'default'),
            'timeout' => 60,
            'retry_after' => 310,
        ],
    ],

    /*
    |--------------------------------------------------------------------------
    | Failed Queue Jobs
    |--------------------------------------------------------------------------
    |
    | These options configure the behavior of failed queue job logging so you
    | can control which database and table are used to store the jobs that
    | have failed. You may change them to any database / table you wish.
    |
    */

    'failed' => [
        'database' => env('DB_CONNECTION', 'mysql'),
        'table' => 'failed_jobs',
    ],
];

Software versions
This Package:
Laravel: 5.5.25
PHP: 7.1.12
Operating System: MacOS

Plan for Azure SDK EOL

What's your plan given that MSFT have announced that they're retiring the PHP SDK for Storage?

Laravel 8 Support

Can we get a release tagged for Laravel 8? I believe through my testing the package should work fine with the Laravel 8 queue.

Laravel 6 support

Summary of issue
No support for Laravel 6.

This Package: 5.8
Laravel: 6.5
PHP: 7.1 / 7.2 / 7.3
Operating System: Linux

Is Laravel 5.4 planned to be supported soon?

Hi!

I want to say thanks for been created this package.

Also I want to ask you if there will be a release for Laravel 5.4?

Thanks!

Laravel 9

Hi,

Are there any plans to migrate to Laravel 9?

Thanks.

QueueEndpoint argument support

Hey! Not really an issue here, but more of a feature question.

On my current project I'm working with a local queue which runs through Azurite. To connect to the instance it uses a link of the following format: http://127.0.0.1:10001/<storage_name>.
Therefore it's not compatible with the package as is, since there is only the "Endpoint Suffix" option available, which does not work with this URL format.

It's not really documented from what I saw, but apparently there is an option called "QueueEndpoint", which can take in a custom URL, and does support the format above. I did test it with the aforementioned project, and it does indeed work just as expected.

So my question is, would it be useful if I submit a PR with a new config argument that has that option available?

References:

This argument in the query string in the official package: here

This argument used by the storage emulator: here

And if you do agree that this addition would be useful, how would you like the config to be named? The endpoint setting is already occupied therefore I'm thinking of "endpoint_url" or "endpoint_custom" of sorts.

Thank you for reading this and really curious to know your opinion on this addition ;)

Will try to make a PR soon so to have something to click through.

Laravel 10

Hi,

Are there any plans to migrate to Laravel 10?

Thanks.

Laravel 5.7 not compatible

Summary of issue

createPayload method call not compatible with Laravel 5.7 createPayload signature. Data not passed to job.

file: squigg/azure-queue-laravel/src/AzureQueue.php line 62

change

public function push($job, $data = '', $queue = null)
{
    $this->pushRaw($this->createPayload($job, $data), $queue);
}

to

public function push($job, $data = '', $queue = null)
{
    $this->pushRaw($this->createPayload($job, $queue, $data), $queue);
}

also shoud be fixed in "later" function like this:

    public function later($delay, $job, $data = '', $queue = null)
    {
        $payload = $this->createPayload($job, $queue, $data);

        $options = new CreateMessageOptions();
        $options->setVisibilityTimeoutInSeconds($this->secondsUntil($delay));

        $this->azure->createMessage($this->getQueue($queue), $payload, $options);
    }

Software versions
This Package:
Laravel: 5.7
PHP: 7.2
Operating System: Windows / Linux / MacOS

PHP 8 Support

Summary of issue
We would like to use this package in our Laravel application running PHP 8.

Software versions
Laravel: 8.57.0
PHP: 8.0.10

MalformedToken

I'm getting 401 Malformed Token when trying to send a message to the queue, any suggestion where i should start looking for?

MicrosoftAzure\Storage\Common\Exceptions\ServiceException: Fail: Code: 401 Value: Unauthorized details (if any): 401MalformedToken: The credentials contained in the authorization header are not in the WRAP format. TrackingId:0a964ef4-6456-48f3-9596-c8bds1da9d23f_G17, SystemTracker:queue-name.servicebus.windows.net:myqueue, Timestamp:2020-11-19T16:59:41. in /application/vendor/microsoft/azure-storage-common/src/Common/Internal/ServiceRestProxy.php:496

**Copy of your config/queue.php file **
'azure' => [
'driver' => 'azure',
'protocol' => 'https',
'accountname' => env('AZURE_QUEUE_STORAGE_NAME'),
'key' => env('AZURE_QUEUE_KEY'),
'queue' => "myqueue",
'timeout' => 60,
'endpoint' => env('AZURE_QUEUE_ENDPOINTSUFFIX'),
],

Software versions
This Package:
Laravel: ^6.0
Lumen: ^6.2
PHP: 7.3
Operating System: Linux

Any other information

Release for PHP 8.0

Summary of issue
Is there a plan to release a new version with the updated composer.json support for PHP 8.0?

We're currently having to pin our composer version to dev-master#50b0df8 and would much prefer having an actual release to use.

How to receive messages from service bus?

How do I receive messages sent to Azure Service Bus? Is it through jobs, events or listeners?
And, how to send message topic using Job?

Timeout issues - finding their way back to the application

Summary of issue
If the queue connection times out, due to being connected for some time without being used, then no attempt is made to reconnect if sending a message fails. With laravel, if a database connection times out, then laravel attempts to detect this, reconnects, then retries the query one more time.

The complete error message, including file & line numbers

I have a model observer that watches for certain events, and dispatches a job to handle certain types of event. Occasionally updating the model will throw this error (returned by eloquent->save() when the observer is triggered:

cURL error 7: Failed connect to myresourcegroup.queue.core.windows.net:443; Connection timed out (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

**Copy of your config/queue.php file **

Software versions
This Package: 5.5.2
Laravel: 5.6
PHP: 7.2
Operating System: Windows / Linux / MacOS: Linux

Any other information

While the connections are up, everything works fine, but it is the timeouts that are the problem, both when sending on a timed out connection (expecting an auto-reconnect) and when listening on a long-running process.

My queue workers are restarted every 10 minutes to try to avoid the queue listener timing out. When it does time out, the MS Azure library either throws an exception or passes a message with the timeout error details (not sure which, but I think it is the latter). When this happens, the listener is sat listening to a dead connection, and so the queues are not served. That timeout should also be caught if possible and a reconnect done. The queue sender can afford to allow the connection to drop, so long as it can restart it when needed (like Laravel database connections). But the queue listener must always make sure its connection is up, running, and open.