foxxmd/laravel-elasticbeanstalk-queue-worker

Laravel 5 - 9.x Queue Worker for Elastic Beanstalk

Use your Laravel application as a worker to consume queues on AWS Elasticbeanstalk

Laravel provides a wonderful array of drivers for consuming queues within your application as well as some documentation on how to manage your application with Supervisord when it is acting as a worker.

Unfortunately that's where the documentation ends. There is no guidance on how to manage multiple workers from a devops context which is a huge bummer. But don't worry fam I've got your covered.

This package enables your Laravel application to manage itself, as a worker, in an AWS Elasticbeanstalk environment.

It provides these features:

• Automated installation of supervisor on first-time deployment
• Automatic updating of supervisor configuration upon deployment
• Two supervisor configuration deployment options:
  • Parsing of EB environmental variables to generate supervisor config
  • Or using a pre-built supervisor config supplied in project

Amazon Linux 1 deprecation

Amazon Linux 1 (AL1) is going to be unsupported soon, it is advised to migrate to use Amazon Linux 2 (AL2) https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.migration-al.html Starts from this release will only support AL2, please use previous releases for use in AL1

Let's get down to business

Installation

Require this package

composer require "foxxmd/laravel-elasticbeanstalk-queue-worker"

or for Amazon Linux 1,

composer require "foxxmd/laravel-elasticbeanstalk-queue-worker@^0.3"

After installing the package you can either:

Add the ServiceProvider to the providers array in config/app.php (for Laravel 5.4 or lower)

FoxxMD\LaravelElasticBeanstalkQueueWorker\ElasticBeanstalkQueueWorkerProvider::class

Then, publish using artisan

php artisan vendor:publish --tag=ebworker

OR

Copy everything from src/.ebextensions into your own .ebextensions folder and everything from src/.platform into your own .platform folder manually

Note: This library only consists of the EB deploy steps -- the provider is only for a convenience -- so if you want to you can modify/consolidate the .ebextensions / .platform folder if you're not into me overwriting your stuff.

Don't forget to add +x permission to the EB Platform Hooks scripts

find .platform -type f -iname "*.sh" -exec chmod +x {} +

Configuration

Enable Worker Mode

In order for worker deployment to be active you must add this environmental to your elasticbeanstalk environment configuration:

IS_WORKER = true

If this variable is false or not present the deployment will not run

Set Queue Driver / Connection

Set the driver in your your EB environmental variables:

QUEUE_DRIVER = [driver]

Since Laravel 5.7 the variable name got changed so will also support the new name:

QUEUE_CONNECTION = [driver]

Add Queues

All queues are configured using EB environmental variables with the following syntax:

Note: brackets are placeholders only, do not use them in your actual configuration

queue[QueueName]     = [queueName]   # Required. The name of the queue that should be run.
[QueueName]NumProcs  = [value]       # Optional. The number of instances supervisor should run for this queue. Defaults to 1
[QueueName]Tries     = [value]       # Optional. The number of times the worker should attempt to run in the event an unexpected exit code occurs. Defaults to 5
[QueueName]Sleep     = [value]       # Optional. The number of seconds the worker should sleep if no new jobs are in the queue. Defaults to 5
[QueueName]StartSecs = [value]       # Optional. How long a job should run for to be considered successful. Defaults to 1
[QueueName]Delay     = [value]       # Optional. Time in seconds a job should be delayed before returning to the ready queue. Defaults to 0

Add one queue[QueueName] = [queueName] entry in your EB environmental variables for each queue you want to run. The rest of the parameters are optional.

That's it! On your next deploy supervisor will have its configuration updated/generated and start chugging along on your queues.

Using Your Own supervisord.conf

Using your own, pre-built supervisor config file is easy too.

Simply set the location of the file in the published elasticbeanstalkworker.php config file:

<?php

return array(
	/*
	 * The path of the supervisord.conf file to be used INSTEAD OF generating one from environmental variables. Note that this can be null if you do not have one.
	 *
	 * This path is RELATIVE to the root of your application.
	 * EX:
	 * Absolute Path: /Users/dev/coding/MyProject/config/mysupervisord.conf
	 * Path to use:   config/mysupervisord.conf
	 */
	'supervisorConfigPath' => 'config/mysupervisord.conf`
);

Now during the deploy process your configuration file will be used instead of generating one.

Note: you can check eb-hooks.log for your EB environment to verify if the deploy process detected and deployed your file. Search for Starting supervisor configuration parsing. in the log.

But how does it work?

Glad you asked. It's a simple process but required a ton of trial and error to get right (kudos to AWS for their lack of documentation)

EB applications can contain a folder that provides advanced configuration for an EB environment, called .ebextensions.

EB applications since AL2 can contain platform hooks that provides hook scripts for an EB environment, called .platform.

This package uses AWS commands files in this folder to detect, install, and update supervisor and its configuration and then run it for you.

1. Ingress Supervisor rules

Supervisor requires port 9001 to be open if you want to access its web monitor. This is an optional step and can be removed if you don't need it by deleting 00supervisordIngress.config

2. Parse Queue Configuration

parseConfig.php looks for either a user-supplied supervisord.conf file specified in configuration. If one exists then it is used.

Otherwise parseConfig.php looks for a json file generated earlier that contains all of the environmental variables configured for elastic beanstalk. It then parses out any queue configurations found (see Add Queues) section above and generates a supervisor program for each as well as supplying each program with all the environmental variables set for EB. The program to be generated looks like this:

[program:$queue]
command=php artisan queue:work $connection --queue=$queue --tries=$tries --sleep=$sleep --daemon
directory=/var/app/current/
autostart=true
autorestart=true
process_name=$queue-%(process_num)s
numprocs=$numProcs
startsecs=$startSecs
user=webapp
environment=$environmentVal

After parsing all queues it then appends the programs to a clean supervisord.conf file in the same directory.

3. Deploy Supervisor

Now a bash script workerDeploy.sh checks for IS_WORKER=TRUE in the EB environmental variables:

• If none is found the script does nothing and exists.
• If it is found
  • And there is no init.d script
    • Supervisor is installed using pip and the custom supervisord init script in this project is copied to /etc/init.d
    • Configuration is parsed
    • Supervisor is started
    • Supervisor is set to start at boot
  • And there is an init.d script
    • Supervisor is stopped
    • Configuration is parsed
    • Laravel artisan is used to restart the queue to refresh the daemon
    • Supervisor is restarted with the new configuration

Caveats

Please check out issues and if you need them please make a PR!

Contributing

Make a PR for some extra functionality and I will happily accept it :)

License

This package is licensed under the MIT license.

laravel-elasticbeanstalk-queue-worker's Issues

Class FoxxMD\LaravelElasticBeanstalkQueueWorker\ElasticBeanstalkQueueWorkerProvider contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (Illuminate\Support\ServiceProvider::register)

When php artisan vendor:publish --tag=ebworker

Error when deploying from a Windows OS

I get the following error only when attempting to deploy with a windows machine.

INFO: Deploying new version to instance(s).
ERROR: [Instance: i-0a52f2c7b21b781e0] Command failed on instance. Return code: 127 Output: /usr/bin/env: bash
: No such file or directory.
container_command 005-workerDeploy in .ebextensions/99deployWorker.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
ERROR: Unsuccessful command execution on instance id(s) 'i-0a51f2c7b81b781e7'. Aborting the operation.
ERROR: Failed to deploy application.

I've used this package for over a year within my application using macOS to deploy and never had this error come up.

I assumed that the shell scripts included in this package (IE. workerDeploy.sh) were being ran within the AWS server, not the local machine deploying the application. Is that not the case?

Documentation Example

Would it be possible to expand the documentation with respect to :

queue[QueueName] = [queueName] [QueueName]NumProcs = [value] [QueueName]Tries = [value] [QueueName]Sleep = [value] [QueueName]StartSecs = [value] [QueueName]Delay = [value]

For example, is the default queue defined as:

queuedefault = default

?

Not working with laravel 9: No queue.

Hi

Testing lead me to that command can not be executed since command today does not include queue name.

Line 9 in parse config thus has to get the queue changed.
I have removed below line which causes it to work again.
$connection$queueVal

Laravel 10 Support?

Thanks for your work,

Is there any plans to add Laravel 10 support to this package?

Problem deploying with codepipeline

I installed the package and when I try to deploy it to amazon ebs I get this error:

An error occurred during execution of command [app-deploy] - [FlipApplication]. Stop running the command. Error: create soft link /var/www/html failed: remove /var/www/html: directory not empty

Laravel 10? Your requirements could not be resolved to an installable set of packages.

Hello, get this error in try install the package... Laravel 10 issue?

Try move the folders manual but deploy return this error

× supervisord.service - SYSV: supervisor is a process control utility.  It has a web based
     Loaded: loaded (/etc/rc.d/init.d/supervisord; generated)
     Active: failed (Result: protocol) since Thu 2023-09-14 22:09:06 UTC; 1min 53s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 27737 ExecStart=/etc/rc.d/init.d/supervisord start (code=exited, status=0/SUCCESS)
        CPU: 10ms

Sep 14 22:09:06 ip-.sa-east-1.compute.internal systemd[1]: Starting supervisord.service - SYSV: supervisor is a process control utility.  It has a web based...
Sep 14 22:09:06 ip-1.sa-east-1.compute.internal supervisord[27737]: Starting supervisord:
Sep 14 22:09:06 ip-.sa-east-1.compute.internal supervisord[27741]: /etc/rc.d/init.d/supervisord: line 79: /usr/bin/supervisord: No such file or directory
Sep 14 22:09:06 ip-.sa-east-1.compute.internal supervisord[27742]: /etc/rc.d/init.d/supervisord: line 82: /usr/bin/supervisorctl: No such file or directory
Sep 14 22:09:06 ip-142.sa-east-1.compute.internal systemd[1]: supervisord.service: Can't open PID file /run/supervisord.pid (yet?) after start: Operation not permitted
Sep 14 22:09:06 ip-1.sa-east-1.compute.internal systemd[1]: supervisord.service: Failed with result 'protocol'.
Sep 14 22:09:06 ip-17.sa-east-1.compute.internal systemd[1]: Failed to start supervisord.service - SYSV: supervisor is a process control utility.  It has a web based.

Questions about 'Add Queues'

2 questions:

1. If I don't have a named queue what do I put in the placeholders - queue[QueueName]=[queueName]?

I just want it to run all pending jobs like queue:listen so I don't have specifically named queues.

2. is the naming convention for queue[QueueName] like 'queue' - no space - 'the queue name'?

Thanks!

Allow supervisor program generation without queue option

Right now all programs are generated with --queue because that's how I use it. Not a good use case for all users ¯\_(ツ)_/¯

Make queue parameters case-insensitive

Low hanging fruit. There's no need for parameters to be case-sensitive.

My queue is not being processed

I have installed the plugin according to instructions. Started with composer, then config/app followed by artisan vendor publish. I added the 3 EB variables IS_WORKER=true, QUEUE_DRIVER=sqs, queueMyemails=myemails.

Uploaded my app and I see no sign of whether the supervisor is setup, working or any errors. It's just quiet. The queue is configured properly and can be manually processed via php artisan queue:listen on my local environment.

Allow use of fixed supervisord.conf

Rather than force everyone to use EB variables to generate a supervisord.conf allow them to include their own config and use that instead.

Better default for queue driver

Should we attempt to find the default queue driver specified in Laravel's /config/queue.php? Or throw an exception and force the user to explicitly set it?

Unescaped chars in env

Good work on this library! Simple to use.

If your environment vars contain a characters unsupported e.g. "%0" in the vars the supervisor will not start.

I am not sure how to escape this. I can not change the variable since it is on a production system. When the bashEnv is processed I get error

Error: Format string "[ALL THE ENV's]" for 'environment' is badly formatted: unsupported format character 'O' (0x4f) at index 550 in section 'program:' (file: '/etc/supervisord.conf')

How can this be solved?

Upgrade to Amazon Linux 2?

Firstly thanks for the great work @FoxxMD! I have enjoyed this as a starting point for my .ebextensions directory on Amazon Linux AMI.

Now since I am upgrading to PHP 7.4, I need to also upgrade to Amazon Linux 2.

It looks like they have strangely removed the very handy "get-config" tool that you utilize in 99deployWorker.config:

command: /opt/elasticbeanstalk/bin/get-config environment > jsonEnv

Have you moved to AL2 yourself or do you plan to do so?

If not, is there another way to work around the supervisor deployment without get-config?

Update readme

• Add directions on using user-supplied supervisord.conf
• Remove caveat about case sensitivity

Deployment with Web & Worker Nodes [Clarification]

I wanted to clarify the deployment procedure in the case of Web and Worker Tiers. Assuming I have the same Laravel codebase deployed on my Web and Worker Tiers, am I correct that only the worker tier(s) should have:

IS_WORKER = true

While all Web tier(s) should be:

IS_WORKER = false

Forgive my potential ignorance, but if the Web tier handles a request and pushes a job to a queue, how does the Worker tier know about it? I assume artisan on Supervisor is always listening to the queue and detects the new job being pushed by the Web tier?