Provides a framework for using job queues with WordPress for asynchronous task running. Provides an integration with Gearman and RabbitMQ out of the box.
Caution
As of 12 April 2024, this project is archived and no longer being actively maintained.
As WordPress becomes a more popular publishing platform for increasingly large publishers, with complex workflows, the need for increasingly complex and resource-intensive tasks has only increased. Things like generating reports, expensive API calls, syncing users to mail providers, or even ingesting content from feeds all take a lot of time or a lot of memory (or both), and commonly can't finish within common limitations of web servers, because things like timeouts and memory limits get in the way.
WP Minions provides a few helper functions that allow you to add tasks to a queue, and specify an action that should be called to trigger the task, just hook a callback into the action using add_action()
During configuration, a number of minions (workers) are specified. As minions are free, they will take the next task from the queue, call the action, and any callbacks hooked into the action will be run.
In the situation of needing more ram or higher timeouts, a separate server to process the tasks is ideal - Just set up WordPress on that server like the standard web servers, and up the resources. Make sure not to send any production traffic to the server, and it will exclusively handle tasks from the queue.
-
Install the plugin in WordPress. If desired, you can download a zip and install via the WordPress plugin installer.
-
Create a symlink at the site root (the same directory as
wp-settings.php) that points to thewp-minions-runner.phpfile in the plugin (or copy the file, but a symlink will ensure it is updated if the plugin is updated) -
Define a unique salt in
wp-config.phpso that multiple installs don't conflict.
define( 'WP_ASYNC_TASK_SALT', 'my-unique-salt-1' );Note: If you are using multisite, you'll also have to add the following to your wp-config.php file, after the block with the multisite definitions. This is due to the fact that multisite relies on HTTP_HOST to detect the site/blog it is running under. You'll also want to make sure you are actually defining DOMAIN_CURRENT_SITE in the multisite configuration.
if ( ! isset( $_SERVER['HTTP_HOST'] ) && defined( 'DOING_ASYNC' ) && DOING_ASYNC ) {
$_SERVER['HTTP_HOST'] = DOMAIN_CURRENT_SITE;
}- Next, you'll need to choose your job queue system. Gearman and RabbitMQ are supported out of the box.
There are a few parts to get this all running. First, the Gearman backend needs to be setup - this part will vary depending on your OS. Once that is complete, we can install the WordPress plugin, and set the configuration options for WordPress.
- You'll need the EPEL repo for gearman, and the REMI repo for some of the php packages. Make sure to enable the appropriate remi repo for the version of php you are using.
wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm && rpm -Uvh epel-release-6*.rpmwget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm && rpm -Uvh remi-release-6*.rpmrm *.rpm
- Make sure that remi is enabled, as well as any specific version of php you may want in
/etc/yum.repos.d/remi.repo yum install gearmand php-pecl-gearman python-pipeasy_install supervisorchkconfig supervisord on && chkconfig gearmand on- If everything is running on one server, I would recommend limiting connections to localhost only. If not, you'll want to set up firewall rules to only allow certain clients to connect on the Gearman port (Default 4730)
- edit
/etc/sysconfig/gearmand- setOPTIONS="--listen=localhost"
service gearmand start
- You'll need the EPEL repo for gearman, and the REMI repo for some of the php packages.
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpmyum install http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
yum install gearmand php-pecl-gearman --enablerepo=remi-php<php version on your system>. For example, if you are using php 7.2 your command would look like thisyum install gearmand php-pecl-gearman --enablerepo=remi-php72- Optionally, install supervisord if you prefer it
yum install python-pipeasy_install supervisor
- If everything is running on one server, I would recommend limiting connections to localhost only. If not, you'll want to set up firewall rules to only allow certain clients to connect on the Gearman port (Default 4730)
- edit
/etc/sysconfig/gearmand- setOPTIONS="--listen=localhost"
- edit
systemctl enable gearmandsystemctl start gearmand
As you go through this, you may need to install additional packages, if you do not have them already, such as php-pear or a php*-dev package
apt-get updateapt-get install gearman python-pip libgearman-dev supervisorpecl install gearman- Once pecl install is complete, it will tell you to place something like
extension=gearman.sointo your php.ini file - Do this. update-rc.d gearman-job-server defaults && update-rc.d supervisor defaults- If everything is running on one server, I would recommend limiting connections to localhost only. If not, you'll want to set up firewall rules to only allow certain clients to connect on the Gearman port (Default 4730)
- edit
/etc/default/gearman-job-server- setPARAMS="--listen=localhost"
service gearman-job-server restart
Filling in values in <brackets> as required, add the following config to either /etc/supervisord.conf (CentOS) or /etc/supervisor/supervisord.conf (Ubuntu)
[program:my_wp_minions_workers]
command=/usr/bin/env php <path_to_wordpress>/wp-minions-runner.php
process_name=%(program_name)s-%(process_num)02d
numprocs=<number_of_minions>
directory=<path_to_temp_directory>
autostart=true
autorestart=true
killasgroup=true
user=<user>- path_to_wordpress: Absolute path to the root of your WordPress install, ex:
/var/www/html/wordpress - number_of_minions: How many minions should be spawned (How many jobs can be running at once).
- path_to_temp_directory: probably should just be the same as path_to_wordpress.
- user: The system user to run the processes under, probably apache (CentOS), nginx (CentOS), or www-data (Ubuntu).
- You can optionally change the "my_wp_minions_workers" text to something more descriptive, if you'd like.
After updating the supervisor configuration, restart the service (CentOS or Ubuntu)
systemctl restart supervisord
service supervisor restart
Filling in values in <brackets> as required, add the following to /etc/systemd/system/[email protected]
[Unit]
Description=WP-Minions Runner %i
After=network.target
[Service]
PIDFile=/var/run/wp-minions-runner.%i.pid
User=<user>
Type=simple
ExecStart=/usr/bin/env php <path_to_wordpress>/wp-minions-runner.php
Restart=always
[Install]
WantedBy=multi-user.target
- path_to_wordpress: Absolute path to the root of your WordPress install, ex:
/var/www/html/wordpress - user: The system user to run the processes under, probably apache (CentOS), nginx (CentOS), or www-data (Ubuntu).
Reload systemd:
systemctl daemon-reload
Enable and start as many runners as you'd like to have running:
systemctl enable wp-minions-runner@{1..n}
systemctl start wp-minions-runner@{1..n}
Where 'n' is the number of processes you want.
Define the WP_MINIONS_BACKEND constant in your wp-config.php. Valid values are gearman or rabbitmq. If left blank, it will default to a cron client.
define( 'WP_MINIONS_BACKEND', 'gearman' );
If your job queue service not running locally or uses a non-standard port, you'll need define your servers in wp-config.php
# Gearman config
global $gearman_servers;
$gearman_servers = array(
'127.0.0.1:4730',
);
# RabbitMQ config
global $rabbitmq_server;
$rabbitmq_server = array(
'host' => '127.0.0.1',
'port' => 5672,
'username' => 'guest',
'password' => 'guest',
);
Note: On RabbitMQ the guest/guest account is the default administrator account, RabbitMQ will only allow connections connections on that account from localhost. Connections to any non-loopback address will be denied. See the RabbitMQ manual on [user management](https://www.rabbitmq.com/rabbitmqctl.8.html#User_Management) and [Access Control](https://www.rabbitmq.com/rabbitmqctl.8.html#Access_Control) for information on adding users and allowing them access to RabbitMQ resources.
## MySQL Persistent Queue (Recommended)
By default, gearman will store the job queue in memory. If for whatever reason the gearman service goes away, so does the queue. For persistence, you can optionally use a MySQL database for the job queue:
#### CentOS
Edit the gearman config at ```/etc/sysconfig/gearmand```, adding the following to the OPTIONS line (or creating the line, if it doesn't exist yet), inserting database credentials as necessary:
```sh
OPTIONS="-q MySQL --mysql-host=localhost --mysql-port=3306 --mysql-user=<user> --mysql-password=<password> --mysql-db=gearman --mysql-table=gearman_queue"
Edit the gearman config at /etc/default/gearman-job-server, adding the following to the PARAMS line (or creating the line, if it doesn't exist yet), inserting database credentials as necessary:
PARAMS="-q MySQL --mysql-host=localhost --mysql-port=3306 --mysql-user=<user> --mysql-password=<password> --mysql-db=gearman --mysql-table=gearman_queue"Note: For some setups, the above will not work as /etc/default/gearman-job-server does not get read. If you don't see the persistent queue setup then:
- Create a
gearmandb in mysql (the database must be present in the database, but when gearmand is initialized the first time it will create the table). - Create a file in
/etc/gearmand.conf - In the file paste the configuration all on one line:
-q MySQL --mysql-host=localhost --mysql-port=3306 --mysql-user=<user> --mysql-password=<password> --mysql-db=gearman --mysql-table=gearman_queueThen restart the gearman-job-server: sudo service gearman-job-server restart.
Once everything is installed, you can quickly make sure your job queue is accepting jobs with the test-client.php and test-worker.php files located in the system-tests/YOURQUEUE directory. The worker is configured to reverse any text passed to it. In the client file, we pass "Hello World" to the worker.
In one window, run php test-worker.php - You'll now have one worker process running, waiting for jobs.
In another window, run php test-client.php "Hello World" - You should see "dlroW olleH" printed on your screen.
ctrl-c will stop the worker once you are done testing.
Once configured and activated, you'll have access to wp_async_task_add(). If you are at all familiar with wp_schedule_single_event(), the way wp_async_task_add() works should be very familiar to you.
The function takes up to three arguments, the first of which is required:
$hook: This is the name of the action hook to execute when the job runs. Your callback function should hook into this withadd_action( $hook, $callback )$args: This is optional, and defaults to an empty array. You can pass an array of arbitrary data to this, and it will be passed to your callback function.$priority: This is optional, and defaults to "normal". Other valid options are "high" or "low". High priority jobs will be run before normal priority jobs, even if they normal priority job has been in the queue longer.
Set an option in the database, when a worker becomes available:
// Add a task, that will call the "myplugin_update_option" action when it is run
wp_async_task_add( 'myplugin_update_option', array( 'mykey' => 'myvalue' ) );
function myplugin_update_option_callback( $args ) {
// In reality, you are probably doing a lot of resource intensive work here
update_option( 'my-option-name', $args['mykey'] );
}
// Add the action that links the task and the callback.
// Notice the hook below is the same as the hook provided to wp_async_task_add.
add_action( 'myplugin_update_option', 'myplugin_update_option_callback' );Once a worker is free, and runs the above task, you'd have an option called "my-option-name" in the options table, with a value of "myvalue", since "myvalue" was passed in via the $args
The following constants can be used to customize the behaviour of WP Minions.
-
WP_MINIONS_JOBS_PER_WORKER- The number of jobs to execute per Worker, default is 1. Running multiple jobs per worker will reduce the number workers spawned, and can significantly boost performance. However too large a value will cause issues if you have memory leaks. Use with caution. -
WP_MINIONS_CLIENT_CLASS- You can also alter the Client class used to send jobs to your job queue. It should match the interface of\WpMinions\Client. -
WP_MINIONS_WORKER_CLASS- Similarly you can alter the Worker class used to execute jobs. It should match the interface of\WpMinions\Worker.
If you identify any errors or have an idea for improving the plugin, please open an issue. We're excited to see what the community thinks of this project, and we would love your input!
Archived: This project is no longer maintained by 10up. We are no longer responding to Issues or Pull Requests unless they relate to security concerns. We encourage interested developers to fork this project and make it their own!
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent