Advanced: Improve Performance by Running a Dedicated Queue Worker

When running asynchronous and batch processed tasks, the process is as follows:

1. Administrator requests that a task is run (from the Dashboard)

2. The task processor creates a batch of however many asynchronous commands that need to run. That means a “Reindex Content” task might generate thousands of commands to run, if your site has thousands of pages.

3. These commands are grouped into a batch, which is then dispatched.

At this point, the Dashboard Tasks interaction is complete. The commands have been dispatched to the task runner. The actual handling and processing of those commands can work one of two ways. By default, every time the Dashboard is loaded, JavaScript checks to see if any pending commands need to be executed, and if so will execute a set block of those. This will happen in the background, in a process called “polling.”

This is usually transparent to the user, and is the default behavior. However, this does mean that the same PHP process used for browsing the site will be used to execute processes, and will generally make the entire processing of a batch take much longer than if a dedicated process were handling it behind the scenes. That’s why we have a dedicated queue worker option. If you have access to the console, it’s highly recommended that you enable the “manual queue listener.” Here’s how to do it.

To enable the dedicated queue worker option, head to the Dashboard > System & Settings > Automation > Automation Settings page, and turn on “Manual Queue Listening”:

The note listed below the options is very important: when you turn on manual queue listening the queued commands will no longer be automatically processed in the background when a user is browsing the Dashboard page. Instead, they are consumed by a separate queue worker which you must start via a console command or some other means. That means manually running the concrete/bin/concrete messenger:consume async command from the console on your web server.

Let’s test this out. Change the setting above, and then run the queue listener from the command line:

concrete/bin/concrete messenger:consume async

The benefits to this change are considerable. With the standard polling/consume method, it takes about two minutes to rescan all 249 pages that are installed as part of the standard Atomik install (this includes Dashboard pages). With the manual worker, it takes about 10 seconds (and only makes it through one polling cycle):

Multiple Queue Workers

If you enable manual queue listening, you can run multiple queue listeners. Simply re-run the queue worker multiple times, and commands will go through the queue more quickly. They operate in parallel. (Note: there are obvious limitations to this strategy, so make sure this configuration will work for your server given its RAM and CPU limits.)