Queueable Jobs are Concrete CMS Jobs that can queue their operations. Oftentimes, Jobs are used for operations that are potentially lengthy or memory-intensive. If portions of a job can be broken down into discreet bits of functionality, Queueable Jobs are an ideal way to tackle these types of problems: since they only operate on a subset of items at once, their memory usage can remain low, and they have a better chance of finishing successfully.
Creating a Queueable Job
A Queueable Job is basically a job that extends a difference class, and implements a couple different methods. It also has to make use of the Concrete \Concrete\Core\Foundation\Queue class.
Extending the Proper Class
Let's consider this job.
class TestJob extends \Concrete\Core\Job\Job
{
public function getJobName()
{
return t("Test Job");
}
public function getJobDescription()
{
return t("We will make this job queueable for better performance.");
}
public function run()
{
$list = new \Concrete\Core\Page\PageList();
$list->ignorePermissions();
$pages = $list->getResults();
$processed = 0;
foreach($pages as $page) {
$page->setAttribute('test_attribute', 'Test!');
$processed++;
}
return t('%s pages processed', $processed);
}
}
This job has one purpose: to loop through every page on your site (without checking permissions, since jobs frequently run in a non-authenticated process anyway) and set the content of the attribute with the handle "test_attribute" to "Test!"). Obviously, a proof of concept. We then return a string with the number of pages processed.
But what if our site has thousands of pages? This job could pretty easily time out, so it's a perfect opportunity to become Queueable.
First, instead of extending \Concrete\Core\Job\Job, make the Job extend the QueueableJob class.
class TestJob extends \Concrete\Core\Job\QueueableJob
Once this is the case, you'll need to implement a few methods, since the QueueableJob class is an abstract.
abstract public function start(\ZendQueue\Queue $q);
abstract public function finish(\ZendQueue\Queue $q);
abstract public function processQueueItem(\ZendQueue\Message $msg);
These methods all work with the Queue class from the Zend Framework. We're going to be created a start method, which will take care of seeding our queue with messages, a finish class that will take care of cleaning up when our Job is finished, and a processQueueItem method, which will be run for every single item that's in our queue. Let's do the start method first:
public function start(\ZendQueue\Queue $q)
{
$list = new \Concrete\Core\Page\PageList();
$list->ignorePermissions();
$results = $list->executeGetResults();
foreach($results as $queryRow) {
$q->send($queryRow['cID']);
}
}
The first part of this method should look familiar. We create a PageList instance and ignore permissions on it. But what's going on in the second part of the method? Instead of getting all page objects (which is the standard behavior of the PageList getResults() method, we use the executeGetResults() method, which executes the query, but doesn't actually return any objects. Instead, it just returns raw query data from the database. This is perfect, because we want the start method to be as lightweight as possible, since it has to run in its entirety in order to get all the data into the queue. So we run our query, we retrieves a list of all page IDs in the system. We then loop through all these page IDs and send them into the queue.
Next, let's implement processQueueItem.
public function processQueueItem(\ZendQueue\Message $msg)
{
$page = \Page::getByID($msg->body);
$page->setAttribute('test_attribute', 'Test!');
}
This should be pretty straightforward. For every item in the queue (represented by the $msg object), we loop through and retrieve the full body of the Message object. Whatever you send into the queue in the start() method using $q->send() will automatically populate the $msg->body property that's accessible through processQueueItem. This is a serialized object in the database, meaning you can store more than simple scalar values. However, since all we have to do is store page IDs as part of this operation, we can get away with sending the page ID as the full message data. We build a page object from the page ID in the $msg object, and we set the attribute on the page.
Finally, we implement our finish() method. Since the finish() method doesn't necessarily know anything about the queue, sometimes it can be a little difficult to get real data about how many items were processed – but many times you don't need to pass this data back anyway.
public function finish(\ZendQueue\Queue $q)
{
return t('All pages processed');
}
That's it! We've turned our potentially problematic Job into a QueueableJob that can scale from 50 pages to 5000.
Scheduling a Queueable Job
Scheduling a Queueable Job works differently than a regular job. A regular Job, if it has as job ID of 1, would be accessed this way in a headless system:
http://www.yoursite.com/index.php/ccm/system/jobs?auth=authenticationHash&jID=1
Where authenticatioHash is equal to the authentication hash displayed in the Dashboard Jobs user interface. Typically something like wget or curl would be used to hit this URL periodically, running the job in the background of a site.
If this is done with a QueueableJob, the entire Job will be run all at once. This is for backward compatibility, and so that we never have a situation where someone thinks they're running a job and it isn't running. But what if you want to schedule a Queueable Job to run periodically, but still have it operate in batches like it does from the Concrete Jobs interface? This requires a different approach. Instead of hitting the above URL, schedule two Jobs to run. The first is the Job URL for this Queueable Job:
http://www.yoursite.com/index.php/ccm/system/jobs/run_single?auth=authenticationHash&jID=1
Schedule this for when you'd like this Job to begin running.
Next, schedule the Job watcher process:
http://www.yoursite.com/index.php/ccm/system/jobs/check_queue?auth=authenticationHash
Schedule this to run frequently. This process simply checks any queues in the system and processes the next X items in the queue. If the queue is empty the process exits quietly. Got multiple jobs you'd like to Queue? Schedule them as above, and schedule a single run of the check_queue script. It will work for multiple Queueable Jobs.
That's it! Now you can have the benefits of Queueable Jobs in a scheduled, headless environment.