Background workers - how do you do it?

[gebeer]

Hi,

in a recent project I had to import large amounts of data that got uploaded through the frontend. To avoid lags in the frontend I pushed the actual import work to processes in the background.

Since timely resources for that project were limited, I resorted to a quite simple method of starting the workers

	public function startSalesImportWorker()
	{
		$path = $this->config->paths->siteModules . "{$this->className}/workers/";
		$command = "php {$path}salesimportworker.php";
		$outputFile = "{$path}/logs/workerlogs.txt";
		$this->workerPid = (int) shell_exec(sprintf("%s > $outputFile 2>&1 & echo $!", "$command"));
		if ($this->workerPid) return $this->workerPid;
		return false;
	}

Here's the worker code

namespace ProcessWire;

use SlashTrace\SlashTrace;
use SlashTrace\EventHandler\DebugHandler;

include(__DIR__ . "/../vendor/autoload.php");
include(__DIR__ . "/../../../../index.php");
ini_set('display_errors', false);
error_reporting(E_WARNING | E_ERROR);
$slashtrace = new SlashTrace();
$slashtrace->addHandler(new DebugHandler());
// $slashtrace->register();

$lockfile = __DIR__ . "/locks/lock-" . getmypid();

// restart when fail or done
function workerShutdown($args) {
	// release lockfile
	if (file_exists($args['lockfile'])) unlink($args['lockfile']);
    echo PHP_EOL . "Restarting...\n";
	$outputFile = __DIR__ . '/logs/workerlogs.txt';
	$command = PHP_BINARY . " " . $args['command'];
	sleep(1);
	// execute worker again
	exec(sprintf("%s > $outputFile 2>&1 & echo $!", "$command"));
}
register_shutdown_function('ProcessWire\workerShutdown', array('lockfile' => $lockfile, 'command' =>$argv[0]));

// wait for other workers to finish
while (wire('files')->find(__DIR__ . "/locks/")) {
	sleep(5);
}

// create lockfile
file_put_contents($lockfile, $lockfile);

try {

	// ini_set('max_execution_time', 300); //300 seconds = 5 minutes
	wire('users')->setCurrentUser(wire('users')->get("admin"));
	echo "starting import: " . date('Y-m-d H:i:s') . PHP_EOL;
	/** @var \ProcessWire\DataImport $mod */
	$mod = wire('modules')->get("DataImport");
	$mod->importSales();
	echo PHP_EOL . "Import finished: " . date('Y-m-d H:i:s');
	// run only for 1 round, then start a new process: prevent memory issues
	die;
} catch (\Exception $e) {
	$slashtrace->handleException($e);
	die;
}

I got the idea for restarting the same worker from https://www.algotech.solutions/blog/php/easy-way-to-keep-background-php-jobs-alive/

Note that I am using https://github.com/slashtrace/slashtrace for error handling since it gives nice CLI output. And I couldn't figure out how to utilize the native PW Debug class for that since I needed stack traces.

Overall this solution worked quite well. But it doesn't give any control over the worker processes. At least there was no time to implement.

Only after having finished the project, I discovered https://symfony.com/doc/current/components/process.html which seems to have everything you need to start and monitor background processes. So next time the need arises I will definitely give it a try. I'm imagining a Process module that lets you monitor/stop background workers and a generic module to kick them off.

How do you handle background processes with PW?