Adam's Perspective

Dennis Snell and I want to sync two WordPress sites and we’ve been looking for a way to run a continuous content indexing in WordPress. One idea was to reuse the PHP process that renders the site and do a little bit of indexing after every page render. We’d need to flush all the buffers, ship the response, close the connection, and nudge the browser we’re now in a “fully loaded” state. Only then we’d do additional processing within that same PHP handler.

We’ve found three approaches. If I were to run this in production, I would add an exclusive lock to ensure the added overhead is limited to a single request at a time.

Check the complete code examples in the gist.

Asynchronous curl_exec()

In register_shutdown_function we spawn a new HTTP request via curl_exec. We then close the connection without waiting for its response:

<?php

register_shutdown_function(function() {
	// Init the request. The postprocess.php script
	// should call ignore_user_abort( true ); to 
	// continue running after the client drops.
	$ch = curl_init('http://localhost/postprocess.php');

	// Don't wait for the response
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
	curl_setopt($ch, CURLOPT_TIMEOUT_MS, 100);        
        // Do not transfer the response body
	curl_setopt($ch, CURLOPT_NOBODY, true);
	curl_exec($ch);
	curl_close($ch);
});

echo "We are exiting the main program.\n";
flush();
ob_flush();
exit;

The spawned request will typically continue executing until completion, although some servers might need a configuration tweak if they’re optimized to stop processing as soon as the client connection drops.

fastcgi_finish_request()

In PHP-FPM, fastcgi_finish_request() cleanly flushes the buffers and closes the client connection. From there, we can use register_shutdown_function to run additional processing:

<?php

register_shutdown_function(function() {
	if(have_less_than_10_seconds_left()) {
		return;
	}
	if(!acquire_exclusive_lock('heavy processing')) {
		return;
	}
	echo "Run extra heavy processing.\n";
});

echo "We are exiting the main program.\n";
fastcgi_finish_request();
exit;

One nice property of register_shutdown_function is that it gets additional time to run even after we hit the PHP’s max_execution_time. Here’s an outdated but illustrative excerpt from the php-src repo:

void php_on_timeout(int seconds)
{
    PG(connection_status) |= PHP_CONNECTION_TIMEOUT;
    zend_set_timeout(EG(timeout_seconds), 1);
    if(PG(exit_on_timeout)) sapi_terminate_process();
}

Content-length

We can nudge the browser to close the connection by providing a fixed Content-length header and then providing that many body bytes:

<?php

ob_start();

register_shutdown_function(function() {
	if(have_less_than_10_seconds_left()) {
		return;
	}
	if(!acquire_exclusive_lock('heavy processing')) {
		return;
	}
	// We cannot echo things anymore. It also seems like we can't write to stderr at this point.
	file_put_contents('output_log.txt', "Run extra heavy processing.\n", FILE_APPEND);
});

echo "We are exiting the main program.\n";
header('Content-Length: ' . ob_get_length());
ob_end_flush();
flush();

exit;

This should also work with Transfer-Encoding: chunked if you need to stream the response.

Again, check the gist for complete code examples.

Aftermath

These ideas are workable and may come handy at one point. They’re not without issues, though, so we’ll kick our work off with the regular wp-cron async job and stick to it for as long as we can.