Fix for High CPU Load for wp-o-matic
-
My server CPU load used to spike insanely high whenever I ran wp-o-matic. I investigated the code, and there’s a scaling issue with the isDuplicate function. I’ll post the fix below!
First, I’ll explain the problem by explaining how wp-o-matic works on Your Site when processing a feed from Random Site:
- First, Random Site publishes, say, the 10 most recent posts to its RSS feed
- Then, wp-o-matic will pull in those 10 posts by sucking in the Random Site RSS feed, as you’ve configured it…
- Next, wp-o-matic doesn’t know whether those 10 Random Site posts are *new* posts, or have already been published at Your Site. So it has to check this. The way wp-o-matic currently checks this is to compare each of those 10 posts in the RSS feed to the existing posts on Your Site that have been pulled from Random Site.
That is where the problem arises. The way wp-o-matic checks if new posts are duplicates generates a LOT of queries. Specifically, wp-o-matic queries the database one time for every post that has come from that feed, multiplied by the number of posts in the RSS feed. For instance, I have syndicated the content of one prolific blogger who has written 5,408 posts that are published on my site. The RSS feed has 10 entries. Thus, for this one feed, wp-o-matic generates 54,080 queries!
To resolve this, I rewrote the first part of function processFeed in the main wpomatic.php file (copy/paste below, up to the line where it says
$count++;). With this code, wp-o-matic will vastly reduce its database load.P.S. You can also delete the isDuplicate function if you want, as it’s not called elsewhere.
function processFeed(&$campaign, &$feed)
{
global $wpdb;
@set_time_limit(0);
// Log
$this->log('Processing feed ' . $feed->title . ' (ID: ' . $feed->id . ')');
// Access the feed
$simplepie = $this->fetchFeed($feed->url, false, $campaign->max);
// Get posts (last is first)
$items = array();
$count = 0;
// Get all rows for this campaign all at once, BUT LIMITED to the last 30 posts
$rows = $wpdb->get_results("SELECT * FROM {$this->db['campaign_post']} "
. "WHERE campaign_id = {$campaign->id} AND feed_id = {$feed->id} ORDER BY post_id DESC LIMIT 30 ");
foreach($simplepie->get_items() as $item)
{
if($feed->hash == $this->getItemHash($item))
{
if($count == 0) $this->log('No new posts');
break;
}
// look to see if $item is a duplicate of an existing $rows row
foreach($rows as $row)
{
if($row->hash == $this->getItemHash($item))
{
$this->log('Filtering duplicate post');
break;
}
}
$count++;
- The topic ‘Fix for High CPU Load for wp-o-matic’ is closed to new replies.
