WordPress.org

Ready to get started?Download WordPress

Forums

WP RSS Multi Importer
Many feeds leads to too much memory use (24 posts)

  1. bkwineper
    Member
    Posted 1 year ago #

    Hi,

    I am trying to set this excellent plugin to pull info from quite a few feeds.

    When the number of feeds I pull information from reaches ~70 feeds the page (the page that a visitor would see) will not display.

    Instead an error message appears:

    "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 196736 bytes) in /nas/wp/www/staging/bkwine/wp-includes/functions.php on line 316"

    Does this plugin consume a lot of memory, or is there for some other reason a limit to the number of feeds that one can have?

    -Per

    http://wordpress.org/extend/plugins/wp-rss-multi-importer/

  2. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    Hi
    I haven't tried using more than 40 feeds but I can say that is a typical WordPress error that is a result of a memory limit (which can be made to go away if you increase memory but I think that is in the core php file). Have you tried turning on the caching in the settings panel which will store the feeds in cache memory?

    Allen

  3. bkwineper
    Member
    Posted 1 year ago #

    Hi,

    Thanks for a rapid answer!

    Yes, I did turn on the caching (30 minutes). Figured it could help, but it didn't make much difference. (none that I could notice actually)

    I'm testing it on a staging site (on WP Engine) and I am not sure if that perhaps limits the caching possibility. I don't think it does, but I am not sure.

    I'm struggling to figure out a way around this, since it is such a nice tool but not sure what I could do.

    -Per

  4. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    Well, what I've done when I'm on a development server is increase the memory limit, which makes the warning go away. Not sure what else I can say...what appears to be happening is that SimplePie, which is responsible for fetching the feeds, puts the memory load...my plugin isn't that memory intensive (there is some memory load when it gets to the stage of resizing images, etc.) but the memory error always occurs way before that...when the feeds are being fetched, and that is being done, as I said, by SimplePie.

    Looks like you have access to the php.ini file, if so, you can increase memory there and see if that solves the problem. You can also go into the wordpress wp-config.php file and use this:

    define('WP_MEMORY_LIMIT', '64M');

    and increase from 64M to 128M, etc.

    Not sure what else I can say...it may just be a limit that I will need to test and see if I can find some other way to optimize the memory for the plugin..which I'll try to do

  5. bkwineper
    Member
    Posted 1 year ago #

    I'll check with my hosting provider to see if there's anything they can do.

    I know nothing about the php files and how it is set up so I am reluctant to manipulate any of them myself.

    -Per

  6. bkwineper
    Member
    Posted 1 year ago #

    They're looking into it.

    Meanwhile, some more details. It is not systematic that I get the error message. Sometimes it works.

    But even when it works the page load time is very long, 30+ seconds. So I guess this is not a good way to do what I want to do.

  7. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    and that happens even when you have the caching on? When the cache is on, it doesn't go out and fetch new feed, but instead gets the data from the database. Assuming this is unrelated to our server, then it may just be a limit as to how many feeds my plugin can handle.

  8. bkwineper
    Member
    Posted 1 year ago #

    Well, perhaps. Caching was set to 30 min and it is just a bit over 30 min since I checked the page, so I guess it passed the limit. I'll change caching to 300 and test again to make sure I'm withing the caching time period.

  9. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    also, if you go to google and type in

    Fatal error: Allowed memory exhausted /wp-includes/functions.php on line 316"

    you'll see this is actually a quite common problem on wordpress..plus you'll see how people are usually solving this by some changes to their wordpress config file, like this one:

    http://wordpress.org/support/topic/fatal-error-allowed-memory-size-6?replies=13#post-1017842

  10. bkwineper
    Member
    Posted 1 year ago #

    You're correct. Caching makes a big difference. From what I can see now.

    How does the caching work? Does it go out and fetch *everything* at the same time once the caching limit is reached? Which means that one will ge the slow response time then. Or is there some kind of progressive update of the different items?

  11. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    Correct. What you could do is set up a cron service to hit the page every 30 min (and set the cache for 30 min). That way nobody will ever go to the page and need to bring in the feeds (they will always be refreshed and in the database). Does that make sense?

  12. bkwineper
    Member
    Posted 1 year ago #

    Except that I don't know what a cron service is. ;-)

    I'll see if it makes sense or if I should do something else. It seems not to be an ideal way of doing it, in view of the issues.

  13. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    A cron service is just something that hits your web page at designated intervals. Like this site
    https://www.setcronjob.com/

    although, there are other ways to do this..I'm going to include a cron service in a future update to the plugin.

  14. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    Hi Per
    I figured out a cron solution...if you want to give it a try, let me know and I'll send you instruction on how to do this within my plugin.

    Allen

  15. bkwineper
    Member
    Posted 1 year ago #

    Allen,

    Thanks for the suggestion! would be interesting to look at.

    You can find an email to reach me on here:
    http://www.bkwine.com/contact/

    However, I think I'll wait a bit with implementing it live. My web host is looking at what might be the reason for the memory issue so I'll let them get on with that first.

    -Per

  16. bkwineper
    Member
    Posted 1 year ago #

    Allen,

    Have had some feedback from my hoster. thought it might be of interest to you. You may understand a bit more of it than I do.

    Here:

    ----------------

    The first error that's coming up is:
    "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 196736 bytes) in /nas/wp/www/staging/bkwine/wp-includes/functions.php on line 316"

    Now, line 316 in "wp-includes/functions.php" pertains to data being serialized so it can be stored in the WordPress database. It's specifically the "return serialize( $data );" in this bit of code:

    /**
    * Serialize data, if needed.
    *
    * @since 2.0.5
    *
    * @param mixed $data Data that might be serialized.
    * @return mixed A scalar data
    */
    function maybe_serialize( $data ) {
    if ( is_array( $data ) || is_object( $data ) )
    return serialize( $data );

    // Double serialization is required for backward compatibility.
    // See http://core.trac.wordpress.org/ticket/12930
    if ( is_serialized( $data ) )
    return serialize( $data );

    return $data;
    }

    And now, the second error:

    "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 321481 bytes) in /nas/wp/www/staging/bkwine/wp-includes/wp-db.php on line 791"

    Which refers to "return addslashes( $string );" in "wp-includes/wp-db.php" in this stretch of code:

    /**
    * Real escape, using mysql_real_escape_string() or addslashes()
    *
    * @see mysql_real_escape_string()
    * @see addslashes()
    * @since 2.8.0
    * @access private
    *
    * @param string $string to escape
    * @return string escaped
    */
    function _real_escape( $string ) {
    if ( $this->dbh && $this->real_escape )
    return mysql_real_escape_string( $string, $this->dbh );
    else
    return addslashes( $string );
    }

    That's turning slashes (of which there are a bunch) into properly escaped slashes for storage in the database. So basically, the feed plugin is running out of memory processing 70+ feeds at once. You need to either take this up with the plugin developer
    Or consider an alternative feed processing script that allows you to stagger how often feeds are checked. A solid one (with great support) is:
    http://premium.wpmudev.org/project/autoblog/

    ------------------------------------------------

    (There's also another one I've looked at, that is not at all as feature rich as RSS Multi-Importer but that does not seem to have the same problem with a large number of RSS feeds: WP RSS Aggregator.)

    -Per

  17. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    Hi Per
    First of all, thanks for the this information. Have you tried to put in 70 feeds into WP RSS Aggregator and it works with no errors. I'm interested because that plugin and mine use the same basic way to input the feeds, which is where the error is occurring. Now, my plugin hasn't had cron support, which theirs does, but will have it in the next update.

    Finally, I'll look into the escaped slashes issue and see if this is what is possibly causing this memory error.

    Thanks again,
    Allen

  18. bkwineper
    Member
    Posted 1 year ago #

    Allen,
    Yes, I've tried it with 70+ feeds. No problem at all and very quick page load time.

    I could show it to you on my staging site, but unfortunately I am currently having some problems with the staging site at the moment (unrelated to this).

    I don't know what's going on behind the scene, technically, but it seem that WP RSS AG imports the feed post headlines into a local WP database and then publishes the headlines from that. Whereas it seems that your plugin directly goes out and picks up the headlines "on the fly" each time the page is accessed.

    On the error messages: just a word of warning. It was a different error msg ever time the page was accessed (unless it loaded OK of course) so I am not sure that those error messages are actually relevant.

  19. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    right...well, that plugin stores everything in a database that gets updated every hour while mine has the option to do that (that's because many people have news on their sites want it updated continuously) ....also, I'm storing it in transient tables rather than a dedicated table (not sure if that changes anything).

    So, in essence because I have many features and capabilities I've had to make tradeoffs and one might be the ability to store 70+ feeds.

  20. lpotts
    Member
    Posted 1 year ago #

    Hi Allen

    I'm having the same problem with just over 40 feeds. It's pretty bad and the site doesn't load most of the time. Can you post the method for setting up the cron/cache to clear this up.

    Will changing the number of feeds displayed help?

    Thanks,
    lee

    http://www.digitalfranciscans.com/

  21. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    Hello,
    If you are using the shortcode with a large number of feeds, then these are some steps I would suggest.

    1) Put the feeds into different categories and only pull those feeds you need on a given page.

    2) Consider using WP Super Cache since that helps the caching method that I use
    http://wordpress.org/extend/plugins/wp-super-cache/

    I would set the cache on the Shortcode settings panel to some number, not 0 since that won't take advantage of the caching technique.

    Finally, regarding the memory, this is partly due to a memory leak in simplepie. Now, having said this, sometimes the problem is due to some of the feeds taking too much time to load (because of their server problems). For this, in my next update I'll have the option to set the time to a few seconds, and if the feed isn't available it will just go to the next feed. If you want to try this out, contact me at http://www.allenweiss.com/contact

  22. tsjwebmaster
    Member
    Posted 1 year ago #

    Help!
    I went to install your plugin on my website (www.lajornadalatina.com/wordpressjornada)
    I activated it and now I can log in and the site is down.
    Here is the message at admin login:
    Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 122880 bytes) in /data/5/1/52/131/1215294/user/1295579/htdocs/wordpressjornada/wp-includes/class-simplepie.php on line 14744

    Here's what you get when you try to visit the site
    Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 122880 bytes) in /data/5/1/52/131/1215294/user/1295579/htdocs/wordpressjornada/wp-content/plugins/jetpack/modules/sharedaddy/sharing-service.php on line 575

    Need help to untangle or undo what has been done. Thanks.

  23. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    If you look at the error message this is about memory usage. My plugin does use memory but from the error it's associated with jetpack that is another plugin that is possibly causing main problem. Also, just activating my plugin doesn't use that much memory.

  24. Allen
    Member
    Plugin Author

    Posted 1 year ago #

    If you are running into memory problems and using the shortcode, contact me at http://www.allenweiss.com/contact and I can send you a beta version of an updated plugin that should solve this.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic

Tags

No tags yet.