WordPress.org

Ready to get started?Download WordPress

Forums

WP Web Scraper
Use this plugin cautiously! (9 posts)

  1. dorianj
    Member
    Posted 1 year ago #

    I was using this plugin to scrape prices from amazon. Apparently every time the plugin is called, it downloads the entire webpage that you are scraping and stores it in the wp_options table. This would be fine if the plugin would delete those stored pages after a certain amount of time but it doesn't! :-(

    MySQL kept crashing (and taking my dedicated server down with it) because the wp_options for my site grew to 1.3GB (yes, GIGABYTE) whereas this table should normally be less than 1MB. Basically, every time someone visited a page on my site, this plugin was adding a new row of ~300kb of data to the wp_options table. YIKES!

    If you have a small site, then maybe this plugin would be okay, but for a larger site that needs to scrape data, this plugin in its current state is not the best solution.

    http://wordpress.org/extend/plugins/wp-web-scrapper/

  2. Jay Collier
    Member
    Posted 1 year ago #

    Is this still correct with the current version?

  3. reportica
    Member
    Posted 1 year ago #

    If you set the Cache expiry, does it delete it from the database?

  4. Akshay Raje
    Member
    Plugin Author

    Posted 1 year ago #

    @dorianj Thanks for bringing this up. May be I should just revert to file based cache or possibly use the native object cache class of WordPress to avoid all such issues.

  5. square_eyes
    Member
    Posted 1 year ago #

    So in the mean time how do we clear the current cache?

  6. websupportguy
    Member
    Posted 1 year ago #

    I also would like to know the answer to this.

  7. WebPixie
    Member
    Posted 11 months ago #

    Storing the info in a custom meta field might be a good answer.

  8. badjesus
    Member
    Posted 10 months ago #

    Just set cron to remove transient files after x time period.
    However, the ability to instead store the data in custom fields would be HUGE!! This plugin would be worth major dollars if you could store the data in a custom field.

  9. Johana
    Member
    Posted 9 months ago #

    My hosting provider just suspended my account because of this, I've been looking all around for a solution and when I find it I can't understand it :cry:!

    @badjesus: Where are this files stored and how can I erase them?

    Could you please explain me how can I do to empty all the old data daily or weekly?

    All my sites are down until I fix this, they say is too "resource hungry" for a shared account.

    Thanks!!!

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic

Tags

No tags yet.