WP Web Scraper
Use this plugin cautiously! (9 posts)

  1. dorianj
    Posted 3 years ago #

    I was using this plugin to scrape prices from amazon. Apparently every time the plugin is called, it downloads the entire webpage that you are scraping and stores it in the wp_options table. This would be fine if the plugin would delete those stored pages after a certain amount of time but it doesn't! :-(

    MySQL kept crashing (and taking my dedicated server down with it) because the wp_options for my site grew to 1.3GB (yes, GIGABYTE) whereas this table should normally be less than 1MB. Basically, every time someone visited a page on my site, this plugin was adding a new row of ~300kb of data to the wp_options table. YIKES!

    If you have a small site, then maybe this plugin would be okay, but for a larger site that needs to scrape data, this plugin in its current state is not the best solution.


  2. Jay Collier
    Posted 3 years ago #

    Is this still correct with the current version?

  3. reportica
    Posted 3 years ago #

    If you set the Cache expiry, does it delete it from the database?

  4. Akshay Raje
    Plugin Author

    Posted 3 years ago #

    @dorianj Thanks for bringing this up. May be I should just revert to file based cache or possibly use the native object cache class of WordPress to avoid all such issues.

  5. square_eyes
    Posted 3 years ago #

    So in the mean time how do we clear the current cache?

  6. websupportguy
    Posted 3 years ago #

    I also would like to know the answer to this.

  7. WebPixie
    Posted 3 years ago #

    Storing the info in a custom meta field might be a good answer.

  8. badjesus
    Posted 3 years ago #

    Just set cron to remove transient files after x time period.
    However, the ability to instead store the data in custom fields would be HUGE!! This plugin would be worth major dollars if you could store the data in a custom field.

  9. Johana
    Posted 2 years ago #

    My hosting provider just suspended my account because of this, I've been looking all around for a solution and when I find it I can't understand it :cry:!

    @badjesus: Where are this files stored and how can I erase them?

    Could you please explain me how can I do to empty all the old data daily or weekly?

    All my sites are down until I fix this, they say is too "resource hungry" for a shared account.


Topic Closed

This topic has been closed to new replies.

About this Plugin

  • WP Web Scraper
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic


No tags yet.