Support » Plugin: WP Web Scraper » Use this plugin cautiously!

  • I was using this plugin to scrape prices from amazon. Apparently every time the plugin is called, it downloads the entire webpage that you are scraping and stores it in the wp_options table. This would be fine if the plugin would delete those stored pages after a certain amount of time but it doesn’t! 🙁

    MySQL kept crashing (and taking my dedicated server down with it) because the wp_options for my site grew to 1.3GB (yes, GIGABYTE) whereas this table should normally be less than 1MB. Basically, every time someone visited a page on my site, this plugin was adding a new row of ~300kb of data to the wp_options table. YIKES!

    If you have a small site, then maybe this plugin would be okay, but for a larger site that needs to scrape data, this plugin in its current state is not the best solution.

Viewing 8 replies - 1 through 8 (of 8 total)
  • Is this still correct with the current version?

    If you set the Cache expiry, does it delete it from the database?

    Plugin Author Akshay Raje


    @dorianj Thanks for bringing this up. May be I should just revert to file based cache or possibly use the native object cache class of WordPress to avoid all such issues.

    So in the mean time how do we clear the current cache?



    I also would like to know the answer to this.

    Storing the info in a custom meta field might be a good answer.

    Just set cron to remove transient files after x time period.
    However, the ability to instead store the data in custom fields would be HUGE!! This plugin would be worth major dollars if you could store the data in a custom field.

    My hosting provider just suspended my account because of this, I’ve been looking all around for a solution and when I find it I can’t understand it :cry:!

    @badjesus: Where are this files stored and how can I erase them?

    Could you please explain me how can I do to empty all the old data daily or weekly?

    All my sites are down until I fix this, they say is too “resource hungry” for a shared account.


Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Use this plugin cautiously!’ is closed to new replies.