Title: Excessive Crawling / CPU Load Last modified: July 29, 2026 --- # Excessive Crawling / CPU Load * Resolved [boosted24](https://wordpress.org/support/users/boosted24/) * (@boosted24) * [1 week, 2 days ago](https://wordpress.org/support/topic/excessive-crawling-cpu-load/) * Hi, * I’ve been using the free version of this plugin for around 18 months. * Most of that time the plugin worked without any issues. About 3 months ago my site started getting very slow to load pages, the hosting CPU usage was exceeding its limits and I was getting repeated 508 errors. In other words, the website had become unusable. * I enabled Cloudflare Under Attack Mode and it helped temporarily, but every time I turned it off the same problem came back. * After a lot of intermittent back and forth with my hosting company, we eventually found last night that the load was being caused by an excessive number of bots crawling filter URLs created by the plugin. I have now set Cloudflare rules to block those requests and the website load times has noticeably improved. * When I checked the plugin settings, I noticed that the Pro version now has an option called “Disable filter links for crawlers”. The description says this replaces `` tags with `` tags to stop crawlers following the filter links and overloading the website, so this appears to be a known issue. * I have a few questions. - Has the free version always used crawlable `` links for filters, or has this changed at some point over the last 3–4 months? - If it has changed, when was it changed and why? - If it has always worked this way, why wasn’t protection against excessive crawling included by default? This issue has now caused me around 3 months of poor website performance and most likely lost sales plus the time I have put into finding the issue. As the plugin had worked perfectly for around 15 months beforehand, I had no reason to suspect it was the cause. - Why is the solution to a problem that can make a website unusable only available in the paid version, when the free version directly creates the problem in the first place? * From my point of view, the free version creates an almost unlimited number of crawlable filter URLs. Bots then follow those URLs, which causes excessive server load, high CPU usage and 508 errors. The Pro version then offers the option to stop crawlers following those same links when its the plugin that caused it to happen. * I’d appreciate it if you could explain if this behaviour has changed over time and whether there is any supported way to prevent this excessive crawling while continuing to use the free version, as I don’t really require any of the other Pro features. I would have no issue buying the PRO version as a one off, but a yearly payment just to stop something directly caused by the free version seems a bit excessive. * Finally, could you also explain whether replacing `` tags with `` tags provides any other benefits to website performance, SEO or general site functionality beyond reducing crawler activity? * Thank you. Viewing 1 replies (of 1 total) * Plugin Support [fesupportteam](https://wordpress.org/support/users/fesupportteam/) * (@fesupportteam) * [1 week, 2 days ago](https://wordpress.org/support/topic/excessive-crawling-cpu-load/#post-18978503) * Hi [@boosted24](https://wordpress.org/support/users/boosted24/) * Thank you for the detailed explanation. * The filters in the plugin have always used tags for filter links. This behavior has not changed in the last few months. You can also verify this yourself by downloading previous versions of the plugin from the WordPress repository here: [https://wordpress.org/plugins/filter-everything/advanced/](https://wordpress.org/plugins/filter-everything/advanced/)( at the bottom of the page). * The Pro version includes an additional option in the latest updates that allows you to replace filter tags with tags. * Regarding why bots started crawling your site only recently, search engines and other crawlers constantly discover new URLs, adjust their crawling patterns, and revisit websites at different frequencies. A URL structure that was not heavily crawled before can naturally start receiving significantly more crawler traffic later, even without any changes being made to the website. * Since the filters use GET parameters, we recommend configuring your robots.txt file to manage crawling of these parameter-based URLs. This helps guide bots away from unnecessary URLs and prevents high CPU spikes on your server. * In general, the recommended approach is to control crawler access through your robots.txt file and your server/CDN settings (such as Cloudflare rules), depending on your website’s specific needs. * With the growth of AI-powered bots, this situation is increasing globally, as the amount of automated crawling traffic continues to grow and, in some cases, exceeds human traffic. Additionally, these bots are becoming more advanced and can mimic human behavior patterns, including interacting with website elements and using filters on pages in a way that appears similar to a real user. * Best regards, Victor Viewing 1 replies (of 1 total) You must be [logged in](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fwordpress.org%2Fsupport%2Ftopic%2Fexcessive-crawling-cpu-load%2F%3Foutput_format%3Dmd&locale=en_US) to reply to this topic. * ![](https://ps.w.org/filter-everything/assets/icon.svg?rev=3570563) * [Filter Everything — WordPress & WooCommerce Filters](https://wordpress.org/plugins/filter-everything/) * [Frequently Asked Questions](https://wordpress.org/plugins/filter-everything/#faq) * [Support Threads](https://wordpress.org/support/plugin/filter-everything/) * [Active Topics](https://wordpress.org/support/plugin/filter-everything/active/) * [Unresolved Topics](https://wordpress.org/support/plugin/filter-everything/unresolved/) * [Reviews](https://wordpress.org/support/plugin/filter-everything/reviews/) * 1 reply * 2 participants * Last reply from: [fesupportteam](https://wordpress.org/support/users/fesupportteam/) * Last activity: [1 week, 2 days ago](https://wordpress.org/support/topic/excessive-crawling-cpu-load/#post-18978503) * Status: resolved