Bots hitting filters

  • Resolved davidburrows

    (@davidburrows)


    1 week, 6 days ago

    Is there any way to stop bots hitting our filter URLs so hard? Applebot in particular has brought our site to it’s knees, consuming 142.33 GB in the past couple of weeks by slamming every combination in our colour filterand it continues to do so.

    I added the following to robots.txt a few days ago

    User-agent: *
    Disallow: /*?_colour=
    	 
    User-agent: Applebot
    Crawl-delay: 10

    But it still seems to rattling through URLs containing “GET /shop/?_colour=* at a rate of 2 per second

    Is blocking via .htaccess the only option?

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support fesupportteam

    (@fesupportteam)

    1 week, 6 days ago

    Hi @davidburrows

    Yes, you can stop this without relying solely on .htaccess. The core issue is that robots.txt can take time for crawlers to process, and Applebot typically ignores the Crawl-delay directive entirely. It only respects standard directives such as Disallow.

    The fastest solution is to fix your robots.txt configuration and block the requests at the Cloudflare/CDN level. If you use a reverse proxy or CDN such as Cloudflare, AWS CloudFront, or Fastly, you can block or rate-limit these requests at the edge before they ever reach your server.

    1. Fix your robots.txt syntax
    Applebot does respect standard Disallow directives, but it ignores Crawl-delay. Explicitly disallow Applebot from crawling URLs containing the _colour= query parameter.

    2. Edge Firewall Rule
    If you’re using Cloudflare, create a WAF Custom Rule with the required configuration, you can scope the rule specifically to requests where the User-Agent contains Applebot.

    This approach is faster and more reliable because the request is blocked at the edge, before it reaches your origin server.

    Best regards,
    Victor

    Plugin Author stepasyuk

    (@stepasyuk)

    1 week ago

    A follow-up: Filter Everything 1.9.6 (free), released today, can now generate the robots.txt Disallow rules for you — Settings → General → Crawlers and bots → «Block filter URLs in robots.txt». It covers every filter parameter (including the _colour= one from your logs), so you don’t need to maintain the syntax by hand — and Applebot respects Disallow, so the bandwidth drain should stop once its crawl queue drains.

    The «Disable filter links for crawlers» option is also available in the free version now, so no new filter URL combinations get discovered.

    Thread Starter davidburrows

    (@davidburrows)

    1 week ago

    Thanks for the update. I have enabled the Disable filter links toggle.

    Unfortunately, although we had added the correct syntax to robots.txt, our website was consistently unavailable due to the colour filter being hammered by both good and bad bots.

    I have removed the colour filter from the filter itself, and added a Cloudflare rule to block any user accessing anything with _colour in the URL which seems to be the only way to bring our site back up.

    Hopefully we can enable the filter again in a couple of weeks and ditch the Cloudflare rule.

Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.