/page/xx scrapers?

Hello @ambientblog,

Dealing with aggressive scrapers and crawlers hitting pagination URLs (like /page/xx/) is a very common battle! Fortunately, Advanced IP Blocker has several built-in layers specifically designed to stop this kind of automated traffic without hurting your real users.

Here is the recommended configuration to clean up your traffic:

1. Enable the Community Defense Network Go to Settings > AIB Community Defense Network and turn on all the switches. Scrapers often use the same IP ranges across multiple sites; this feature will automatically block IPs that have already been flagged as malicious by our community.

2. Enable User-Agent Protection Go to Settings > User-Agent Protection and make sure you activate:

• Verify Known Bots
• Verify AI Bots (CIDR)
• Verify Monitoring Bots (IP List) This ensures that bots claiming to be legitimate search engines are actually real. It will drop fake bots and aggressive unverified crawlers immediately.

3. Set up a JavaScript Challenge (The ultimate scraper killer) If these scrapers are triggering your security rules, go to the Attack Signature Engine and configure the mitigation action to JS automatic challenge / transparent (or use Turnstile / hCaptcha if you have them configured). Why this works: Most automated scrapers and dumb crawler scripts cannot execute JavaScript. They will fail the challenge and be blocked, while your real human visitors will pass the transparent challenge seamlessly without even noticing.

4. Block Specific Scrapers Manually If you’ve checked your analytics and identified a specific unwanted scraper (for example, Sogou web spider/4.0...), you can block it directly. Go to Blocking Rules > User Agents and simply add a keyword like Sogou to your blocked list.

A quick reminder about your stats: Please keep in mind that seeing some crawler traffic on your /page/xx/ URLs in your stats is completely normal. Legitimate search engines (like Googlebot or Bingbot) need to crawl your pagination to index your older posts properly. As long as you have “Verify Known Bots” enabled (Step 2), the good bots will do their job safely, and the malicious scrapers will be kept out.

Hope this helps you clean up your traffic! Let us know if you need any help setting this up.