AHREFS crawling issue

Hi @borkk85 ,

Thanks for reaching out, and I appreciate the detailed information!

Just to clarify — AIOSEO doesn’t generate a physical robots.txt file on your server. Instead, it adds rules to the default virtual robots.txt file that WordPress dynamically serves at yourdomain.com/robots.txt using the robots_txt filter.

So if you’re seeing a 403 error when tools like Ahrefs try to access it, that usually points to a server-level or security rule blocking the request — not something coming from AIOSEO.

Regarding the BadBotBlocker.php file you mentioned — you’re right that it exists in our codebase for backward compatibility, but the Bad Bot Blocker feature is deprecated and no longer active for new installs.

If you don’t see a “Bad Bot Blocker” tab under All in One SEO > Tools, then it’s not enabled and not affecting anything on your site. In that case, it’s safe to ignore that file.

That said, a 403 response for robots.txt typically comes from:

• A firewall or security layer (e.g., server rules like ModSecurity)
• Host-level restrictions (even if Ahrefs IPs are whitelisted)
• Custom .htaccess rules

I’d recommend asking your host to double-check any server-level firewall, security modules, or .htaccess rules that may be denying access to robots.txt requests — even from known bots like Ahrefs.

Let me know what you find and I can take a closer look.