Python aiohttp bots can’t be blocked?

  • Resolved cybeardjm

    (@didierjm)


    1 week, 6 days ago

    Hi,

    Been using AIOS to block all kind of bots and crawlers, through different methods:
    – via robots.txt (classic way, for those that respect it),
    – via IP ranges through Firewall / Block & Allow Lists (it seems the UserAgent part doesn’t work at all), that’s why I moved to the option below,
    – via a custom .htaccess rule checking UserAgents.
    They usually work fine, but I can’t block one kind of bot, although I deny the various IPs it comes from and the UserAgent: “Python/3.13 aiohttp/3.14.1” (at least it’s not a fake UA).
    It seems to be able to bypass all rules, and instead of a 403, it receives a HTTP 200 OK accessing all images (JPEG and Webp) from the /uploads folder. But the IP range is blocked in theory!
    I suppose it uses a direct access through the Apache server, that bypasses WordPress as a whole.
    But access to /wp-content/uploads/YYYY/MM/ is forbidden (You don’t have permission to access this resource.), meaning the bot probably (my hypothesis) finds the info by scanning HTML pages then gets direct access to it knowing/deducing the proper URL.
    That’s very annoying as I get a lot of requests from this bot using various IPs all day long and so far, I can’t do anything…

    Is there a solution here to protect /uploads via AIOS? Thx.

    Sincerely
    DJM

Viewing 8 replies - 1 through 8 (of 8 total)
  • Plugin Support hjogiupdraftplus

    (@hjogiupdraftplus)

    1 week, 6 days ago

    Hi @didierjm,

    Generally, on Apache servers that support .htaccess, WordPress .htaccess rules allow the web server to serve static files (such as JPEG images) directly. These requests do not pass through WordPress as dynamic content, so AIOS firewall rules cannot restrict them.

    However, you can add a custom .htaccess rule under AIOS > Tools > Custom .htaccess Rules to restrict access for this specific user agent:

    Please back up the .htaccess file before making this change.

    # Block Python aiohttp requests for image files
RewriteCond %{HTTP_USER_AGENT} "Python/3\.13 aiohttp/3\.14\.1" [NC]
RewriteRule \.(?:jpg|jpeg|png|gif|webp|svg|ico)$ - [F,L]

    This rule will return a 403 Forbidden response when a request for one of the specified image file types is made with the Python/3.13 aiohttp/3.14.1 user agent.

    Regards

    Thread Starter cybeardjm

    (@didierjm)

    1 week, 5 days ago

    Thx, will try it.
    DJM

    Plugin Support hjogiupdraftplus

    (@hjogiupdraftplus)

    1 week, 5 days ago

    Hi @didierjm,

    You’re welcome! Sounds good. Let me know how it goes.

    Would you mind writing a quick five-star review on wordpress.org?

    https://wordpress.org/support/plugin/all-in-one-wp-security-and-firewall/reviews/#new-post

    Reviews also help others to make confident decisions about our plugin.

    Regards

    Thread Starter cybeardjm

    (@didierjm)

    1 week, 4 days ago

    OK, it clearly doesn’t work. All reqs still get a HTTP 200.

    I edited the code, as Bingbot is doing the same:

    # Block Python requests for image files
    RewriteCond %{HTTP_USER_AGENT} "Python/3\.13 aiohttp/3\.14\.1" [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} "bingbot/2\.0" [NC]
    RewriteRule .(?:jpg|jpeg|png|gif|webp|svg|ico)$ - [F]

    FYI, it’s my second custom rule at the top of .htaccess. The first one is:

    RewriteCond %{HTTP_USER_AGENT} (bot1|bot2|...|botx) [NC]
    RewriteRule ^ - [F]

    The list of bots also includes aiohttp, BTW, and it seems to usually work.

    DJM

    Plugin Support hjogiupdraftplus

    (@hjogiupdraftplus)

    1 week, 3 days ago

    Hi @didierjm,

    Do you have server-level caching, Cloudflare, or anything similar enabled? If so, the rule should be placed at that level.

    The .htaccess check is applied to the exact image URL. For example, /image.jpg will be blocked, but /image.jpg?size=large may not be. This needs to be checked.

    The working rule blocks any request whose User-Agent matches one of the specified bot names. That is difference please corss-check.

    Regards

    Thread Starter cybeardjm

    (@didierjm)

    1 week, 3 days ago

    Duplicate msg – deleted

    Thread Starter cybeardjm

    (@didierjm)

    1 week, 3 days ago

    Thx for your reply,
    I edited the syntax for RewriteRule and it now works fine, sending a 403 to the Python bots. I also added a no-hotlinking block, just in case (although I know it’s active via AIOS settings already, mine is less complex using SetEnvIf and covers more image formats).

    SetEnvIf Referer sample\.server\.com localreferer
    <FilesMatch ".(jpg|jpeg|png|gif|webp|svg|ico)$">
    Require env localreferer

    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} "Python/3\.13 aiohttp/3\.14\.1" [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} "Python/3\.12 aiohttp/3\.13\.3" [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} "bingbot/2\.0" [NC]
    RewriteRule ".(jpg|jpeg|png|gif|webp|svg|ico)$" "-" [F,NC]

    DJM

    Plugin Support hjogiupdraftplus

    (@hjogiupdraftplus)

    1 week, 2 days ago

    Hi @didierjm,

    Glad to know edited syntax rule worked.

    Regards

Viewing 8 replies - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.