• We need our SEO tooling to call the WordPress REST API (/wp-json/*) with Application Passwords on two sites protected by CleanTalk (Anti-Crawler / SpamFireWall):

    What works elsewhere
    On https://torchsa.com/ we allowlisted User-Agent wildcard *WeblabSEOBot* at the edge (Cloudflare). Requests using:

    WeblabSEOBot/1.0 (+https://theweblab.co.za; SEO agent)

    then get 200 on the homepage, /wp-json/, and authenticated /wp-json/wp/v2/users/me.

    What fails on CleanTalk
    The same User-Agent still receives CleanTalk’s “Blocked: Security by CleanTalk” 403 on /wp-json/ (and often /robots.txt on osight.africa). Homepages can load; REST does not.

    We tried Personal List CSV IP allow (140.248.50.53,allow), but our bot egress IPs rotate, so a single IP whitelist is unreliable. Your docs also say custom User-Agents cannot be added to the Anti-Crawler trusted list ourselves — only the built-in bot list.

    Request
    Please either:

    1. Add WeblabSEOBot (match substring / wildcard *WeblabSEOBot*) to the trusted / allowed User-Agents for these two websites, or
    2. Tell us the supported way to allow this custom User-Agent for SpamFireWall / Anti-Crawler so REST API traffic is not challenged or blocked.

    We only need read/write via official WordPress Application Passwords; not form spam bypass.

    Thank you.

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support amagsumov

    (@amagsumov)

    Hello @weblabza,

    Unfortunately, there is currently no option to allow custom User-Agents in Anti-Crawler. Furthermore, the “Blocked: Security by CleanTalk” response indicates that our Security Firewall stopped the request, and it does not support custom User-Agent lists.

    I searched your SpamFireWall and Security logs and found only a single IP address belonging to the WeblabSEOBot User-Agent: 104.30.180.111.

    In the meantime, please try adding the entire subnet 104.30.180.0/22 to your SpamFireWall and Security allow lists.

    We will discuss potential improvements with our development team and update you within 2–3 business days.

    Plugin Support amagsumov

    (@amagsumov)

    Hello @weblabza,

    We reviewed the issue with our team. Right now, the only available solution is to add your IP address or IP range to the allowlist in SpamFireWall and Security FireWall.

    Thread Starter weblabza

    (@weblabza)

    Ok thank you. I will try the IP range and see if this works.

    Plugin Support denisxam

    (@denisxam)

    Hello @weblabza
    Thank you for your response.

    Please keep us posted if this helped you.

Viewing 4 replies - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.