{"id":19030151,"date":"2026-09-26T17:54:47","date_gmt":"2026-09-26T17:54:47","guid":{"rendered":"https:\/\/wordpress.org\/support\/topic\/allow-custom-user-agent-for-wordpress-rest-api\/"},"modified":"2026-09-26T17:54:47","modified_gmt":"2026-09-26T17:54:47","slug":"allow-custom-user-agent-for-wordpress-rest-api","status":"publish","type":"topic","link":"https:\/\/wordpress.org\/support\/topic\/allow-custom-user-agent-for-wordpress-rest-api\/","title":{"rendered":"Allow custom User-Agent for WordPress REST API"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">We need our SEO tooling to call the WordPress REST API (<code>\/wp-json\/*<\/code>) with Application Passwords on two sites protected by CleanTalk (Anti-Crawler \/ SpamFireWall):<\/p>\n\n\n\n<ul>\n<li><a href=\"https:\/\/olightsa.co.za\/\">https:\/\/olightsa.co.za\/<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/osight.africa\/\">https:\/\/osight.africa\/<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What works elsewhere<\/strong><br \/>On <a href=\"https:\/\/torchsa.com\/\">https:\/\/torchsa.com\/<\/a> we allowlisted User-Agent wildcard <code>*WeblabSEOBot*<\/code> at the edge (Cloudflare). Requests using:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>WeblabSEOBot\/1.0 (+https:\/\/theweblab.co.za; SEO agent)<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">then get 200 on the homepage, <code>\/wp-json\/<\/code>, and authenticated <code>\/wp-json\/wp\/v2\/users\/me<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What fails on CleanTalk<\/strong><br \/>The same User-Agent still receives CleanTalk\u2019s \u201cBlocked: Security by CleanTalk\u201d 403 on <code>\/wp-json\/<\/code> (and often <code>\/robots.txt<\/code> on osight.africa). Homepages can load; REST does not.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We tried Personal List CSV IP allow (<code>140.248.50.53,allow<\/code>), but our bot egress IPs rotate, so a single IP whitelist is unreliable. Your docs also say custom User-Agents cannot be added to the Anti-Crawler trusted list ourselves \u2014 only the built-in bot list.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Request<\/strong><br \/>Please either:<\/p>\n\n\n\n<ol>\n<li>Add <code>WeblabSEOBot<\/code> (match substring \/ wildcard <code>*WeblabSEOBot*<\/code>) to the trusted \/ allowed User-Agents for these two websites, or<\/li>\n\n\n\n<li>Tell us the supported way to allow this custom User-Agent for SpamFireWall \/ Anti-Crawler so REST API traffic is not challenged or blocked.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">We only need read\/write via official WordPress Application Passwords; not form spam bypass.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thank you.<\/p>\n","protected":false},"template":"","class_list":["post-19030151","topic","type-topic","status-publish","hentry"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic\/19030151","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic"}],"about":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic\/19030151\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/media?parent=19030151"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}