Hi @123nadav, thanks for contacting us about this.
What did Live Traffic report at the time the request to the chatbot was sent? Wordfence will always log a block made and show the reason here. You may be able to allow it immediately due to the feedback given there.
Many thanks,
Peter.
@wfpeter I checked the block list, nothing there…
i tried to do live traffic and look on the log in the same time i order gemini and chatgpt to read the:
https://locksmithunit.com/llms.txt
but i don’t see any traffic. is very wired. any solutions?
Hi @123nadav,
I tried to access the file via a paid ChatGPT account, regular free ChatGPT, and Claude to see if I was blocked from seeing it. I thought a block in one of these services might be able to tell me more about the reason it was being blocked. All 3 attempts succeeded and it returned a summary of the file’s contents, how it was generated, and what it is for.
Gemini on a mobile device returned a similar response to the one you were seeing but stated the server response couldn’t be returned as HTTP status codes and headers aren’t designed to be returned in its browsing tool. If Wordfence was blocking access to the file, I would expect all of the AI attempts to fail or at least the service that was being blocked to have its IP and reason logged in Live Traffic. How does it do when Wordfence is temporarily disabled or set to Learning Mode?
It looks like it may be stopped instead by a server firewall or CDN (possibly due to its User-Agent if Claude/ChatGPT are making it through from my side), or it’s adhering strictly to something in robots.txt about Google crawlers.
Thanks,
Peter.
I Don’t use any CDN… my server is OpenLiteSpeed server on google cloud:
https://docs.litespeedtech.com/cloud/images/wordpress/?_gl=11h67vol_gcl_au*MzA3MTQ1OTgwLjE3NTc1NjAzNzk.#__tabbed_1_4
When i tried from free chatgpt i got block as well… you said from your side its look ok, sound wired.
I will try to talk with litespeed support, but i pretty sure is not a server problem.
I need to try turn off wordfance, but tell you the truth i prefer not to do it.
Hi @123nadav,
Yes I tried being logged in and also incognito just with the free version of ChatGPT and it returned similar information as Claude about the file.
It does seem like a bot/User-Agent restriction could be happening for Gemini/Google but without it being able to feed back what kind of blocking page it saw it hasn’t really narrowed it down. It could certainly be worth a temporary deactivation of Wordfence for a few minutes would help understand if it’s involved at all. Just make sure not to delete tables/data when asked so that reactivation is smooth with all of your existing settings in tact.
Thanks again,
Peter.
