Hi @kvm1!
If you browse to one of those URLs what are you seeing? You should be seeing a blank page. Occasionally we see themes that implement the WordPress template redirect hook incorrectly which results in all requests that have a query string redirecting to the sites frontpage. In those cases, the wordfence_lh URL will return the full website instead of a blank page. This will cause Google to think it’s an actual page it should index. There is also a known issue with WPX hosting. They block all wordfence_lh requests. If you are on WPX hosting you can either disable Wordfence Live Traffic or you can reach out to WPX hosting and ask that they remove the block.
Thread Starter
mIPs
(@kvm1)
Hi @wfasa,
if I browse to one of the URLs, I see a blank page. We are not hosting on WPX hosting. So, if I understand you right, I should speak with the developers of the theme?
Hi @kvm1,
If you are seeing a blank page when you visit those URLs then everything is as it should be. You do not have to reach out to your theme devs.
It has to be something else going wrong then. What exactly is the error you are getting from Google?
Thread Starter
mIPs
(@kvm1)
Hi @wfasa,
this is the error description I see (translated from german with google translator):
“Googlebot could access your website, but certain URLs are blocked for the Googlebot mobile for smartphones in your robots.txt file. These URLs do not appear in the Blocked URLs report and may have been unintentionally blocked.”
Hi again,
So it’s saying that it’s being blocked via your robots.txt file when browsing via mobile. Have you checked the robots.txt file to see if there is anything in there that shouldn’t be there?
Thread Starter
mIPs
(@kvm1)
Hey @wfasa,
this is the content of the robots.txt:
# global
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Disallow: /readme.html
User-agent: Cliqzbot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
Disallow: /widerrufsbelehrung
Disallow: /support/supportformular
I do not see anything suspicious there!?
You are disallowing all requests with query strings from all user agents including Google.
This means “rules for everyone including Google”
User-agent: *
And these rules say “anything with ? (query string) is disallowed
Disallow: /*?*
Disallow: /*?
The /?wordfence_lh follows that pattern so that would be why it’s blocked.
I’ll be honest I’m not an SEO expert but I don’t think it should be necessary to block anything for Google bot. It should behave and not crawl things it’s not supposed to.
-
This reply was modified 7 years, 8 months ago by
wfasa.
Thread Starter
mIPs
(@kvm1)
That makes a lot of sense. I’ve removed the “Disallow: /*?*” and kept the “Disallow: /*?”, and will monitor if this brings any change to the Search Console error log. I’ll write a reply with the results on monday.
Thread Starter
mIPs
(@kvm1)
Hi @wfasa,
I removed the lines
Disallow: /*?
Disallow: /*?*
It took some days, but now there are no more errors reported.
Thanks for your assistance.
That’s good news. Thanks for the update @kvm1!
