Hi there!
From my end, everything seems fine. The recurring problem I find is that common robotic user-agents are (incorrectly) blocked. This could be either via the robots.txt file or via various server (.htaccess) rules.
See the latest known robots.txt version by Google here. If the output doesn’t look like the one from theseoframework.com (reference), then hit the “Submit” button at the bottom-right, and then select “Submit” again.
If that wasn’t the case, could you double-check your server’s configuration (with your hosting provider?) to see if Google can access your website? You can learn how to test this here: https://support.google.com/webmasters/answer/2387297?hl=en
Thread Starter
linzD
(@linzd)
Thanks for the note.
I’ve asked my host (Siteground) and they say that the the Googlebots can search the page OK. They even showed me on AW Stats that the wider site is being crawled.
I’ve tried to use the robots testing tool, but it seems to be broken – when I got there it doesn’t show my ‘property’. When I click ‘add property’ it shows it but doesn’t give me anyway to use it within the robots testing tool.
I’m at a bit of a loss – any other ideas please?
Hi again 🙂
Thanks for writing out your journey. Google did away with the old Search Console last year and will close it by the end of this month. Not all their tools have been migrated into their new interface, so the Search Console is a bit of a maze. So, I suggest following the link I shared again, now you’ve added your property, to access the “robots.txt Tester”.
To follow up, have you recently published your website, and did you recently turn off the “Discourage search engines from indexing this site” option at /wp-admin/options-reading.php?
It can take some time for them to engage on your website if the above is true. Keep an eye on their “robots.txt Tester” results, when it matches your live robots.txt file, ask them to crawl your website if they haven’t already.
