Robots.txt files causing problems with Webmaster Tools

  • cloncallowelk

    (@cloncallowelk)


    9 years, 5 months ago

    Hi there,

    My Fetch as Google test on Webmaster Search Console keeps coming back as ‘Temporarily Unavailable’because of a problem with the robots.txt file.

    I contacted my host about it. They checked the DNS and said they fixed some small error with that. Their only advice about the robot.txt file situation was to deactivate the SEO plugin – I did that and nothing has changed.

    When I search for kitchenandlarder.ie/robots.txt I just get my site. And when I search for kitchenandlarder.ie/wp/robots.txt I just get a ‘page not found’ result on my site.

    I cannot find any other way to access this mysterious robots.txt file via my dashboard – And still don’t know what to do if I did find it!

    I would appreciate any help as I am at my wit’s end with this problem!

    These are the messages I am getting from Webmaster Tools.

    In ‘Crawl errors’ it says ‘Google couldn’t crawl your site because we were unable to access your site’s robots.txt file. More info.’

    In ‘robots.txt fetch’ its says
    ‘failed. You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file. Learn more.’

    When I test the Sitemap it says
    ‘Network unreachable: robots.txt unreachable. We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.’

    I really hope somebody has the answer to this for me! 🙂

    Thank you,
    Ella

Viewing 2 replies - 1 through 2 (of 2 total)
  • snakedeshazer

    (@snakedeshazer)

    9 years, 5 months ago

    I have seen this a few times, I used to work for a hosting company, and usually it would be resolved by enabling it through Mod_security. For some reason your server may see googles bot as malicious and block it out right. If this is the case, you would need to confirm with your host, it may even require higher level of permissions, at least it did where I worked.

    Hope that helps!

    Thread Starter cloncallowelk

    (@cloncallowelk)

    9 years, 5 months ago

    Ok thank you. My hosts are a bit rubbish at replying to emails and being helpful. I think it may be time to move!

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Robots.txt files causing problems with Webmaster Tools’ is closed to new replies.