Support » Fixing WordPress » Robots.txt Fetch Error in Google webmaster tools

  • Hey All
    i have a problem with the robots.txt file, Google can not crawl it.
    i have moved my site to a new server few days back.
    check the robots.txt at : http://technoriderz.com/robots.txt

    what should i do?? please guide me about this error.
    my site traffic has fallen to 20 unique visitors per day from 2k.
    🙁 🙁 🙁

Viewing 7 replies - 1 through 7 (of 7 total)
  • You could set the float on #main to the left. Do you know how to do it? How familiar are you with child themes? Do you know how to edit your CSS file(s)?

    i did not understand what to do 🙁

    i have a problem with the robots.txt file, Google can not crawl it.

    What’s the problem? What error message did Google post in your GWT account?

    i have moved my site to a new server few days back.

    Did you change any links/URLs?

    BTW, you have to take up crawling, indexing, traffic, etc. with search engines such as Google as these are neither WordPress problems nor WordPress has any control on such matters.

    What’s the problem? What error message did Google post in your GWT account?

    Google couldn’t crawl your site because we were unable to access the robots.txt file.
    i see this message in GWT account in site errors.
    DNS connectivity, server connectivity shows green signal, while robots.txt fetch show yellow error icon.

    Did you change any links/URLs?

    No, no URLs are changed, URLs and the domain is the same.
    also ib GWT in the blocked URLs section it shows the old (wrong) robots.txt code even after i have udated the robots.txt
    sir please guide me, as i m new and don’t know now what to do.
    .
    thanks in advance.

    You may replace your robots.txt file that is currently:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /cgi-bin/
    
    Allow: /
    Sitemap: http://technoriderz.com/sitemap.xml

    With

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /cgi-bin/
    
    Sitemap: http://technoriderz.com/sitemap.xml

    the above one. Then submit a new sitemap and wait for the results. You need to regularly look for errors and correct them at your GWT. The problems and solutions for them can be found at your GWT account and the help links there. As I mentioned before, this is NOT a WordPress problem.

    Thanks Dear Sir, I have updated my robots.txt as you suggested.
    i understand that it is not a WP wrror, but here i was waiting for experts to help me In GWT ac.

    Thanks for your encouraging reply and help.

    Hello,
    Friends I am also facing this issue.te robots.txt file is showing like that.

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Sitemap: https://www.csofts.net/sitemap.xml.gz

    thanks
    Web hosting Provider
    I am waiting your reply.

Viewing 7 replies - 1 through 7 (of 7 total)
  • The topic ‘Robots.txt Fetch Error in Google webmaster tools’ is closed to new replies.