WordPress.org

Forums

Robots.txt Fetch Error in Google webmaster tools (8 posts)

  1. Heeran
    Member
    Posted 1 year ago #

    Hey All
    i have a problem with the robots.txt file, Google can not crawl it.
    i have moved my site to a new server few days back.
    check the robots.txt at : http://technoriderz.com/robots.txt

    what should i do?? please guide me about this error.
    my site traffic has fallen to 20 unique visitors per day from 2k.
    :( :( :(

  2. Borko
    Member
    Posted 1 year ago #

    You could set the float on #main to the left. Do you know how to do it? How familiar are you with child themes? Do you know how to edit your CSS file(s)?

  3. Heeran
    Member
    Posted 1 year ago #

    i did not understand what to do :(

  4. Krishna
    Volunteer Moderator
    Posted 1 year ago #

    i have a problem with the robots.txt file, Google can not crawl it.

    What's the problem? What error message did Google post in your GWT account?

    i have moved my site to a new server few days back.

    Did you change any links/URLs?

    BTW, you have to take up crawling, indexing, traffic, etc. with search engines such as Google as these are neither WordPress problems nor WordPress has any control on such matters.

  5. Heeran
    Member
    Posted 1 year ago #

    What's the problem? What error message did Google post in your GWT account?

    Google couldn't crawl your site because we were unable to access the robots.txt file.
    i see this message in GWT account in site errors.
    DNS connectivity, server connectivity shows green signal, while robots.txt fetch show yellow error icon.

    Did you change any links/URLs?

    No, no URLs are changed, URLs and the domain is the same.
    also ib GWT in the blocked URLs section it shows the old (wrong) robots.txt code even after i have udated the robots.txt
    sir please guide me, as i m new and don't know now what to do.
    .
    thanks in advance.

  6. Krishna
    Volunteer Moderator
    Posted 1 year ago #

    You may replace your robots.txt file that is currently:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /cgi-bin/
    
    Allow: /
    Sitemap: http://technoriderz.com/sitemap.xml

    With

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /cgi-bin/
    
    Sitemap: http://technoriderz.com/sitemap.xml

    the above one. Then submit a new sitemap and wait for the results. You need to regularly look for errors and correct them at your GWT. The problems and solutions for them can be found at your GWT account and the help links there. As I mentioned before, this is NOT a WordPress problem.

  7. Heeran
    Member
    Posted 1 year ago #

    Thanks Dear Sir, I have updated my robots.txt as you suggested.
    i understand that it is not a WP wrror, but here i was waiting for experts to help me In GWT ac.

    Thanks for your encouraging reply and help.

  8. clubhoster
    Member
    Posted 1 year ago #

    Hello,
    Friends I am also facing this issue.te robots.txt file is showing like that.

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Sitemap: https://www.csofts.net/sitemap.xml.gz

    thanks
    Web hosting Provider
    I am waiting your reply.

Topic Closed

This topic has been closed to new replies.

About this Topic