WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] robots.txt and google (13 posts)

  1. ruatuki
    Member
    Posted 1 year ago #

    My site has been indexed by google but says "A description for this result is not available because of this site's robots.txt ". I could not find a robots.txt file on my site and I had set wordpress privacy to "Allow search engines to index this site." and it has been this way since I created the site. I have done some google research but can't find an answer.
    I have removed all plugins and am using squirrel pro theme which is the only one installed.
    I created a robots.txt file containing "User-agent: *
    Disallow: /wp-content/plugins/"
    and uploaded that to the root folder but going to http://www.anact.co.nz/robots.txt still shows
    "User-agent: *
    Disallow: /"
    Don't know where to go now...

  2. David Choi
    Member
    Posted 1 year ago #

    Maybe your FTP client is not overwriting the file? Another possible reason is that your site's web server is using a cache.... file is not updated on time (ask your host).

  3. ruatuki
    Member
    Posted 1 year ago #

    Thanks for the help David. There was no robots.txt file in the root folder before I uploaded one. My research suggested it could have been a dynamically generated one and that uploading a new, corrected file might solve the problem but it appears not.

  4. David Choi
    Member
    Posted 1 year ago #

    Then what if you actually edit that file, using the File Manager from your host's panel (like the cPanel, DirectAdmin, etc.)

  5. ruatuki
    Member
    Posted 1 year ago #

    The robots.txt file that is now in the root folder was the one I created and uploaded I'm not sure what editing it will achieve. It is the only robots.txt file that has ever been there. It contains "User-agent: *
    Disallow: /wp-content/plugins/"

  6. David Choi
    Member
    Posted 1 year ago #

    Checked the link

    http://www.anact.co.nz/robots.txt

    but it's still

    "User-agent: *
    Disallow: /"

    so I think it's the cache problem from your web server. See what your host support has to say.

  7. ruatuki
    Member
    Posted 1 year ago #

    If I go to http://www.anact.co.nz/robots.txt then, in Firefox go tools>page info it says "Modified Friday, 3 November 2006 2:20:08 p.m." whilst the one I uploaded has todays date. Suggesting it is dynamically generated?

  8. David Choi
    Member
    Posted 1 year ago #

    Maybe you are editing the wrong robots.txt file?

    Try a test, create a file called

    test.txt

    and put something in it, see if you can access it directly by

    http://www.anact.co.nz/test.txt

  9. ruatuki
    Member
    Posted 1 year ago #

    Yep that worked.

  10. David Choi
    Member
    Posted 1 year ago #

    Then indeed, I think it's replacing that file internally like you said. Only your host support can solve the problem.

  11. ruatuki
    Member
    Posted 1 year ago #

    The robots.txt file in the root folder does not match the result I get when I go to http://www.anact.co.nz/robots.txt. The problem is not resolved.

  12. ruatuki
    Member
    Posted 1 year ago #

    OK, maybe my post 'yep that worked' was misleading, I meant 'yep that worked I can upload a test file no problem' NOT yep that fixed my robots.txt problem. The original problem remains.

  13. tashb
    Member
    Posted 1 year ago #

    Hi,

    Please refer to my post here:

    http://wordpress.org/support/topic/googlebot-is-blocked-by-robotstxt?replies=12#post-3129828

    I waited a couple of days, and it seems Google has crawled my site again because it's no longer saying the robots.txt file is blocking site indexing, even though I changed nothing.

    Waiting seems to work.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags