unwanted robots.txt (3 posts)

  1. hm2k
    Posted 7 years ago #

    For whatever reason, when I visit http://www.example.com/robots.txt it displays the following:

    User-agent: *

    Yet, the file robots.txt does not exist, and in privacy settings I have...

    "I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers" selected, and "I would like to block search engines, but allow normal visitors" deselected.

    The latter was previously selected, but I changed this, and upgraded from 2.3.3 to 2.6.

    I can't work out how to get rid of this robots.txt file...

    Please help, this site is no longer listed in google because of this issue.


  2. rawalex
    Posted 7 years ago #

    You may want to check with your host... they may have put a global block in to limit bot traffic.

  3. 1. That robots.txt is generated by WordPress.
    2. That particular robots.txt is not blocking anything, including Google. A blank Disallow: line means that everything is allowed. Look at the documentation on http://www.robotstxt.org/
    3. If you really want to get rid of it, make a blank robots.txt file and put it in your web root.

Topic Closed

This topic has been closed to new replies.

About this Topic