WordPress Robots.txt

  • will888

    (@will888)


    18 years, 9 months ago

    I received a message from gg webmasters tools saying

    We can’t currently access your home page because of an unreachable error.

    when i checked the robots.txt analysis it says

    User-agent: *
    Disallow:

    i tried to delete the file in my FTP but i cannot see it. I looked on all DIR

    i hope some can help me

    thanks

Viewing 7 replies - 1 through 7 (of 7 total)
  • Chris_K

    (@handysolo)

    18 years, 9 months ago

    Check your settings at Options -> Privacy.

    Do you have it set to block search engines by chance?

    Thread Starter will888

    (@will888)

    18 years, 9 months ago

    I have set it but I unchecked it

    Thread Starter will888

    (@will888)

    18 years, 9 months ago

    Where can I find the robots.txt

    Chris_K

    (@handysolo)

    18 years, 9 months ago

    That option that you unchecked was probably the issue.

    WP doesn’t install a robots.txt. If you have manually added one, you presumably added it to either your site’s main/root directory or your WP directory if not in the root.

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    18 years, 9 months ago

    If you have that privacy option set and don’t actually have a robots.txt file, then WordPress will auto-generate a pretend one when you hit it on the website.

    Check your robots.txt URL again, you’ll find it’s not giving that same disallow thing anymore.

    Chris_K

    (@handysolo)

    18 years, 9 months ago

    Oh ho – does it? Live and learn!

    I had just assumed (shame on me) that WP would do robots stuff via META on each page when the privacy option was on.

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    18 years, 9 months ago

    It does that meta stuff too, but the robots.txt rewriting was intro’d at about the same time as that.

    There’s even special code to handle it. is_robots() returns true on a robots.txt rewrite, it calls the ‘do_robots’ and ‘do_robotstxt’ action hooks, that sort of thing.

Viewing 7 replies - 1 through 7 (of 7 total)

The topic ‘WordPress Robots.txt’ is closed to new replies.