WordPress Needs a Default robots.txt File and More...

  1. Ipstenu (Mika Epstein)

    IIRC, without a trailing slash it only blocks wp-admin, but NOT wp-admin/index.php

    Posted: 6 years ago #
  2. Bryan Hadaway


    Actually, that's not correct.

    However, I should note that what I've used does run a risk in an extreme scenario.

    I'll explain better so everyone understands:

    WITH a trailing slash.
    - - - - - - - - - - - - - - - -
    This will ONLY block that directory and it's contents.

    WITHOUT a trailing slash.
    - - - - - - - - - - - - - - - -
    This will block that directory and it's contents and ALL sub-directories and their contents. But, also any URL that starts with that.

    So my solution is actually a lot more simple and thorough at the same time, however a bit open-ended...

    So, one issue could be let's say you write an article:

    "WP Admin: Making it Secure"

    Your URL would come out as:


    Which would in effect get blocked in robots.txt (which we don't want).

    So, I can see why WordPress has it that way by default. Many of the reasons WordPress does things a certain way is to make them foolproof for the general end-user which I'm understanding more and more.

    In any case, there are still some valid points throughout this idea topic that could/should perhaps be cherry picked and created into their own ideas. As far as the virtual robots.txt default, I now see there isn't much more they could do to improve it.

    Thanks, Bryan

    Posted: 6 years ago #
  3. Ihor Vorotnov

    Bryan is right about the trailing slash. Personally I always install WordPress in separate folder and use a different folder for uploads. Then my robots.txt looks like this:

    Allow: /{wp-install-dir-name}/static
    Disallow: /{wp-install-dir-name}/wp-

    The first line allows to index uploads folder, the second one blocks everything starting with wp- - folders and files, recursively. Assuming you have some unique {wp-install-dir-name} (which is good for security) chances you'll run in a situation described by Bryan (blocking article with url starting with wp-) are close to zero.

    Posted: 6 years ago #
  4. Bryan Hadaway


    I suppose at the end of the day it lies in our own hands to take care of ourselves.

    Although once something becomes so big, even if it's "free" like WordPress or Google it enters into a whole new tier of responsibility, public responsibility.

    Right now I'm discovering all sorts of privacy/security holes with BlueHost/HostMonster (what WP recommends and I do too).

    I spent two hours with HostMonster support to finally convince them I'm not a dumb end-user and that there are indeed holes that need to be patched.

    I have a ticket going now, so once that gets resolved I'm going to revamp all my .htaccess and robots.txt files. I'd like to put together a definitive .htaccess file to lock down WordPress. I'll share it in an article at that time. Maybe others can help as well.


    Posted: 6 years ago #
  5. iptvnews

    thanks, this is still useful tip. Recently I ran into issues with new site getting indexed properly. Then I have to use WP Robots Editor plugin to black /tags pages.

    Posted: 3 years ago #
  6. Doug

    Guys you are wrong:
    User-agent: *
    Disallow: /wp-admin/

    User-agent: *
    Disallow: /wp-admin

    Have the exact same effect. Use webmaster tools robots.txt tester.
    The first blocks everything inside wp-admin folder.
    The second blocks any URL that contains wp-admin on it.

    The result is the same.

    Posted: 3 years ago #
  7. James

    @ mAsT3RpEE - The link you posted to an infected web page violates the terms on this forum.

    I hope a Moderator bans your account after removing that link.

    Posted: 3 years ago #
  8. Ipstenu (Mika Epstein)

    The link used to work. Looks like the domain was taken over.

    Don't be so quick to judge.

    Anyway. I'm closing this. Robots is in WP by default, it works, and it's extendable. We're good.

    Posted: 3 years ago #

RSS feed for this topic

Topic Closed

This topic has been closed to new replies.

  • Rating

    17 Votes
  • Status

    This idea has been implemented