WordPress.org

Ideas

WordPress Needs a Default robots.txt File and More...

  1. Ipstenu (Mika Epstein)
    Administrator

    IIRC, without a trailing slash it only blocks wp-admin, but NOT wp-admin/index.php

    Posted: 3 years ago #
  2. Bryan Hadaway
    Member

    12345

    Actually, that's not correct.

    However, I should note that what I've used does run a risk in an extreme scenario.

    I'll explain better so everyone understands:

    WITH a trailing slash.
    - - - - - - - - - - - - - - - -
    This will ONLY block that directory and it's contents.

    WITHOUT a trailing slash.
    - - - - - - - - - - - - - - - -
    This will block that directory and it's contents and ALL sub-directories and their contents. But, also any URL that starts with that.

    So my solution is actually a lot more simple and thorough at the same time, however a bit open-ended...

    So, one issue could be let's say you write an article:

    "WP Admin: Making it Secure"

    Your URL would come out as:

    http://website.com/wp-admin-making-it-secure/

    Which would in effect get blocked in robots.txt (which we don't want).

    So, I can see why WordPress has it that way by default. Many of the reasons WordPress does things a certain way is to make them foolproof for the general end-user which I'm understanding more and more.

    In any case, there are still some valid points throughout this idea topic that could/should perhaps be cherry picked and created into their own ideas. As far as the virtual robots.txt default, I now see there isn't much more they could do to improve it.

    Thanks, Bryan

    Posted: 3 years ago #
  3. Ihor Vorotnov
    Member

    Bryan is right about the trailing slash. Personally I always install WordPress in separate folder and use a different folder for uploads. Then my robots.txt looks like this:

    Allow: /{wp-install-dir-name}/static
    Disallow: /{wp-install-dir-name}/wp-

    The first line allows to index uploads folder, the second one blocks everything starting with wp- - folders and files, recursively. Assuming you have some unique {wp-install-dir-name} (which is good for security) chances you'll run in a situation described by Bryan (blocking article with url starting with wp-) are close to zero.

    Posted: 3 years ago #
  4. Bryan Hadaway
    Member

    12345

    I suppose at the end of the day it lies in our own hands to take care of ourselves.

    Although once something becomes so big, even if it's "free" like WordPress or Google it enters into a whole new tier of responsibility, public responsibility.

    Right now I'm discovering all sorts of privacy/security holes with BlueHost/HostMonster (what WP recommends and I do too).

    I spent two hours with HostMonster support to finally convince them I'm not a dumb end-user and that there are indeed holes that need to be patched.

    I have a ticket going now, so once that gets resolved I'm going to revamp all my .htaccess and robots.txt files. I'd like to put together a definitive .htaccess file to lock down WordPress. I'll share it in an article at that time. Maybe others can help as well.

    Thanks

    Posted: 3 years ago #
  5. iptvnews
    Member

    thanks, this is still useful tip. Recently I ran into issues with new site getting indexed properly. Then I have to use WP Robots Editor plugin to black /tags pages.

    Posted: 1 year ago #
  6. nicoblog
    Member

    Guys you are wrong:
    User-agent: *
    Disallow: /wp-admin/

    User-agent: *
    Disallow: /wp-admin

    Have the exact same effect. Use webmaster tools robots.txt tester.
    The first blocks everything inside wp-admin folder.
    The second blocks any URL that contains wp-admin on it.

    The result is the same.

    Posted: 10 months ago #
  7. Jim Hall
    Member

    @ mAsT3RpEE - The link you posted to an infected web page violates the terms on this forum.

    I hope a Moderator bans your account after removing that link.

    Posted: 7 months ago #
  8. Ipstenu (Mika Epstein)
    Administrator

    The link used to work. Looks like the domain was taken over.

    Don't be so quick to judge.

    Anyway. I'm closing this. Robots is in WP by default, it works, and it's extendable. We're good.

    Posted: 7 months ago #

RSS feed for this topic

Topic Closed

This topic has been closed to new replies.

  • Rating

    12345
    17 Votes
  • Status

    This idea has been implemented