WordPress.org

Ready to get started?Download WordPress

Forums

Robots.txt from pages (5 posts)

  1. ewald
    Member
    Posted 7 years ago #

    ok, my problem is my robots.txt-file is 404 from all pages except the frontpage.
    how do i set up my .htaccess to have http://www.mywordpressblog.com/my-page-nice-huh/robots.txt to be found?

  2. whooami
    Member
    Posted 7 years ago #

    what does your .htaccess look like right now?

    And, honestly, why do you want to do this, just to allow ppl to look at it?

  3. chinatour
    Member
    Posted 7 years ago #

    If you post you .htaccess file some experts here might be able to help you.

  4. jkozuch
    Member
    Posted 7 years ago #

    Personally, I wouldn't want the average joe looking at my robots.txt file. There are things in there that a private (ie, secret URLs) that are private for a reason.

    So, going back to whooami's question, why do you want to do this?

  5. whooami
    Member
    Posted 7 years ago #

    jkozuch,

    robots.txt by their very nature need to be readable. thats how they work. And guess what, if Googlebot can read it, I can too, and so can everyone else.

    robots.txt are NOT intended to hide/keep private/etc.. directories, it's intended to keep a directory from being indexed. and thats NOT the same thing. Additionally, since a robots.txt file needs to be readable, by design, putting the name of a directory that you want to be "hidden" (and I use that term loosely) inside it, defeats the purpose.

    --

    In a nutshell:

    a directory which you desire to be "hidden" (and I use that term loosely) should not be linked to. NOT just because a searchbot will find it, but because a human will, inevitably.

    a directory that you dont want indexed is probably going to be one with dynamic content, files that are downloadable, perhaps a directory that once contained 'things' but no longer does, and google (for instance) keeps going back to check on from time to time.

    --

    apples and oranges.

Topic Closed

This topic has been closed to new replies.

About this Topic