Support » Fixing WordPress » Robots.txt from pages

Viewing 4 replies - 1 through 4 (of 4 total)
  • whooami

    (@whooami)

    Member

    what does your .htaccess look like right now?

    And, honestly, why do you want to do this, just to allow ppl to look at it?

    chinatour

    (@chinatour)

    If you post you .htaccess file some experts here might be able to help you.

    jkozuch

    (@jkozuch)

    Personally, I wouldn’t want the average joe looking at my robots.txt file. There are things in there that a private (ie, secret URLs) that are private for a reason.

    So, going back to whooami’s question, why do you want to do this?

    whooami

    (@whooami)

    Member

    jkozuch,

    robots.txt by their very nature need to be readable. thats how they work. And guess what, if Googlebot can read it, I can too, and so can everyone else.

    robots.txt are NOT intended to hide/keep private/etc.. directories, it’s intended to keep a directory from being indexed. and thats NOT the same thing. Additionally, since a robots.txt file needs to be readable, by design, putting the name of a directory that you want to be “hidden” (and I use that term loosely) inside it, defeats the purpose.

    In a nutshell:

    a directory which you desire to be “hidden” (and I use that term loosely) should not be linked to. NOT just because a searchbot will find it, but because a human will, inevitably.

    a directory that you dont want indexed is probably going to be one with dynamic content, files that are downloadable, perhaps a directory that once contained ‘things’ but no longer does, and google (for instance) keeps going back to check on from time to time.

    apples and oranges.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Robots.txt from pages’ is closed to new replies.