robots.txt

do you want to know what its for?

http://www.robotstxt.org/

or what ppl typically put it in?

Its a browsable file, just look for some on the net:

http://www.google.com/robots.txt

look at mine.. 🙂 the easiest way to answer that question is just to check out some, like I already suggested.

Note that the only sites that are going to have them are ones where the site op is paying attention to their logs, or is actually cognizant of what a robots.txt is for and it’s effects.

umm yeah, whats your point? click it..its no more ‘exposed’ that it is being linked off every page on my site.

Thats the entire point OF a robots.txt, to exclude directories from being spidered .. how do you think you do THAT without listing them.. I DONT want that directory spidered.

And guess what, google doesnt spider it:

http://www.google.com/search?hl=en&q=file-downloads+village-idiot&btnG=Google+Search

too bad you dont understand that, and/or dont read links that provide explanations. (tech support, lemme guess, level one)

“too bad you dont understand that, and/or dont read links that provide explanations. (tech support, lemme guess, level one)”

Damn whooami! Chill out! Flying off the handle like that is one of the main reasons why people (like myself) choose NOT to contribute to free tech support. Ridiculed when we’re only trying to help? If I was incorrect with what I had said, then my bad, but that gives you no reason, nor right to react and comment the way you did! Seriously, how many forums to you actively participate in?

That all said, I was merely offering valid input to people that are either new to web development or could possibly (for some reason) not realize something very important. Correct me if I’m wrong, but what I’ve read about the robots.txt file is that unless access to your robots.txt is password protected, when you list directories in your robots.txt file, you are exposing these directories to the public. In other words, if you do not want your average person finding out about a directory, then be sure to take extra precautions to prevent access to these directories.

FYI, I’m not a web developer, nor am I “level 1” tech support, so keep your guessing cuz I find it quite humerous:)

That all said, I was merely offering valid input to people that are either new to web development or could possibly (for some reason) not realize something very important. Correct me if I’m wrong, but what I’ve read about the robots.txt file is that unless access to your robots.txt is password protected, when you list directories in your robots.txt file, you are exposing these

Quite honestly, given your obvious lack of knowledge on this topic, anyone reading is better off without you posting. Again, my first reply in this thread provided a link that would happily correct if youre wrong.. but youve obviously not checked it … yet.

So lets..

valid input

Its not valid, because youre wrong. Hence my responses. And you just keep repeating stuff thats not valid too.

not realize something very important..

Its not important because youre wrong in your premise. And thats precisely why I provided a link that explained its purpose more clearly. SO that someone reading this thread would understand its NOT to be used for “hiding” a directory.

..password protect..

Where have you read this? Please feel free to share a link that suggests password protecting the robots.txt file.

Then lets apply some common sense: And how is it that you think that a robot is going to going to “read” the robots.txt once youve password protected it?

Thats patently false.

if you do not want your average person finding out about a directory,

Thats not the point of a robots.txt file. Otto explained, the link I provided explained (had you read it..)

Again, patently false, given the context of this thread.

—

What you dont seem to be grasping is that the point of a robots.txt is NOT to hide the directory, its to prevent spidering, which is entirely different.

Here is a scenario that might bring this down to something more understandable for you:

The link you posted .. the one on my domain. Its linked off all but very few (2-5 pages) of my site. Its no secret. I take no issue with anyone looking at it.

BUT …

Given the dynamic nature of it’s and any subdirectory’s content, I dont want Google to index it.

Why? Because theres a very good chance someone will follow an outdated link.

Because I want the user to have to see the directory in its entirity rather than through the lens of Google’s truncated listing.

Correct me if I’m wrong …

Consider yourself corrected.
—

RoboPower,

This thread isnt about me or you, it’s about the correct usage of a robots.txt file.

That said, what does your version of netiquette tell you about posting a url on a public forum that you “thought I was trying to hide”? I guess you missed the fact that I opted to NOT question why you would do such a thing?

As for my courtesy, my responses to you were no less courteous than Ottos:

Uhhh.. What would be the point of that?

In fact, the only personal remarks Ive made regarding you questioned your lack of reading, as evidenced by your replies.

And I even went so far as to apologize. And yet you continue, apparantly still w/o reading.

On the other hand.. we have your remarks.

…I celebrate christmas with my family, we have a great time. My mom is intelligent and articulate.

As am I.

I’ll be sure to tell her you said hello.

As this really isn’t about robots.txt anymore, I’m closing it.