http://www.robotstxt.org/ would be a good place to start.
Or use analytics that allow you to filter out robots (like, do you really not want to show up in Google searches?)
Yes, I don’t want to be in Google or Yahoo, I want my blog to be seen by the people I know, no one else.
PS: Thanks 😉 What’s that analytics thing you talked about?
So basically I just write this:
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
in a robot.txt file and upload it on the server? That’s it?
kill the caps in that, it’ll break your validation, and no, that would go at the top of your actual pages ..
youre confusing methods:
http://www.barnettcomputerservices.com/robots.htm
and from the horses mouth:
http://www.google.com/support/webmasters/bin/answer.py?answer=40364
Google is providing what would go in your robots.txt, just so you know.
Thanks matey. But that’ll only stop the google bot, won’t it? What about Yahoo and all the others? Where do I get the names?
you might want to create the Robots.txt and move your site to a different URL
because once Google already knows your URL, I think your links they already indexed stay there even if you put it in Robots.txt for no future indexing
Also, you want to go into your WP-Admin > Options > and turn off pinging, that is: delete Ping-o-matic from the pinging list, so that your WP does not automatically notify technorati and other sites each time you Publish a post.
Thanks matey. But that’ll only stop the google bot, won’t it? What about Yahoo and all the others? Where do I get the names?
how about Google? : http://www.google.com/search?hl=en&lr=&q=robots.txt+user+agents&btnG=Search
http://www.searchtools.com/robots/robots-txt.html
^ the third result and it explains quite nicely how to block all bots that respect robots.txt
Search engines are these miraculous new things, I hear.
Thanks a lot guys, I applied it and hope it’ll work!
Nice Regards, me.
I know this is an old topic, but just for the record, Google is really good about removing your links from their site if you ask them to. It might take a little time, but they will do it. Do a little searching on Google help to find the right reporting/requesting method.
In both cases I had to ask them to remove links, it was an emergency situation. In one case, the name of a person who was under threat of personal harm had been inadvertently published on a public web page. In another case, some very private professional discussions had been archived to a very public forum accidentally. In both cases, Google removed the cached pages from their site immediately, and the links themselves after a few days.
I really think that ping-o-matic should be turned OFF by default in order to protect the privacy of the new user who has no idea this option is even there, and starts blogging indiscriminately, thinking he is playing in his own private sandbox.
I really think that ping-o-matic should be turned OFF by default in order to protect the privacy of the new user who has no idea this option is even there, and starts blogging indiscriminately, thinking he is playing in his own private sandbox.
I disagree, but thats me. Thats one more way of dumbing things down I guess though. The average intelligence quotient to use wordpress right now is in the range of 75 – I s’pose theres 74 more numbers to go though before we hit 0.
Eventually, my (dumbest)dog will be able to use it. Oh yay. I cant wait for that subset of bloggers to hit the web.
Isn’t it common sense that something thats posted on the Internet is there for ALL to see??? I think I even saw a Public Service Announcement on TV reminding people of that just yesterday.
put this meta tag in between the <head> and </head> tags in your html
<meta name="robots" content="noindex,nofollow" />
dylan: In WordPress 2.1 and up, you can go to the Options->Privacy page and set it to block from search engines to achieve the exact same thing. No need to manually edit the template.
