Support » Fixing WordPress » block Searchbots?

  • Hi guys. How can I stop searchbots like Yahoo or Google from accessing my site? I don’t want them to crawl my blog and appear in the statistics (SlimStat)

    Please help me!

    Regards, Johannes.

Viewing 12 replies - 1 through 12 (of 12 total)
  • http://www.robotstxt.org/ would be a good place to start.

    Or use analytics that allow you to filter out robots (like, do you really not want to show up in Google searches?)

    Yes, I don’t want to be in Google or Yahoo, I want my blog to be seen by the people I know, no one else.

    PS: Thanks 😉 What’s that analytics thing you talked about?

    So basically I just write this:

    <META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>

    in a robot.txt file and upload it on the server? That’s it?

    whooami

    (@whooami)

    Member

    kill the caps in that, it’ll break your validation, and no, that would go at the top of your actual pages ..

    youre confusing methods:

    http://www.barnettcomputerservices.com/robots.htm

    and from the horses mouth:

    http://www.google.com/support/webmasters/bin/answer.py?answer=40364

    Google is providing what would go in your robots.txt, just so you know.

    Thanks matey. But that’ll only stop the google bot, won’t it? What about Yahoo and all the others? Where do I get the names?

    you might want to create the Robots.txt and move your site to a different URL

    because once Google already knows your URL, I think your links they already indexed stay there even if you put it in Robots.txt for no future indexing

    Also, you want to go into your WP-Admin > Options > and turn off pinging, that is: delete Ping-o-matic from the pinging list, so that your WP does not automatically notify technorati and other sites each time you Publish a post.

    whooami

    (@whooami)

    Member

    Thanks matey. But that’ll only stop the google bot, won’t it? What about Yahoo and all the others? Where do I get the names?

    how about Google? : http://www.google.com/search?hl=en&lr=&q=robots.txt+user+agents&btnG=Search

    http://www.searchtools.com/robots/robots-txt.html

    ^ the third result and it explains quite nicely how to block all bots that respect robots.txt

    Search engines are these miraculous new things, I hear.

    Thanks a lot guys, I applied it and hope it’ll work!
    Nice Regards, me.

    I know this is an old topic, but just for the record, Google is really good about removing your links from their site if you ask them to. It might take a little time, but they will do it. Do a little searching on Google help to find the right reporting/requesting method.

    In both cases I had to ask them to remove links, it was an emergency situation. In one case, the name of a person who was under threat of personal harm had been inadvertently published on a public web page. In another case, some very private professional discussions had been archived to a very public forum accidentally. In both cases, Google removed the cached pages from their site immediately, and the links themselves after a few days.

    I really think that ping-o-matic should be turned OFF by default in order to protect the privacy of the new user who has no idea this option is even there, and starts blogging indiscriminately, thinking he is playing in his own private sandbox.

    whooami

    (@whooami)

    Member

    I really think that ping-o-matic should be turned OFF by default in order to protect the privacy of the new user who has no idea this option is even there, and starts blogging indiscriminately, thinking he is playing in his own private sandbox.

    I disagree, but thats me. Thats one more way of dumbing things down I guess though. The average intelligence quotient to use wordpress right now is in the range of 75 – I s’pose theres 74 more numbers to go though before we hit 0.

    Eventually, my (dumbest)dog will be able to use it. Oh yay. I cant wait for that subset of bloggers to hit the web.

    Isn’t it common sense that something thats posted on the Internet is there for ALL to see??? I think I even saw a Public Service Announcement on TV reminding people of that just yesterday.

    put this meta tag in between the <head> and </head> tags in your html

    <meta name="robots" content="noindex,nofollow" />

    dylan: In WordPress 2.1 and up, you can go to the Options->Privacy page and set it to block from search engines to achieve the exact same thing. No need to manually edit the template.

Viewing 12 replies - 1 through 12 (of 12 total)
  • The topic ‘block Searchbots?’ is closed to new replies.