WordPress.org

Ready to get started?Download WordPress

Forums

block Searchbots? (13 posts)

  1. johanneshahn
    Member
    Posted 7 years ago #

    Hi guys. How can I stop searchbots like Yahoo or Google from accessing my site? I don't want them to crawl my blog and appear in the statistics (SlimStat)

    Please help me!

    Regards, Johannes.

  2. Chris_K
    Member
    Posted 7 years ago #

    http://www.robotstxt.org/ would be a good place to start.

    Or use analytics that allow you to filter out robots (like, do you really not want to show up in Google searches?)

  3. johanneshahn
    Member
    Posted 7 years ago #

    Yes, I don't want to be in Google or Yahoo, I want my blog to be seen by the people I know, no one else.

    PS: Thanks ;) What's that analytics thing you talked about?

  4. johanneshahn
    Member
    Posted 7 years ago #

    So basically I just write this:

    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

    in a robot.txt file and upload it on the server? That's it?

  5. whooami
    Member
    Posted 7 years ago #

    kill the caps in that, it'll break your validation, and no, that would go at the top of your actual pages ..

    youre confusing methods:

    http://www.barnettcomputerservices.com/robots.htm

    and from the horses mouth:

    http://www.google.com/support/webmasters/bin/answer.py?answer=40364

    Google is providing what would go in your robots.txt, just so you know.

  6. johanneshahn
    Member
    Posted 7 years ago #

    Thanks matey. But that'll only stop the google bot, won't it? What about Yahoo and all the others? Where do I get the names?

  7. Dgold
    Member
    Posted 7 years ago #

    you might want to create the Robots.txt and move your site to a different URL

    because once Google already knows your URL, I think your links they already indexed stay there even if you put it in Robots.txt for no future indexing

    Also, you want to go into your WP-Admin > Options > and turn off pinging, that is: delete Ping-o-matic from the pinging list, so that your WP does not automatically notify technorati and other sites each time you Publish a post.

  8. whooami
    Member
    Posted 7 years ago #

    Thanks matey. But that'll only stop the google bot, won't it? What about Yahoo and all the others? Where do I get the names?

    how about Google? : http://www.google.com/search?hl=en&lr=&q=robots.txt+user+agents&btnG=Search

    http://www.searchtools.com/robots/robots-txt.html

    ^ the third result and it explains quite nicely how to block all bots that respect robots.txt

    Search engines are these miraculous new things, I hear.

  9. johanneshahn
    Member
    Posted 7 years ago #

    Thanks a lot guys, I applied it and hope it'll work!
    Nice Regards, me.

  10. kalico
    Member
    Posted 7 years ago #

    I know this is an old topic, but just for the record, Google is really good about removing your links from their site if you ask them to. It might take a little time, but they will do it. Do a little searching on Google help to find the right reporting/requesting method.

    In both cases I had to ask them to remove links, it was an emergency situation. In one case, the name of a person who was under threat of personal harm had been inadvertently published on a public web page. In another case, some very private professional discussions had been archived to a very public forum accidentally. In both cases, Google removed the cached pages from their site immediately, and the links themselves after a few days.

    I really think that ping-o-matic should be turned OFF by default in order to protect the privacy of the new user who has no idea this option is even there, and starts blogging indiscriminately, thinking he is playing in his own private sandbox.

  11. whooami
    Member
    Posted 7 years ago #

    I really think that ping-o-matic should be turned OFF by default in order to protect the privacy of the new user who has no idea this option is even there, and starts blogging indiscriminately, thinking he is playing in his own private sandbox.

    I disagree, but thats me. Thats one more way of dumbing things down I guess though. The average intelligence quotient to use wordpress right now is in the range of 75 - I s'pose theres 74 more numbers to go though before we hit 0.

    Eventually, my (dumbest)dog will be able to use it. Oh yay. I cant wait for that subset of bloggers to hit the web.

    Isn't it common sense that something thats posted on the Internet is there for ALL to see??? I think I even saw a Public Service Announcement on TV reminding people of that just yesterday.

  12. dylan
    Member
    Posted 7 years ago #

    put this meta tag in between the <head> and </head> tags in your html

    <meta name="robots" content="noindex,nofollow" />

  13. Samuel Wood (Otto)
    Tech Ninja
    Posted 7 years ago #

    dylan: In WordPress 2.1 and up, you can go to the Options->Privacy page and set it to block from search engines to achieve the exact same thing. No need to manually edit the template.

Topic Closed

This topic has been closed to new replies.

About this Topic