Privacy & robots.txt

  • Mark (podz)

    (@podz)


    19 years, 8 months ago

    Seeing as we get quite a few posts asking about privacy, and search engines need to be taken into account….this looks good
    http://www.webmasterworld.com/robots.txt
    I don’t think it’s the complete solution by quite some distance, but it’s better than nothing unless .htaccess and lots of regular expressions is a walk in the park for you….

Viewing 3 replies - 1 through 3 (of 3 total)
  • Sushubh

    (@sushubh)

    19 years, 8 months ago

    trust me robot files rarely work. When I was on Blog:CMS, even with robot file properly denying access to spiders to one of the php files, I still got karma points from Inktomi and Google Spiders. 🙁

    desertjo

    (@desertjo)

    19 years, 8 months ago

    robots files rarely work because only the legitimate search engines will read them before trying to read your site.
    there are far too many other bots that people use to troll websites, and they never hit the robots file, so they are never following the access rules you’ve set up.
    when that kind of abuse gets out of hand on my sites, i block them directly in the httpd.conf 🙂 BaiDuSpider out of China is one of them.

    southerngal

    (@southerngal)

    19 years, 8 months ago

    I use the referrers.php found here.
    I have to add that in my stats there are a lot of robot.txt blocked hits. So that’s also working for me. 🙂

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Privacy & robots.txt’ is closed to new replies.