Privacy & robots.txt (4 posts)

  1. Mark (podz)
    Support Maven
    Posted 11 years ago #

    Seeing as we get quite a few posts asking about privacy, and search engines need to be taken into account....this looks good
    I don't think it's the complete solution by quite some distance, but it's better than nothing unless .htaccess and lots of regular expressions is a walk in the park for you....

  2. OperaManiac
    Posted 11 years ago #

    trust me robot files rarely work. When I was on Blog:CMS, even with robot file properly denying access to spiders to one of the php files, I still got karma points from Inktomi and Google Spiders. :(

  3. DesertJo
    Posted 11 years ago #

    robots files rarely work because only the legitimate search engines will read them before trying to read your site.
    there are far too many other bots that people use to troll websites, and they never hit the robots file, so they are never following the access rules you've set up.
    when that kind of abuse gets out of hand on my sites, i block them directly in the httpd.conf :) BaiDuSpider out of China is one of them.

  4. southerngal
    Posted 11 years ago #

    I use the referrers.php found here.
    I have to add that in my stats there are a lot of robot.txt blocked hits. So that's also working for me. :)

Topic Closed

This topic has been closed to new replies.

About this Topic


No tags yet.