WordPress.org

1. homerm06
   Member
   Posted 4 years ago #
   

   I wanted to know if a password protected page would be seen by spider search engines. Thanks!

2. Saurus
   Member
   Posted 4 years ago #
   

   If you want to be SURE no bots can crawl your site - you should add something like the following to your .htaccess file. The ones listed are the most common and unwanted, but you can add others using the format shown.
   

   RewriteEngine On
   RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
   RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
   RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
   RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
   RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
   RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
   RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
   RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
   RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
   RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
   RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
   RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
   RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
   RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
   RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
   RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
   RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
   RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
   RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
   RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
   RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
   RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
   RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
   RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
   RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
   RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Zeus
   RewriteRule ^.* - [F,L]

3. homerm06
   Member
   Posted 4 years ago #
   

   WOW that's complicating :(

4. mikey1
   Member
   Posted 4 years ago #
   

   http://wordpress.org/extend/plugins/wp-sentry/

5. homerm06
   Member
   Posted 4 years ago #
   

   I just tried it but when I downloaded the 0.6 version, my wordpress tells me that I have 0.5.1 :(

Topic Closed

This topic has been closed to new replies.