WordPress.org

1. Mark (podz)
   Support Maven
   Posted 7 years ago #

   Okay, so it's all SE's.....

   Is it possible to block bots from wp pages individually ?
   Could that be done with .robots.txt ? .htaccess ? rewrite rules ?

2. XeroCool
   Member
   Posted 7 years ago #

   Podz: robots.txt.

   You can put something like this in robots.txt:

   User-agent: googlebot
   Disallow: /mypornfiles

   googlebot will not index /mypornfiles If you add that to robots.txt. :)
   To block all bots (Not just google), Make the user agent this:

   User-agent: *

   That's all. robots.txt goes inside the root folder, and that's it. For multiple files, Just keep adding the disallow code (ex:

   
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /themes/
Disallow: /admin/
Disallow: /staff/
Disallow: /static/
Disallow: /downloads/

3. angsuman
   Member
   Posted 7 years ago #

   Yes.
   How to prevent robots (Google Bot, MSNBot, Yahoo Slurp! etc.) from displaying cached pages of your site

   You can insert the following meta tag using meta tag plugin:
   <META NAME="ROBOTS" CONTENT="NOARCHIVE NOINDEX">

4. Jinsan
   Member
   Posted 7 years ago #

   podz vs google - brought to you by Warner Bros :)

5. Mark (podz)
   Support Maven
   Posted 7 years ago #

   LOL !

   Xerocool - freudian slip there maybe ?

   Cheers though - I shall use the info :)

Topic Closed

This topic has been closed to new replies.