WordPress.org

1. drumat5280
   Member
   Posted 6 years ago #
   

   How do I disallow robots from spidering my archive folders in WordPress?

   Here is how my URL's are structured:

   http://www.example.com/blog/index.php/2007/04/

   I tried testing this in Webmaster Central, but it returned back with an allow status:

   User-agent: *
   Disallow: /*/2007/$

2. kalico
   Member
   Posted 6 years ago #
   

   I have never seen the $ used in a robots.txt file -- but I am no expert either.

   Try this website on creating good robots.txt files:

   http://www.robotstxt.org/

   If you find the right combo, post it here!

   HTH :)

3. whooami
   Member
   Posted 6 years ago #
   

   User-agent: *
   Disallow: /blog/2007

   will work just fine.

4. drumat5280
   Member
   Posted 6 years ago #
   

   How do I test it to make sure I have done it right?

Topic Closed

This topic has been closed to new replies.