WordPress.org

1. itmdesign
   Member
   Posted 2 years ago #

   Hi, I've recently been working on reducing the amount of duplicate content in my blog by denying robot access to category/archive pages etc.

   My website has a robots.txt in the root with the blog sitting in a subfolder. I've noticed that WordPress generates a robots.txt file that I can't seem to delete this file.

   Would it being there effect my robots.txt file in the root folder?

   Example of root robots:
   

   Disallow: /blog/wp-admin/
   Disallow: /blog/wp-content/
   Disallow: /blog/wp-includes/

   WordPress generated robots:
   

   User-agent: *
   Disallow:

2. imak47
   Member
   Posted 2 years ago #

   I'm facing the problems with this virtual robot.txt in WordPress.org 2.8.6
   I have the privacy settings to Allow... and when I see the robot.txt file it says: Disallow (??) - I'm totally perplexed by that one. I swear I have hit the save settings button atleast 3 times here. Then I installed a plug in and that doesnt help either. Can anybody please give a solution to this?

   Thanks

Topic Closed

This topic has been closed to new replies.